Last time, I shared how instead of coding, I spend my time building and tuning "the factory" in which agents, me, and my teammates collaborate to get a ton of stuff done quickly, with high quality, and without nagging people about unimportant approvals or details. The post was more about how I think of my job now compared to before. I talked about how I spend my time now on tuning a "software factory" that builds a bunch of things in parallel, working through the tedious parts on its own while involving me only where I want or need to be involved. This post gets into some of the specifics.

The day-to-day and week-to-week of shipping software has changed a lot. To kick things off, here's a sort of "day in the life" of how I used to get things done and how I get things done now.

In the days of yore

Let's start with project work. This meant things like designing the user experience and the systems to drive it, implementing it, testing it, fixing it, and getting it operationally ready. This is work the team did while marching toward a feature launch. To make sure we built the right things, we wrote design documents and held meetings to talk about them. We wrote code and iterated on it until we were ready for it to be reviewed by teammates. If teammates had comments, we reworked the code to address the feedback. We each brought our own experience and preferences about how the code should be structured, and deliberated on it to make sure it's factored well for future plans. We planned sprints and had standups to coordinate on things to make sure we're not stepping on each other's toes, and to help each other or adjust the plan when we're blocked.

While many already feel nostalgia toward those days, it was not always rosy. On designs, we spent sometimes many rounds of back-and-forth, deliberating to make sure they were future-proofed and avoided pitfalls. And if we found partway through implementation that we were wrong or learned something new, we'd have to backtrack and rework, or would wring our hands to try to figure out how to plow forward rather than throw away code. On code reviews, we quibbled over code style and design patterns, blocking each other from checking in until we either came to an agreement (or a settled disagreement) and moved forward. And we'd wait hours or days between rounds of feedback because we were each in the zone on our hands-on work, or in a meeting or just working slightly different hours and couldn't drop it all on a dime to give the next round of feedback. Sprint planning and standups were mind-numbingly boring and consumed a ton of time of estimating work and thinking further down the road into the plan than at least I had patience for. But on balance, this stuff was necessary. It's how we put our heads together, learned from each other, built high-quality systems, and moved things along.

The world of tomorrow (today!)

Now, project work looks different, from design, to coding, to review, to sprint planning and standups.

Today, designs have a few different starting points. They may start with shorter conversations with teammates about the "what" and "why". As we collaborate, we want to have a cohesive vision for how the thing is being built and what we're building. I might start with writing a shorter paper talking about the problem statement and a shape of how I want to solve it. I do a lot of brainstorming with an agent, having it do research on tools or designs, having it pick apart my ideas. But throughout this process, I'm writing, reading, researching, sitting back and thinking, communicating, weighing tradeoffs, bringing in my expertise, experience, and taste, and also learning plenty. But in the end, with all of this back and forth, I produce some kind of design or spec that I'm happy with. Depending on the task, this might be a structured approach like spec-driven development, or any kind of /plan mode or prompting.

I don't code anymore, and code review looks very different too. I have an understanding of the code and the project, and an opinion on how it should be shaped going forward. I use this to steer the design and implementation plan, but that lengthy time I would have spent coding is freed up. My review loop rarely is about the end code. I do most of my review and back-and-forth in that earlier design phase, where I iterate with the agent on sketches of the implementation, like what interfaces it will add or change, architectural changes it needs, and what instrumentation it will add to make it debuggable. With this amount of detail, I can decide whether the agent's approach meets the long-term structure that we need to move fast.

Team planning is different too. Now things move easily 4x faster, so a detailed sprint plan like before is hardly worth doing. Standups on my team are very different than before. Instead of a routine status dump, we cover whatever design ideas, future plans for the next few days, or explore the customer experience of mocks or more often of working implementation. It's a hodgepodge of discussion that we don't plan until we get there. If there aren't things to review, discuss, or plan, they can be 10 minutes. Our team's record for longest standup was 10 minutes short of 3 hours. When someone described what they were going to do next, we realized we had some design and architecture that we had been deferring but finally came due, so we did it as a group. We still need to coordinate so we don't step on each other's toes with big changes to the same parts of the system, duplicate work, or even contradictory work. Coordination and communication still feel like the constraints for how much we can get done, so we break down tasks in a more coarse-grained, independent way than before. And we still agree on our team milestones so we don't go off in a bunch of directions and end up spending our coordination tax on the things that are needed less urgently.

Loop engineering

In an earlier blog post, Agentic bumper bowling, I talked about agentic loops. In the software factory, there are also loops outside the agentic loops of the design agent, or coding agent or testing agent, or review agent, or whatever agent. If the loops aren't built into the factory, then you're the one nudging everything along between steps or sending things back for rework. This is super tedious.

Some 20 years ago, a coworker described having to run tedious commands himself as "typing it with his face" - not the most ergonomic, speedy, or fun way to do things. Or later, some coworkers were teasing me when I tried to show someone some log analysis tricks on their Mac laptop. But since I was used to Windows (mostly for the productivity tools - my coding was all on a remote Linux machine), I didn't know how to do simple stuff like jump words. I forget who drew the comparison, but the imagery we ended up with was me being a t-rex who was attempting to type, but since t-rex arms are short I'd be using two of those grabby extension things, but of course each with a t-rex head doing the chomping. And so I immediately commissioned an artist on Fiverr to render that for us.

So what do I do to make this ergonomic in the factory? I make loops. Here are a few of the concrete loops that I make sure are always working smoothly. But guess what? In a lot of ways, it's the same stuff I was doing before.

Agent onboarding

Whenever I've moved to another team or product, I start by going through the onboarding docs. These often point to design docs, outline where the code is, link to our operational dashboards, deployment pipelines, customer feedback channels, and even some helpful historical documents that shaped the architecture like a few select retrospective "root cause analysis" docs. After all, I learn a lot from when things go wrong and from the decisions about what we changed as a result. Invariably though, these onboarding docs are a little stale. Maybe the commands to get things running locally aren't quite right anymore, or they were meant for how to install things on an older operating system. Maybe links are broken, or the architecture docs are out of date. I fill in the gaps by talking to teammates, and I make sure to spend some time to improve and fix those onboarding docs for the next person who joins the team. Or if I know that a new person is joining the team, I'll give my team's onboarding docs a quick pass to see if there's anything glaringly out of date.

When I start up an agent, it knows zero about the system I'm working on, how we build, how we test, what standards we have, what our roadmap is, or how to tell how things are performing in production. Every time an agent starts up, it's like I'm onboarding a new team member. When those "onboarding docs" are wrong, instead of me being able to fill in the gaps occasionally by chatting about things with teammates at lunch, there are at least tens of agent sessions that I have to start up every day and have that same conversation with to nudge it to do the right thing. That's exhausting.

So the ROI of that work has changed. Instead of "onboarding instructions" being a thing I could triage away in favor of more important and urgent work, this "team and project context" becomes one of the most important things to continuously improve. Failing to do so creates an enormous amount of frustrating work where I'm repeating myself all the time.

Quality gates

Another thing I used to do was build automatic quality gates. We'd have "push hooks" to make sure you have a clean build that passes all of the tests, checkstyle, linters, findbugs, code coverage, and whatever other scanners we could piece together to look for specific mistakes that we could tell from static analysis. And you know what? I fucking hated this. I hated getting a commit ready with all the tests passing, coverage awesome, and code reviewed by a teammate with a back-and-forth or two, to have some linter block my push because I had 161 characters on a line instead of 160. I'd have to fix it and wait another 4-24 hours for a teammate to hit the thumbs up on the new change. Such bullshit. Of course we'd try to position those checkers so they'd happen as early as possible, and ideally get used by the IDE itself to autoformat, but tuning these rules always seemed like a distraction because they were just "ergonomics", and I could tolerate a lot of, well, bullshit. It's just part of the job, and we'd have the judgement on how to balance all of the priorities.

Now, those quality gates - those deterministic annoying checkers, coverage standards, you name it - those are awesome. It's their time to shine. Agents don't care at all if they need to rework something. They thrive on feedback. They need feedback. So we task them with writing exhaustive tests using testing practices that used to be a little tedious or overkill sometimes, like property-based testing for exhaustively exploring the state space. Or mutation testing, where you intentionally break the code to make sure the tests fail. Or pen testing. There are even some pretty amazing new techniques to catch bugs in the requirements doc before any code is written using automated reasoning. These testing techniques are things we didn't always have time for before. But now these are the exact tools that the agent needs to do its job right, and I find they lead to higher quality code than before when I was doing things by hand and deciding which technique I had time to do by hand given the task at hand.

Code review

One of the hottest topics out there is on human review of code changes. It's time-consuming, and I've never met someone who loved doing it as a part of their job. But it's essential, right? How else would you know whether the agent is doing a good job or if it's going to create a bunch of slop?

Well let's talk about what your job is in a code review. Is it to find bugs? Eh, not really. Sometimes I find a bug when reviewing code, but I'm not a compiler and I'm not a unit test framework. I'm not checking out the code and finding gaps in testing and writing them myself. What I am looking for though is whether this code is taking the right approach for where we're taking the product, doesn't miss some important design consideration, or blurs the encapsulation of some part of the code in a way that will be hard to maintain later. If we're not reviewing the code, how do we avoid vibecoded slop?

Well at first, you do review the code. But every comment in a code review is a defect. Not a defect in the code that was written, but a defect in the codified rules enforced by auto-reviewers. If we want code to be styled a certain way, add a linter. Add a team of review agents that look for problems or a failure to enforce best practices, like security, resiliency, usability, you name it. If we want to make it so you can't break encapsulation, add agent steering to your review agent to check for that. And then of course add a gate that prevents the agent from claiming that it's done until it has addressed all of the auto feedback, or if it raises its hand for a human because it thinks the feedback is unreasonable or points to a larger problem in the requirements.

There's still plenty of human specification and review that needs to happen. Designs need to be deliberated between you and the agent and even as a team. Sometimes for a task you do want to thoroughly review the code yourself. Great - have the agent wait for you for that task. If there's a part of the code that you always want detailed human review of, add some kind of agent or workflow rule that requires human approval when that part of the code is changed, like the authorization code or the replication algorithm code, or whatever your most thoughtful business logic is.

And if you still need to review code for some kind of policy reason, then 1) challenge the specific need and focus on the outcome that the requirement is actually getting at, and 2) consider when the review needs to happen. In my design loop where I'm going back-and-forth with the agent early in a task, I have it describe the interfaces that it's going to change and what they'll look like. I know people talk about "shift-left" for testing, but reviewing code before it merges to mainline is such a slowdown in throughput. Maybe some code review should "shift-right", and happen after integration testing instead. And 3), consider what you actually want to look at when you review. If in my feature definition or design feedback I said I wanted the UI to look a certain way or the implementation to be done a certain way, I might want to focus my attention on those areas. If the coding agent found new things that needed to happen that it didn't talk about in the design, then I need that called out so I can see if it made changes that I was happy with.

Independent, thorough, fast test cycles

Another thing I used to do when I'd join a team is set up a version of our service running entirely on my development machine. This was even my first task when I joined amazon.com 20 years ago: get the web server called "Gurupa" to run on your developer desktop under my desk. To do that I had to learn all the tools and how to troubleshoot it. It taught me a lot about how the system worked. But other than learning, why did I need to set this up? So that cycle time of coding to testing would be faster. Who wants to wait hours for their tests to run before they find out if their change breaks everything? And you especially want those tests to run before you push the changes and break the shared "devo stack". Instead of that, it was important to be able to run the whole system - often many microservices - together on my developer desktop. And for more complex testing, to have my own deployed stack in my own AWS account where I could do end-to-end testing without risking "breaking mainline" or "breaking devo".

But sometimes I'd join a team who let this "local stack" hygiene slip. So when I'd join, I'd tidy this up and get it running locally. But then over time, I admit sometimes I'd let this slip on my own service, or it'd get too complicated to have a perfect simulated version of every dependency.

But now, having great local running versions is absolutely critical. Agents move so fast and push changes so often that the cost of a regression in mainline is a huge problem that stalls progress. If an agent has to wait an hour for their tests to run on every iteration, it'll still take forever to get anything done (even though you can get a bunch done in parallel).

Fortunately, it's also way easier than before to build high-quality simulators of your dependencies and wire it all up locally. It's a relatively simple task for a coding agent. Sometimes it requires a little code refactoring to make it happen, but it's easy. So do it!

I spend a lot of my time making sure my coding agents have a setup where they can run high-fidelity tests locally (or at least pre-merge in a dedicated stack). And for tests that can't happen there, I focus on high-quality automated testing in a post-merge, pre-production environment, and building the machinery to automatically address failures.

The more things change…

In the world of frontier engineering, I'm not really doing new things that I wasn't doing before; I just spend a ton of my time on things that I used to do only occasionally. The economics have changed; instead of focusing on new team member onboarding docs every few months, I update them daily. Instead of begrudgingly accepting style checkers to gate changes in my codebases, I embrace them. Instead of fixing flaky tests begrudgingly, I do so urgently (ok but still a little begrudgingly).

Some of these tasks aren't as glamorous or endorphin-generating as coding was. They don't always activate the same part of the brain as before. Some of these were tasks that languished on backlogs not only because "we weren't given enough time to do them" (I'll leave a rant about ownership for another day), but also because they were boring. Sure today I'm still doing that inventing, designing, researching, and exploring, but proportionally more of my time is spent on tasks that feel more like eating my vegetables.

I already had this "part 2" sketched out when I posted part 1, so that didn't leave me a lot of wiggle room to bring in the ideas that folks shared with me after part 1. So that means I now have to sign up for a part 3 to actually do that. There have been some really insightful things that folks have told me about, from things that worry them about the future, to some angles to what we can learn from cognitive science and psychology in Cat Hicks' What helps an identity crisis?, to thinking about how job role definition can help or hurt as things change, with Annie Vella's talk called The Specialization Trap. I don't know what shape part 3 will take yet, but it'll probably talk about this kind of thing.

What am I missing here? What do you do now that you weren't doing before? What are you still doing, but where you've found a new way to do it more easily or more effectively? Or what hasn't changed at all? What parts aren't solved yet that need something to change? And in lieu of a Dune-universe-like revolution against 'thinking machines', what should we do about it?

"With the AI doing the coding, do we just spend all day reviewing its output?" The rapid onset of impressively capable coding agents continues to raise this question in conversations with peers, to press interviews, to podcasts. I remain an optimist, convinced that this is the magical technology I've always wanted - a way to get more done, in less time, with higher polish than before. I can reach further into my backlog than ever before, reaching for the nice-to-have tasks that were triaged out of every sprint plan.

While I used to spend a lot of time interacting directly with code - writing, reviewing, debugging, fixing, refactoring, and operating (dealing with the consequences of mistakes), I have always viewed that as only part of my job: a means to an end, not the end. So then what is my job?

What's in a name?

When I tell people what I do, sometimes I say "programmer" to have a short, understandable, and non-pretentious sounding description of my job to people who don't code. But that's not my job title. The thought of being called a "coder" somehow feels almost derogatory, because of how shallow or superficial it is as a description of what I do. It's one of the things I was hired to do, but it's not the full picture at all, often not the hard part, and not my job title. So what is my job?

What about Software Architect? Well I'm not a Software Architect. I love hearing Real Architects describe the concepts around buildings - their design, aesthetic, married with function and usability for the people who occupy them, with adaptability toward future unknown use. My friend of 23 years and college roommate for 3 of those, Steve, is an architect, and whenever we are near or in a building, I love it when he goes on an explanation of the things about buildings that are not obvious to a non-architect like me. He describes the customer - the people in or around the building - and how the design is meant to be intuitive, helpful, but grounded in the reality of the tradeoffs of physics and cost to build. A lot of the Architect title sounds like what I do day to day when it comes to designing systems and how they meet my customer's needs, but still, Software Architect isn't my job.

Some people describe the job as Software Engineering. While I'm not technically an engineer from a professional license standpoint, there are plenty of parallels to what real engineers do. For example, Civil Engineers don't build bridges. They design, they plan, they model, they test, they measure, they feel the weight of responsibility for safety and resilience, scale, and they reflect on the process that they go through along the way to make sure it's right. That same friend Steve also has a Civil Engineering degree in addition to being an Architect, which is a hell of a combination. With both architecture and engineering degrees, he works across both worlds: the artistic, focusing on the beauty of design and the lives of the people who occupy the buildings, and also the unglamorous but life-safety behind-the-scenes reality of what it takes to prove that a design can be brought into reality. While I can draw connections to the testing, simulation, and planning work I do in the digital world, Software Engineer is not my job either.

So what is my job? My job title is Software Development Engineer. Isn't that the same thing as Software Engineer? Well it has an extra word in there, so I guess not. That extra word, "Development", is a process. The "Engineer" term is a modifier on "Development". So that means I'm engineering the process of software development. But that sounds like a pedantic unimportant difference, like Michael Scott correcting Dwight Schrute's title as being Assistant TO the Regional Manager. But if I think about what I've been doing for the last 20 years, the work I've done around process has been more lasting and impactful than the code. For a given project, I'm working on design, implementation, testing, and operations. But the most impactful and durable parts of that work are the in-betweens where I'm making those things easier the next time around. If something didn't go quite right the first time around, I make sure that I, and my team, or organization, doesn't make that same mistake or hit that roadblock the next time around. Essentially, when instead of engineering the software, I'm engineering the process of engineering the software.

Of course this distinction of "SDE" vs "SWE" is a convenient play on words. Even though role names and responsibility vary across companies, I think these roles are actually equivalent and all focus on all of these aspects, from designing to building, to maintaining. And all throughout that, people in these roles spend time and effort trying to improve their process to take things they've learned and somehow "codify" for themselves and for their teams so that things are better the next time around. (Wait, the word "codify" also looks similar to "coding"! Foreshadowing!)

What does software development look like today?

Building software today is a lot like an automated factory. An idea gets turned into a requirements and design, which gets turned into code, which gets tested, merged, and deployed. But it's a little different than a linear assembly line, where a conveyor belt moves a product (or in this case a "feature") from one stage to the next. Like a conveyor belt style, this one has tons of quality gates. When a feature, design choice, or important decision needs to be addressed by a human, the feature is set aside until someone answers, and then the feature resumes its path on the assembly line. In the meantime, the rest of the features move forward on the assembly line. But maybe unlike an assembly line, this one has loops. The in-progress code change gets sent back for rework a ton of times until the tests pass and all of the quality gates approve it. It also has merges where features (commits) come together into the mainline branch, and they're further tested and released together. If these tests fail, there's a different loop in the assembly line that figures out what to do about the failure. It triages, and figures out which commit broke the tests or the deployment, and finds the best way to unblock the assembly line. It might either rush a patch while more changes pile up, or find and kick out the offending change and let the rest of the feature development advance.

In a sense, this is a lot like a normal agentic loop. In a nutshell, agents are super simple - a while() loop that tries something, sees if it's done, and if not, tries something a little different. But it's more complicated than that. We can't just take a coding agent and say, "Hey design and build this feature. Make no mistakes." The agent won't follow my rules, might make decisions or assumptions I don't want it to make, could break my application, or any number of things. So this factory is all about having the right deterministic checks and gates in place so that it can't proceed until its work is correct (verified by tests and testing agents, and sometimes humans), and done the way I want it (verified by agents and sometimes humans).

The machinery to drive the factory is rapidly changing, with new tools sprouting up every day. It seems like every 3rd person I meet has made their own. There are a ton of open source ones out there, from Gas Town to more unopinionated building-block tools like Kiro Crew.

So where do I fit into this as an SDE? When coding agents first came out, it started with me pressing "yes, approve this tool use", and copy-pasting error messages from one tool to the next. Sure it was an improvement over doing everything myself sometimes, compared to copy-pasting the error into stack overflow and fixing the error myself. But it wasn't a huge step function multiplier advancement in productivity.

To get that big improvement, I - and my team - had to change from babysitting the agents to nurturing them. We spend our time focusing on building and tuning the machinery, so that we can be in the way of the assembly line only when we truly want and need to be to make sure the right things happen.

Unlike physical factories that make physical things, software factories are easy to change. There's no need to move heavy machinery around the factory floor. Improving the factory can be as simple as adding a bullet point to an AGENTS.md file. Or better yet, having an agent run periodically to find rough spots in the process from looking at past runs, and propose changes to the factory code and configuration to improve it.

So then what is my job?

Every day I'm defining, designing, evaluating, and shipping things for my customers. I have more time than before to listen to their needs, respond to their questions. I have more time to be curious and dig in about the operational performance of my systems, and look around corners. But the biggest flywheel comes from continuously improving the software factory.

Clare Liguori wrote the Frontier engineering manifesto that breaks down how we at Amazon approach software development in this new world. It describes a shift in mindset and effort, toward focusing on the process of making sure the coding agents are held to the standards that you set for yourself. It's a good read. Principal Engineers across Amazon weighed in on it, but Clare is the mastermind who brought it all together in this compact and dense form. Here's a short version.

• You are the architect, not the typist. Write intent, direction, and targeted review of what you want.
• Maximize agent time, minimize your involvement. Rethink when you need to be involved in a coding agent's session. Challenge yourself to have multiple sessions running in parallel, and structure their tasks so they won't constantly nag.
• Build your codebase for agents. A new human team member onboards to your team once and maybe every few months or years. With agents, you have team members onboarding 1,000 times a day.
• Give agents a fast feedback loop. Give your agents the tools they need to run fast, local tests that are as end-to-end as possible. Give the agent a web browser tool to do its own evaluation of the interfaces it touches.
• Execution is cheap. Direction is everything. Collaborate back and forth with the agent at the important points, like on design and definition.
• Treat code as disposable. It's way easier to discard and replace than ever before.
• Hold AI output to human standards. With a well designed factory, you can get higher quality code than before.
• Trust the boundaries, not the agent. Put the agent in a sandbox with a thoughtful boundary.
• Use agents for everything, not just code. I even used an agent to find my lost coffee mug, which was missing for weeks.
• Continuously tune your agent setup. I view every steering I have to do as a possible defect in my factory.

As a Software Development Engineer, I build and tune the factory. I add review agents and improve their steering, direct the agent to write better tests, review their output (but strategically at just the right time and just the right way), look at the process to see where I'm wasting time or where the agent is getting stuck. But what does this mean? The Frontier engineering manifesto describes the types of things I spend time on at a high-level, but for blog form, check out part two!

If Angine de Poitrine hasn't invaded your Instagram feed yet, then by clicking on any of these clips, it will start following you around the internet. They're a guitar, bass, and drum duo with a style that balances jam-band rock-out fun with more cerebral, less-head-bobbable elements. Internet people call them "microtonal math rock". "Microtonal" refers to the fact that their guitar and bass has extra frets which lets them play notes that aren't expressible on western sheet music. "Math rock" is a genre that involves complex shifting time signatures, like Dream Theater.

In this post I'm trying to unpack why I'm so obsessed with Angine de Poitrine. I've played bass guitar in jam bands off and on over the years, but my regular instrument is French horn. On horn I've played pop tunes in marching band, and have also "played the hits" of the last few hundred years of orchestral repertoire, currently in the Puget Sound Symphony Orchestra in Seattle. For some reason, Angine de Poitrine activates my classical music brain more than my bass guitar brain, and I'm trying to figure out why. So hopefully this blog-post-turned-first-semester-undergrad-music-major-term-paper helps explain why, and maybe therapeutically helps me get this amazing song out of my head for at least a little while.

So buckle up for an overdone explanation of Angine de Poitrine's song Fabienk. I called in for some classical music knowledge help from my brother-in-law-and-professional-orchestra-conductor Edward Leonard, so we'll see if I can get a C- or better on the term paper.

The machinery

YouTube is flooded with explanations of how Angine de Poitrine's songs "work", like by David Bruce Composer who breaks down the beats, or Davie504 who is a funny bassist whose channel I've subscribed to for awhile. It's probably silly to try to do a walkthrough of this in written form, but I'm going to try anyway.

Loopers, phrases, and time signatures

To understand how it works, I should first describe one of the duo's structural constraints - the looper.

There are only two musicians in the band, and yet they have tons of instruments playing at once. This is achieved using a looper - a thing the guitar plugs into that can record on the fly and play back on repeat. The loop is a fixed duration. In the case of Fabienk, it's 14 quarter notes long, at a steady tempo of quarter note at 128 beats per minute. So how does that result in such a unique sound?

The "14 quarter notes" causes an interesting musical effect right away when it comes to phrasing. A "phrase" in music is a connected melody that contains an idea. Like in the alphabet song, "abcdefghijklmnop" forms one phrase. You take a breath at the end, before you move onto the next phrase. When you write out the music for that song, you break up the phrase into "measures" so it's easier to read. The alphabet song is in 4/4 which means there are 4 beats per measure. The beat is a quarter note, which in this case "abcdef" are quarter notes of one beat each, and "g" takes up two beats. So "abcdefg" takes up eight beats, or two full measures. I'll write this out where each character is a beat represented as either a | or a -, where | is the first beat in a measure and - is another beat in the measure. I put "v" on top for each beat because the | and - difference was looking a little confusing. So the alphabet song looks like this:

vvvvvvvvvvvvv
|---|---|---|
abcdefg hijkl - hmm

Wait how do we write out lmno? They're faster! Well every beat can actually be broken down into two. Half of a quarter is "1/8" so these are eighth notes. Here's that written out, "subdivided"

v v v v v v v v v v v v v v v v
|-------|-------|-------|-------
a b c d e f g   h i j k lmnop   

There are 16 v's so 16 beats in this alphabet phrase. That's a "normal" phrase that is symmetric. The two halves of the phrase are the same length (8 beats), and there's some balance there that is very familiar. The short break after "g" is the same length as the short length after "p". This phrase ending is clear and leaves you room to take a breath (literally if you're playing a wind instrument). Side note: musicians will sometimes chop phrases in half, making the music sound clumsy and choppy. If you leave the same space after "g" as you do "p", this chops the phrase. But if instead you sing the "g" for longer, sustaining it into the break, then it makes for a nice phrase that builds in the middle and then backs off at the end.

PHRASE ONE                      PHRASE TWO
v v v v v v v v v v v v v v v v v v v v 
|-------|-------|-------|-------|-------|
a b c d e f g   h i j k lmnop   Q R S   ...

Where this gets weird in Angine de Poitrine. The looper pattern is 14 beats long, so doesn't really repeat in as natural of a way because it's short by two beats. That makes it so the alphabet song doesn't have room for the letter "p" before the next phrase begins with "q"! You can change the tune to make room for it, but if you keep the same 14 count phrase you'll eventually run out of runway at the end.

PHRASE ONE                  PHRASE TWO                  PHRASE THREE
v v v v v v v v v v v v v v v v v v v v v v v v v v v v v
|-------|-------|-------|---|-------|-------|-------|---| ... 
a b c d e f g   h i j k lmnoQ R S   T U V   W   X   Y   &
                            ^ what happened to "p"!?    ^
                                                        ^
                               "and" is supposed to go here,
                            but that's into the next phrase
							                             

The weird phrase length is just one of the oddities. The other is that the loop is broken into different time signatures in different parts of the same song. At the start of the song for example, the measures are 7 beats long, where the beat is twice as fast (eighth notes).

First part:       |------|------|------|------   (4 measures of 7/8)
Rock out section: |-------|-------|-------|---   (3 measures of 4/4 and 1 measure of 2/4)

One thing that makes 7/8 is it makes it hard to tap your foot to. The way musicians tend to think about this is to split the measure into 3 beats, but make one of the beats take longer than the others. For example, you might count it like this. Every time you say "one", that's when you tap your foot or bob your head.

v v v  v v v  ...
|------|------|------|------
12121231212123...

Depending on what's happening in the music, you'll change out which of the 3 "beats" is the long one - choosing between 123-12-12 or 12-123-12 or 12-12-123.

Ok whatever. The point is 7/8 is hard to dance or walk to. At a rock concert, it could be awkward to jump up or pound your fist in the air on the beat. The beats are different lengths so you feel like a huge nerd if you try to line it up to the music.

The cool part about the layering that Angine de Poitrine does is that different time signature loops are still compatible with each other. A 4x7/8 loop can play at the same time as a 4-4-4-2 loop because they're all 14 counts long. Or to make it easier to think about with the 7/8 meter - they're always 28 eighth notes long, they're just in different time signatures. So when they blend, they sound SUPER weird and cool.

For example, in the first half, the layering looks like this - 7/8 layered beneath 4-4-4-2. You can hear that in this part of the song. The new melody (the part Khn is playing in real-time on the guitar), and the drums are now in the 4-4-4-2 pattern. But all that cool fast guitar noodling is still present from the previous section, and is still in 7/8! They interact in a very cool way.

|------|------|------|------ (this is the weird syncopated part at the beginning)
|-------|-------|-------|--- (this is the distinct melody in the jam sections)

In the second half, there's no 7/8 layer anymore (at least that I could tell), so it's much more of a steady "rock out", but with the awkwardness of being two bars short of an "alphabet song" style typical phrase. But, there are two different groupings that layer and still make it sound fun. From how I hear it, the solo bass line is 4-4-2-4, whereas that melody that comes back from the first half is still 4-4-4-2. They sneak in that half measure in different places in the phrase, but they still line up when they turn around to the beginning on each loop.

|-------|-------|---|------- (this is the sick baseline that starts the second half)
|-------|-------|-------|--- (here's when the melody from the first half comes back)

Microtones

The second part of unique machinery are microtones. The guitar and bass have extra frets that are closer together than typical guitar and bass. This lets them play in a way that sounds way different than we're accustomed to in mainstream music. It's likely that most music you hear uses notes placed on the "standard western semitone" scale. In this system, you have notes named A, B, C, D, E, F, and G. But there are more than 7 notes. When you put a flat (labeled b) or a sharp (labeled #, which in this case is neither a hashtag, pound sign, number sign, nor a hash sign) in front of these notes, you get a note that's halfway between that note and an adjacent note. For example, B-flat is halfway between A and B. Quarter notes, which I've never seen written, are half-way between a note and its next half step. And yes I know that some notes don't have a half step in between them, I'm just simplifying this because that isn't important here. Do not @ me.

If you're a math person, you might say, "hey aren't there an infinite number of notes between two notes"? After all, the mean value theorem proves that there are. And funnily enough, that pedantic math observation is meaningful here. There are music systems all over the world that define notes in completely different ways. I'm having to google them in order to write this so I'm no expert, but there are a ton - Arabic and Persian Maqam systems, Indian Shruti and Swara, and tons of variations throughout Africa and East Asia.

I don't know what microtone system that Angine de Poitrine guitarist "Khn" uses, but he can play a lot of notes that sound surprising. I noticed sometimes he uses them in a way that sounds like a "bend", where a guitarist bends the string to the side to make it lean a little higher. But instead of bending, he just plays a more precise in-between, sometimes as a passing note just like how a bend takes you slowly between notes but eventually making its way up to the next one. The other way they use the microtones happens more in other songs like Mata Zyklek. This makes me think of music from an Arabic, Indian or East Asian system, but I don't know anything about them other than that they sound cool.

The play-by-play

The beginning of Fabienk has this hyper-syncopated, ascending chromatic line with a fast flutter thing to wind it back to the start of the pattern. Even the drums join in and play the same weird syncopated beat in unison. It's not a melody like from a song you'd hear on the radio, but it's not so abstract that you can't even hum it. If you listen to it 100 times a day like me, you start to feel it. But at first you can't nod your head to it or tap your foot to it because the beat is so hard to find. But when I stop thinking about the notes and the beat, and just think of the tune, then I can get into a trance irregularly head bob where I anticipate where each of the little riffs go.

Next, just when you start to feel comfortable in this already unsettling rhythm: chaos. Everything you thought you understood about the melody gets flipped on its head. The bass enters, and the drums switch to a groove that is very unexpected and difficult to understand. I could not really figure this out, so I had David Bruce Composer's YouTube video explain it to me. It turns out the melody from the very beginning actually had an even more bizarre syncopation than you thought; it's happening ONE SIXTEENTH NOTE later than you think. WHAT IS HAPPENING? This is chaos. Your mind cannot comprehend this chaos. Your only hold on reality is grounded by the melodic fragments that you have been hearing the whole time - they're just not happening when you think they should. What already sounded really hard to play individually and even harder together, I realized is twice as hard. I thought maybe they do some kind of "shift pattern by 1/16th" trick with the looper, but I seriously doubt it. They're super talented musicians.

This chaotic 7/8 syncopation repeats and builds, until you hear their weird alien scream, and then suddenly - relief. You are being rescued by the triumphant introduction of a new melody and new subdivision of the 28 1/8th note loop - a 4-4-4-2 pattern.

I've been trying to pin down why this melody has such a triumphant feeling, like a heroic melody that's here to save the day from the syncopated chaos. But in writing this I think I figured it out. It sounds a lot like the Battlestar Galactica melody. Which, now that I think about it, also has this layered "canonic" style. If you don't believe me, hum the Battlestar melody on top of where the guitar starts, and you'll see that they line up on the same 3 8th notes. Coincidence???

Or better yet, I went ahead and messed with pitch and speed in Ableton and then had my coding agent whip up a nice little web control for you to listen to it yourself:

🎛️ This post has an interactive tool here for blending and pitch-shifting Fabienk against Battlestar Galactica — RSS feeds can't run the audio control, so check it out on the original post.

The satisfying jam loops and builds, but at its peak, the chaotic 7/8 returns. At least this time it's with a less syncopated drum pattern so it's easier to follow. And then - a break. Literally a break - the song pauses for a moment.

Out of the silence, a SICK bass line. After I write this maybe I'll finally dust off my bass to learn this lick. The bass drum beats a repeating quarter note, keeping the beat grounded but not helping you parse the bass line. The first many times I listened to this, I thought the time signature was all over the place because the bass line felt like a set of short thoughts of a few notes each. Over time I realized it's just a 4-4-2-4. Similar to the last groove, but the chopped 2/4 measure is in the middle of the sequence instead of at the end.

Then more of the ridiculousness I'm here for - the aliens do this arm wave thing that reminds me of the wiggly arms the characters have in Adventure Time. I never watched the show and only know it through memes. Who knows if that's what inspired them to do this.

The layers build, giving you more sequences to shift your mind in and out of, and then: the return of the triumphant melody again! What a cool restatement of the melody with a totally different underpinning. Unlike the first half that had a layer of the 4x7/8 sequences, this stacks the 4-4-4-2 melody on top of the 4-4-2-4 bass line. It's less "busy" sounding this time around as a result. And then the guitar adds a harmony on top of itself in a twin-lead, that reminds me of guitar solos in metal songs like Metallica's Master of Puppets. And the microtones start to come in too, but they feel natural because they sound like guitars do when they're purposely bending between notes.

The song ends with the hilarious, weird, distorted alien voice singing thing, which seems perfect. Funny enough they even do the voice thing during interviews apparently, like in this one, where their manager did most of the talking but they spoke for themselves some in their alien squawk language. That's the kind of hijinks I'm here for.

Classical comparisons

The first thing about Fabienk that activated my "classical" brain was how it pieces together "micro melodies". These are melodic snippets, sometimes only 3 notes long that string together but are less connected than an approachable phrase. It's not a lyrics-first song like would play on the radio and get stuck in your head, and yet it's still catchy enough to hum to.

One of the mechanics at play here is that it flips between time signatures, known as "mixed meter" - the concept I really beat to death in the machinery section. It's a style that's used in a bunch of music that I play in orchestra, so it got me making connections between Angine de Poitrine's music and other classical pieces I've played.

One of the early composers to break from the familiar, steady time signatures, was Stravinsky. One of the more famous pieces is Rite of Spring. Notice how chaotic that is. It uses mixed meter, dissonance (clashing notes), and there's often no lyrical melody. Instead of lyrical melody, he uses cells, which are a small group of a few notes, that he stitches together in what is called "block form". (No I didn't know those terms before, I was just searching for what they're actually called for this blog post.) And yet it can still get into your head. This was so radical and provocative at the time in 1913, that when this piece (accompanying a ballet) was premiered, the audience rioted - yelling, throwing stuff at the stage, and even fighting each other until the police came.

This historical anecdote might sound smart if you're not an orchestral musician, but the truth is that I don't have a real music history education. For all I know, to "real" orchestra nerds, this Rite of Spring comparison could be a pretty lukewarm, cliche take. So I thought I'd get a more informed opinion from a professional musician, so I texted my brother-in-law-and-professional-orchestra-conductor Edward Leonard, who is founder and Music Director of the Chamber Orchestra of Pittsburgh and has been guest conductor with orchestras around the US.

And sure enough, Angine de Poitrine had already invaded Edward's Instagram too, and he had some ideas! I asked Edward about which classical composers or genres Angine de Poitrine resembles.

For my Stravinsky take, Edward pointed me more toward The Soldier's Tale as a closer comparison, which I hadn't heard before. And sure enough - there are lots of bursts of melodies in there. Sometimes there are steady "walking" rhythms (I guess the soldier is marching or something) and other times it's unpredictable and irregular. The narration threw me - I don't think I've been to a narrated performance before.

One comparison Edward drew is to Bach. He called Angine de Poitrine "canonic", since it repeats over and over with evolving variations. So it's kind of like Pachelbel's Canon in D, which you might have heard at weddings, or learned if you play piano (at least those are my own relationships to the piece). It's made up of a bunch of layers of repeating lines that are introduced one by one and stack on top of each other throughout the piece.

Edward pointed me to Steve Reich's Clapping Music as an interesting example of this layering. It involves two people, each clapping a different rhythm at the same time. One part repeats exactly, while the other one morphs throughout the piece. Sounds super hard to do.

Edward also drew a similarity to minimalism. And wow that makes a lot of sense. A Los Angeles Chamber Orchestra blog post defines minimalism succinctly:

..."minimalism" refers to music that often features short phrases repeated a large number of times or that may gradually go through a process of change or transformation. There is routinely a steady pulse in pieces like these, and there is not necessarily a harmonic goal, so the music becomes more about the process rather than traditional harmonic language, or narrative or representative stories.

My favorite minimalist composer is John Adams, who has been described as a "minimalist bored with minimalism". I first encountered Adams in PSSO where we performed Short Ride in a Fast Machine and The Chairman Dances. We got through it ok, but when I heard the Seattle Symphony (the pros) perform Short Ride, my jaw was on the floor the entire 5 minutes. (Here's BBC Symphony Orchestra performing it). Something about this genre of music makes my brain very happy - which now makes total sense that I'd be drawn to Angine de Poitrine.

Like Fabienk, Short Ride is deceptively complicated to play. At times some instruments are playing essentially in a different time signature than others, and frequently with different syncopation than the others. And the melodic snippets or phrases can be short and not particularly intuitive or memorable. In theory us musicians deal with complex rhythms by being super precise, subdividing measures, and anticipating an exact entrance according to the math. But I admit that with complex syncopations and cells like in Short Ride, I end up getting through it because I am in the "groove", putting the little "riffs" in the right places to fit in with the orchestra. That might be the point where the collection of notes turn into music. It's that same feeling in Fabienk where these micro "licks" turn into melodies even though they're not lyrical in a traditional way. I guess they're also using block form!

So yeah, I see why Edward labeled Fabienk by Angine de Poitrine as minimalistic. Steady pulse? Check. It's driven by a pulse in both in the hyper-syncopated parts, and in the rock-out sections, just like the wood block in Short Ride in a Fast Machine. Short phrases gradually transforming? In a sense. It's cool how that original Battlestar Galactica-esque "triumph melody" returns in the second half on top of a different foundation of the new bass groove, vs the first half's different layering.

Conclusion

There's a lot going on in Angine de Poitrine's songs. The musicians Khn and Klek are extremely talented and creative. Parts of their songs are super cerebral and takes a lot of concentration to follow what's going on. But you can also turn off your brain and just enjoy the emergent complexity that rises from layering of loops, even more complex sounding as they layer different time signatures on top of each other. But they don't only stick to that complex stuff. They also pay off your concentration on the more complex parts by rewarding you with that fun jam-band sound that rocks out.

If you think it's neat to unpack music and find its hidden complexity or intent, check out this deep dive on The Matrix score on the Sideways channel. It totally blew my mind to learn about what the music was doing to me subconsciously while I was watching this amazing movie. That breakdown was actually a retelling of a PhD dissertation by Christopher James Heckman Taking the Red Pill: An Analysis of Don Davis' Score for The Matrix Volume. So check that out the long-form if you want to see how deep the rabbit hole goes.

And yes I have tickets to see them in Seattle in December. They'll be here this August too, but I'm not a huge music festival person, so I'll wait where I can sit down in chairs. Now that I think about it though, I bet I'll end up with FOMO for being down on the floor with everyone who's dancing and getting into the band's super fun antics, from this triangle symbol, to doing squats for some reason. Of all the concert video clips that sail by on my Instagram feed, and in all of the subreddits I flip through, the Angine de Poitrine following seems wholesome and fun. I love how things that are fun and goofy bring out the best in people and attract friendly people, from the multitude of online communities that have popped up around my favorite Star Trek podcast, The Greatest Generation, to the Phish-like following that Angine de Poitrine is starting to spin up. It's fun to be around people who are both having fun, and who want you to have fun too.

The conversation starter was on the last 20 years of AWS. I just passed my official "20 years at Amazon" milestone last week, which depending on how I exactly count it, was anywhere between 15-17.5 years at AWS, and the rest working for Amazon.com where I automated operations of the web server fleet.

It's amazing to think about how much of the stuff I've done over the last 17.5 years at AWS have been motivated by stuff that I found annoying while working on that first team. Scaling forecasting, code deployments, database operations, server operations, incident management, production service troubleshooting and observability, and even annoying client/server communication and SDKs (or a lack thereof).

We talked about making AWS' web service framework (think gRPC), and picking an abstraction that is easy to use out of the box, but not overbearing or restricting. Jason likened that design goal to Grafana, which lets you have a bunch of easy dashboards, but also lets you customize to your heart's content.

It was funny when I brought up the weekly AWS ops meeting, where Jason stopped me and asked for clarification: "...wait what do you mean everybody in AWS gets together?" The weekly ops meeting has changed shape a little bit from time to time, but it does involve representation from all service teams in AWS sharing what we've learned about ops with each other, from victories, to hard learned lessons.

We also talked a lot about testing. I shared how Kiro helps me with property-based testing and spec-driven development to make sure the agent stays on track and course corrects - essentially how tests help agents play bumper-bowling instead of regular bowling. Jason had a lot to share about testing too, and how he got coding agents to be more effective when he prompted it to do mutation testing instead of standard unit test boundary-case testing. Jason knows a lot about testing, and has his own testing guide The Beginners Guide to Rails Testing.

Anyway it was a fun chat Jason, thanks for having me on your podcast!

I'm going to start having more fun with ridiculous edits. This time I went with a digital jump cut. I asked my coworker if it ended up zoomed enough for comedy, and he said no, I had to zoom in more. How's this, Dylan?

I love bowling. I was never incredible at it, but I learned how to throw a sort of underhand spin technique in high school, so at least it looks cool. You throw it diagonally to the right, but put spin on it so it curls back to the left instead of ending up in the gutter. But it's pretty volatile. Sometimes it doesn't spin enough and slides into the right gutter, or it spins too much and it ends up in the left gutter. But when you do it right, the weird angle that it hits the lead pin seems to cause more helpful chaos than straight on, and more pins to fall.

I was putting a talk together about Frontier Agents. Frontier agents run for hours or days without intervention, learn as they go, and behave as a team member would with the same context and tools. I work on AWS DevOps Agent, which is always-on, waiting for an alarm to fire to root cause it, and regularly scanning in the background for things it can notice about your application infrastructure that needs improving.

In the talk I was trying to describe the properties of an agent that makes it autonomous. There is a lot to the frontier agents that makes them powerful, from learning, to guardrails, to parallelism, but the key ingredient I was focusing on in the talk was deterministic feedback in the agentic loop.

The agentic loop is the while (!isDone()) { keepTrying() } algorithm that drives agents. They are given a goal, and then they keep using reasoning from the LLM, tool calls to interact with the outside world, and some logic to tell them when they're off course or if they've reached the goal. Sometimes the LLM gets to decide when it's done. The Claude How the agent loop works article says that once the LLM returns no tool calls for the next iteration (or when it uses all of its budget), it means it has reached its goal or is done.

As a starting point this makes sense as an agent, but it's an "open loop" style of agent. The LLM decides that it has reached the goal. Hopefully it didn't comment out the unit tests to decide that they pass and it's time to check in code! When we build frontier agents, we build a harness for the agents that keeps them from wandering off like a Ouija board, or coming to an incorrect conclusion. I like to think of this as "bumper bowling" for the agents. Instead of lining the bowling alley lane with gutters, give them bumpers to push the ball back on track!

In essence, this adds a step to the while loop to prevent the program from declaring success or from picking a weird direction to wander into. There are plenty of techniques for this, but we like to use deterministic code for this. One simple class of bumpers are permissions checks. If the agent isn't allowed to call a tool a certain way - either as a server-side check by the receiving end of a tool call, or as a harness-checked block where the agent asks for approval to call a tool and you say "no" - the agent hits the bumper and changes its direction.

The most sophisticated check I've seen so far is a new feature added to Kiro's spec-driven development called Requirements Analysis. Spec-driven development is a mode in Kiro where you start a task by working through requirements, design, and a task list. Even without Requirements Analysis, spec-driven development acts as bumper bowling, because it helps the agent break the larger project down into pieces and gives it clear requirements to write tests against. This pairs very well with property-based testing frameworks, which write tests that drive comprehensive ranges of inputs and validate invariants along the way. For example if you're building a traffic light system, you'd want to guarantee the invariant of "at most one direction is green at a time", and run all possible sequences and state changes through to validate that.

The new Requirements Analysis feature takes this a step further to help avoid bugs that creep in at the beginning when you're defining requirements. It checks for logical inconsistencies - cases where requirements conflict with each other, and where there is ambiguity where things aren't fully specified. When I ran it the other day, it noticed that I hadn't specified certain behavior in an edge case: whether to update an IoT Shadow after rejecting a state change earlier in the algorithm.

Requirements Analysis uses some big-brain stuff to create powerful bumper bowling guardrails. It uses neuro-symbolic techniques to encode the requirements into a formal model, using SMT solvers to find logical inconsistencies. This blog post goes deep into the implementation, discussing topics like formal verification, automated reasoning, auto-formalization, and semantic diffs. It's impressively deep stuff, but applied in a way that fits naturally into the coding flow.

And if you're wondering about whether I generated these bowling alley images, the answer is no. I enjoy making this pixel art stuff. It looks just bad enough to be good. The lines are squiggly because I draw with my mouse, because using a pen on tablet would look too good. The lines are pixelated because I turn off anti-aliasing. Was it worth the 2–3 hours I spent drawing the bowling alley? Absolutely not.

Was it fun? Yes! And I find that over time I build up a sort of "asset pack" of graphics, so putting a talk together gets easier each time. If I need a pixel art magnifying glass, I have one from a couple years ago. A picture of a person in a desert searching for something buried? Yep, have that one ready too.

Anyway, next time you design an agent, think of bumper bowling. Don't let the agent wander off, claim it's done when it's not, or make incorrect decisions. Keep your agent out of the gutter!

Corey's AWS banter has always been on point, and hilarious. He pokes fun at where AWS is rough around the edges, but out of a desire to help shape it.

My favorite clip of Corey remains this 2019 interview at re:Invent, where he says, "The AWS product strategy remains what it has been, and that is simply 'yes'." (With sarcastic airquotes, of course.) He continues, "If you'd asked me to predict what they would have released at Midnight Madness, I would have had several guesses, none of which would have been, 'Well it's a piano keyboard thing that also does machine learning'. And my followup would be, 'Well of course it is! Does it also make fries?'"

So when I heard I'd be on Corey's podcast Screaming in the Cloud, I was starstruck, and nervous, just wanting to do my best to be on something I've been following for so long.

I was doing my best to keep up with his rapid-fire wit! Like when I said I still carry a pager 📟, he said, "As an Amazonian, I assume you carry six pagers 📜". (The joke, if you're not an Amazon dork like me)

Check out the link for links to YouTube for video, or all of the podcast platforms for audio, and for a transcript.

In last week's Dune as an operational model post, I combined two of my obsessions — Dune and Ops — into one allegory. This week, I talk about how we can evolve beyond the existing operational models, thanks to AI. Which of course is ironic because in Dune, society rejected AI through the Butlerian Jihad. But anyway...

As a short recap to what became somewhat of a manifesto, I talk about how harsh environments like the planet Arrakis lead to hardened fighters. In ops, a hands-on environment leads to better operators. And in operational models where those same people are empowered to automate and solve their problems, and given the space to be altruistic and expand their tools for others to use, that yields the best operational outcome for customers and the most efficiency for a company.

And yes, as I write this I'm listening to the soundtrack from Dune Part 2. Thank you Hans Zimmer for this beautiful gift to the world.

Platform Engineering: The Next Generation

This post looks to the future of how AI transforms Platform Engineering teams in exciting ways.

Of course AI helps all of the models. The latest service I've been working on — AWS DevOps Agent — is being used by teams of all configurations, from customer support teams, to frontline ops, to SRE, to DevOps, and even Platform Engineering. It troubleshoots typical alarm-driven "find the root cause", but also helps with adhoc ops like researching a specific customer's reported issue, to scanning across teams' stuff for best practices, like finding missing or misconfigured alarm configuration.

But in Platform Engineering, the thing that AI can help with is particularly interesting to me. A Platform Engineering team is one who builds tools that help other teams build, test, deploy, secure, and operate their systems with lower effort and with better outcomes. These teams try to push the ownership boundary as much as possible to simplify things for other teams. But it's hard to do and is a balancing act. Each team's environment varies just enough that building software to automate the act of applying, configuring, and using a tool in every bespoke environment is simply impractical.

I've worked on a team like this before. We built the web service framework that does all of the request/reply serialization, authentication, admission control, and that kind of stuff for teams across Amazon and AWS. By defining the API model, we could expand that out to take care of all that stuff, and even generate SDKs in a bunch of languages, all automatically.

Trading off solving the problem for customization

But there was only so far we could go with the abstraction. Sure, we had features like caching that people could configure, but we couldn't decide to add caching automatically without knowing more about the guarantees behind their APIs and the expectations of their clients. If we wanted to pursue that, we could provide a sort of "configuration wizard" but it would ultimately be a tedious 20 questions exercise. Everyone would have to decide to bother using the wizard, and then the hard part of testing and verification would be an exercise left to the reader.

In other cases, Platform Engineering teams have been able to push the decisions and ownership quite far. I've seen a couple of solutions that automatically set up certain types of system alarms (CPU, file descriptors, etc) and try to tune their thresholds, but that only works for customers who have specific tool setups, and specific architectures. Customization on top of these would have limited knobs that had to be coded up, but more knobs leads to more complexity, and pushes customers to just do the whole thing themselves. And it leaves everyone to set up their own application health alarms, so adding infra alarms isn't that hard to do once you're in there setting up alarms anyway.

It's a tough problem. On one hand, a Platform Engineering team can either stop short of solving the complete problem and give checklists and tons of configuration knobs that everyone has to sort through, understand, and apply. On the other extreme, a Platform Engineering team can solve the whole problem, but only if the customer is using very specific technologies, tools, and architectures.

Flexibility without configuration knob hell

The missing ingredient in both extremes is adaptability. Adaptability means that the tool can handle and reason about differences that it wasn't coded to handle. An agentic solution can say, "Oh, your service has 2 load balancers behind one DNS record? Great, let's set up alarms for all of them." But adaptability also needs customizability. But that's also solved with agents through steering, skills, and deterministic tools that gate the agent. If the agent didn't realize that one of those load balancers behind the DNS record is for a blue/green deployments, you can write a couple bullet points in a markdown file to guide it to understand that it's normal for one to be weighted to 0% traffic, and to adjust the alarms accordingly.

This really works. Last week I was looking at a retrospective where a system had a misconfigured alarm that would have helped the team get an earlier start on the problem. Some types of alarms are unfortunately easy to misconfigure, either pointing to the wrong underlying resource, or using the wrong statistic like "avg" where it was supposed to be "sum", or that kind of thing. Well to test my "agents are great" hypothesis, I tweaked my own test service to have the same alarm misconfiguration, and prompted AWS DevOps Agent to audit my alarms. And sure enough, it found the misconfiguration, and even recommended some others to add. Since I configure alarms as IaC, I fed its report to Kiro and had it fix and add the missing ones. To organize the alarms the way I like, I described my alarm strategy of rollups with composite alarms — something that I hadn't written down before because it's complicated and varies a little bit by every application — and it shifted the alarms to set them up better than I had it in the first place.

AI pushes the boundary on "how much of the full problem can you own" so much further than before. Instead of dumping pull requests of code upgrades to every team, you can trigger their load test suite. Sure, this was possible before, but it was impractical because there was no "run my bespoke load test suite and interpret the results in an isolated test environment" API. At least that's not a W3C standard that I'm familiar with. But with agents that can adapt, and try repeatedly until they get what they need, doing it is practical at scale.

Crossing bloodlines

Ops remains a cultural problem, and for the longest time people try to treat the cultural problem as a technology problem and fail to solve it. So while this new agentic AI stuff is a very powerful new technology, Platform Engineering teams still need to consider the DuneOps philosophy: strength comes from experience. When I was on a Platform Engineering team, the longer I was on it, the less of an intuition I had for where the real problems were for my customers. Sure, I'd talk to as many customers as I could, attend ops meetings and read retrospectives of where things went wrong, and all that. But my personal hands-on experience of the pain of ops was a snapshot in time that kept moving further into the past.

So maybe there's a new DuneOps model for the next generation of Platform Engineering. AI makes it so those teams can reach even further into the end solution — at scale — than ever before. Maybe this gives those teams just enough time to take on the ops for a production system that looks more like what their customers run. Sure, Platform Engineering teams operate the systems that they build, but those tend to have different operational requirements and be built differently. This way they still have the charter to build for everyone, but they gain the firsthand understanding that they are (or aren't) building the right thing for everyone. Wait, but this sounds like it has properties of an SRE team or a DevOps team or a Frontline Ops team! A sort of combination of parts of each!

In Dune, the Bene Gesserit worked in the shadows over 90 generations on a eugenics program, crossing bloodlines of the major houses with the end goal of producing a human with a mind so powerful that it could see past space and time — a being they called the Kwisatz Haderach.

DuneOps is exactly this. With AI, you can now cherry-pick genes from each operational model and combine them, to borrow strengths of other operational models to offset each of any inherent weaknesses in the core model you choose. (And if you think your operational model lacks weaknesses, read last week's Dune as an operational model and then "@" me on twitter or whatever.)

And so just like how Paul Atreides defeated Emperor Shaddam and his Sardaukar by harnessing his Harkonnen heritage, AI is the leverage we needed to be able to borrow from and combine the operational models to take them further than we could ever before.

I am obsessed with the new Dune movies by Denis Villeneuve. The filmmaking, the music by Hans Zimmer, the restraint and pacing, sweeping visuals, the use of space - it's a masterpiece. The subsequent books get a little...interesting for me, but I made it through them. Internet person NatureBased publishes a steady stream of Instagram reels to fill in the gaps in my understanding of the lore.

One recurring part of the lore of the Dune universe is how some groups of people become so much more powerful fighters than others. There are two groups in particular: the Sardaukar, who are the Emperor Shaddam's personal army, and the Fedaykin, who are Fremen fighters. Frank Herbert, the author of Dune, writes that both factions of fighters are stronger because of the environment from which they are forged. The Sardaukar are raised on the planet Salusa Secundus, which has a harsh environment that trains them to reject weakness. The Fremen live on the planet Dune, which is an even harsher environment, leading the fighters to be even more powerful and fanatical.

This idea about being "forged by harsh environments" has amusing parallels to my experience being a software developer and operator. I've learned to be a better operator by being in an intense operational environment. This in turn has made me a better software developer. It's taught me to avoid making the same mistake twice, but it's also taught me a certain operational paranoia that makes me question everything in terms of its operational safety. So I'll call this idea - becoming a better operator and better developer through a harsh ops experience - to be known as DuneOps.

What are the different operational models?

I've seen, participated in, and talked with people from engineers to CTOs about different operational models - the pros and cons, applicability to a given company or environment, and there's no consensus around which is universally "the best". Different situations call for different approaches.

To start with some bookkeeping, I'll give my personal definition of SRE, DevOps, Frontline, and Platform Engineering. These definitions aren't going to be nuanced and they'll be imprecise, but at least they'll serve as ragebait for engagement on social media. Consider these to be caricatures of each, so if I draw the forehead of your operational model to be comically large, just know it's all in good fun.

• On one extreme, you have Frontline Ops. This is just having a separate ops team of people whose job it is to do pure ops. Think carry the pager and take all of the alarms. They unblock CICD pipelines, patch hosts, you name it. The dev team throws everything over the fence to the front line team.
• On the opposite end, you have Platform Engineering. This is a central team who writes tools and systems that everyone can use, but they don't own any aspect of the operations of others' systems. They might build CI/CD tools or best practice scanners or golden path abstractions that hopefully solve real problems that people have day to day.
• Somewhere in between is Site Reliability Engineering (SRE). I've heard some interpretations where these teams do take the first-pass at every alarm. But they build their way out of pain, are experts in best practices and how to set up everything from alarms to pipelines, and they have strict contracts with the dev teams so that if the systems become unwieldy, the dev team agrees to pause everything and burn down the ops backlog.
• And then there's DevOps. Here the dev team is on the hook for everything, and they all carry the first-responder pager. If their pipeline is flakey, they own cleaning it up. If their alarms are either too sensitive or not sensitive enough, they're the ones who need to figure out the right way to deal with it. If their ops are too manual, they need to automate it.

Of course you don't have to pick one model. Platform Engineering supplements all of these approaches. Some systems might shared require specific ownership that lends itself to SRE, like a Kubernetes cluster, database service, web server, or API gateway.

DevOps

I've spent the vast majority of my career on teams that do DevOps - where developers do all of their own ops. As developers, we can build our way out of any problem. We're all at least a little bit prima donnas so when ops gets too painful, we bring pitchforks to our backlog prioritization meetings and fix it or we walk. We can see the pager pain of our team compared to other teams, so that transparency is helpful incentive even for managers. Fewer people will transfer to a team that has excessive ops, and more people will leave such teams.

I swear by the DevOps model, and I think in the age of agentic development tools, developers will become more generalist and wear more hats, not fewer. I'm part developer, working on code. I'm part people manager - especially the more senior I get - because plenty of getting things done involves involving and coaching the right people and getting everyone marching in the same (or compatible) direction. I'm part customer support, because by hearing straight from customers I can tell where the product I'm working on is bad or what it's missing. I'm part sales and marketing, so I can make sure the product sounds good and has easy and intuitive getting started and attractive features to buyers. I'm part product manager, since I tend to work on products where I'm also the customer. And of course I'm part operator since I can architect and build the most amazing sounding system in the world, but if I miss some important detail that matters a ton in practice, then the system will fail and nobody will care about how it works in theory. With agentic coding, this gives us more time and need to wear these multiple hats. So I think where we're headed is into more of this DevOps model vs one of specialization.

At least in the environments I've been in, DevOps has led to the best long-term quality of the services we build, and the best operational outcomes for customers. When operational pain creeps up, the team naturally trades off other work to respond to resolving the pain. Since the team is in touch directly with customers, they hear about things that are bad about the operational or customer experience, even if the metrics don't measure or represent the pain well. And most importantly, they teach developers about what actually matters. You can have the most amazing theoretical architecture in the world, but if your database replication doesn't catch up even during a load spike and node down situation, then your customers are going to see latency or outages.

But of course there are multiple sides to a coin (typically two). DevOps' downside is you never have enough time. The ops backlog is infinite. There's always some alarm that you can tune to "cry wolf" less often, or another alarm to be made more sensitive to detect problems earlier. In DevOps, when you build things to automate away the pain, you build it for yourself. It's the Ayn Rand model of operations. Actually I never read any of her stuff so I could be totally off base. What I mean is that teams are responsible for solving all of their problems themselves. They're on the hook for everything, from security to availability to customer service. Hopefully someone helps provide tools (like a platform engineering team), but each team is left cobbling the tools together and deals with the consequences of the gaps in the tooling.

Within DevOps teams, sometimes an altruistic engineer comes along and makes their bespoke automation tool reusable for other teams, but that requires vision from the engineer, and an understanding manager who gives them room to force multiple instead of overrotating on that one team's goals. I've been that altruistic engineer and have spent a ton of time cheerleading the other ones around, providing covering fire when needed. Sometimes these people find themselves rotating into a central team to spend all of their time on their side project with the goal of helping everyone in the company. And since AWS' business is all about making ops easier for customers, sometimes these people rotate onto a product team who aims to solve that problem for everyone.

DevOps' other downside is that you can miss things. Yes, even an experienced DevOps engineer like myself has somewhat recently missed having an auto-rollback alarm in a pipeline to kick off a roll back of a deployment before anyone even gets paged, leading to the oncall having to get paged and trigger the rollback themselves. It wasn't the end of the world, but it left me facepalming wondering how I missed something so obvious.

Platform Engineering

To complement the DevOps model, you need a great platform engineering team. These teams get ahead of the problems faced by all DevOps teams, and build abstractions that makes everyone's lives easier. They also try to help teams see what things they might be missing. To figure out what to build, they can perform "tool harvesting" where they go around and look at all of the bespoke tools that people have made, look for patterns, and make the general version. To make it so DevOps teams don't miss things, platform engineering teams also make "scanners" or "auditors" or "recommenders". For example if a pipeline is missing an auto-rollback alarm, the pipeline system will reach out to you and let you know you're missing it. For certain types of misconfigurations, the pipeline can even refuse to deploy until you fix it.

I've also worked on a Platform Engineering team, building a web service framework called Coral. This framework handles request and response serialization, validation, protocols, authentication, rate limiting, SDK generation, and whatever else we can make more convenient for service owners and consumers. I joined it because I was fed up with service teams providing REST APIs and leaving actually calling the service from a typed language like Java as an exercise for the reader. When I heard we were making a framework that would let service owners have their cake and let consumers eat it too by auto-generating those clients, I said, "sign me up".

This web service framework team was interesting. In some ways, we had zero oncall burden. After all, we were making a framework, not running a service. On the other hand, our ops burden was the highest of any team I'd been on. Why? Because we fielded every question from every Amazon developer who got stuck and suspected the framework. Sure, we had rough edges in the framework, and gaps in our documentation, but any time someone saw a stack trace with our framework name somewhere in it (every stack trace), there was a chance that they would do the low-effort thing of just asking us to help. We'd be as patient and helpful as humanly possible in helping people figure out their own problems in their own code, but we also had plenty of sharp edges that we had to smooth out and documentation to improve.

In a sense were a DevOps team operating this framework. When ops spiked, we'd find the common patterns in the questions we got, and would deprioritize new feature development until we got it under control. Every customer question we answered without having that answer be available and discoverable in our internal documentation was a defect, and we worked to drive that down to zero. We also spent time on general education on how to help oneself and "how to help us help you", frequently citing this brilliant manifesto "How to Ask Questions The Smart Way" by Eric Steven Raymond and Rick Moen. (Eric and Rick, THANK YOU.)

The biggest downside is that you lack a first-hand signal about whether you're solving the real problems for customers. Every year I spent on the platform engineering team, the less confident I was that I was solving the biggest source of pain from customers feature-wise. One solution is to rotate in and out of a team like this. When peers have been super passionate about an idea and abstraction to solve a problem broadly, I encouraged them to rotate into our platform engineering team for a while (long enough to see it through and iterate on it). And when they're done, some stay but others rotate back out into the fray. I know plenty of folks who have stayed on a platform engineering team working on problems that take a really, really long time to solve and adapt to changing times, and they're doing great at it. So it isn't necessary to rotate out, but it's very necessary to rotate in.

The second downside to platform engineering is scaling ops. At the end of the day if it costs someone nothing to ask a question, and costs a great deal to answer that question, that's an asymmetric scaling problem that turns into a DDoS. Scaling support for that leads to the loss of signal about what's bad in your stuff, reducing overall quality.

Site Reliability Engineering (SRE)

While I haven't been on a "true" SRE team before, I've been on a team that seemed a lot like one. This was my first job out of college at Amazon on a team called Website Hosting. We ran the web server environment for the fleets that rendered HTML for amazon.com and its regional variants. Unlike the rest of Amazon, we weren't exactly DevOps. We didn't run the code we wrote. I didn't write the website rendering framework. I didn't write the perl/mason code that decided what to actually show on the page. I ran the servers. So I needed to make sure we forecasted how many we'd need, estimating traffic and efficiency of each page render. I ran the CI/CD pipeline that everyone fence-threw their releases onto. And I dealt with single-server failures and crashes, and OS upgrades and performance tuning. After all, someone had to do this stuff. It couldn't be done by a shared oncall rotation of hundreds of developers who knew nothing about the server environment.

The upsides were great in this model. Our only job was to automate the operations of this fleet, so we weren't distracted by any features to build. We were connected directly to the pain of what mattered, so we were the perfect product managers to decide what to build. And while we ran server ops for a countable number of fleets, we weren't on the hook for everything in the company. But we owned just enough fleets that everything we built needed to be configurable and extensible - making it an easy extension to make our tools usable by other teams too. Essentially we could be a platform engineering team that was slowed down by operating an unrelated system to the tools we built, but it sharpened our focus into the pragmatic. In fact one of the alarm aggregation systems we built on this team got adopted across all of Amazon, and eventually became (essentially) the CloudWatch composite alarms feature to roll up many alarms into one page. We realized we needed this feature when every web server paged us individually resulting in a loop of some 500 back-to-back pages, to which the oncall responded by ripping the battery out of the pager and hurling it against the wall. (In retrospect either of those actions would have likely succeeded, but why not both?)

There were certainly downsides to this model. We were hired as software developers, but we were doing the shit work writing and running scripts while our peers did the glamorous work building features and frameworks. We spent our day writing Greasemonkey browser scripts for provisioning servers, perl scripts for killing runaway processes, and VBScript to automate Excel-driven forecasting. And of course someone had to run those scripts and processes, so that was us too. Don't get me wrong, I love solving real practical problems of any shape or form, but we were doing this while we watched our peers build distributed systems, protocols, and frameworks in C++. Of course we thought big and worked on the long term solutions, like a replacement for the Excel sheet involving some cool managed R environment with a time series database. But it wasn't glamorous.

We also got paged all the time. Whenever there was an "order drop" (the best metric for "is there a problem with the website right now" is "have people stopped buying things"), our team was on the default engagement list because sometimes the problem was with our web servers or our load balancers in front of them. Even if it wasn't our fault, we could help because we were really good at reading the tea leaves (the logs and metrics and traces, even though they weren't called that yet) to figure out which microservice in the ball of wax architecture was the culprit. Engineers tended not to stay super-long on this team before they rotated out to other teams. But it was a hell of a Dune-like environment for us Fedaykin Fremen fighters to grow stronger in ops.

Frontline Ops

The only model that I can't say I've personally participated in is frontline ops. I suppose being a bank teller in high school was essentially this, since we were doing manual transactions for people. My attempt at reducing ops was to teach customers how to get ATM cards. But we were actually attached to sales to try to bring in more money to the bank, so that part of the job was not my jam.

But I've experienced frontline ops plenty and have studied it somewhat. Often frontline ops comes in the unenviable position of customer support. One day I tried to order a replacement physical pager so I could go on call, but the reordering system was broken. Long story short, I found that there was a re-ordering edge case that some people hit, and the resolution to the problem was to email Brenda at a 3p pager vendor directly. The support team for this kind of corporate system support had gotten over a hundred of these tickets in a couple years, but none of the awareness of the problem had bubbled up to the team who owned pager reordering. Once I pointed out the problem the team who owned paging fixed its integration with the 3p right quite quickly. Everyone involved was very helpful and great owners; it was the structure that removed the signal from the team who could stop the pain. Sure frontline ops teams produce reports and data and ticket trends so central teams can know what to fix. This is even described in the SRE books. But I find it's too easy for those best intentions mechanisms to break down, or for nuance to be lost in translation.

The place I've seen frontline ops used the most effectively is when scaling and automation is not yet possible, from certain physical or network infrastructure ops, to customer support. Every time I shadow customer support I get hit with a wave of learning about everything that sucks. But it's energizing. I feed off of that with optimism that we can fix things. I just need to go back to the well often so that I get fill up with the next problem to carry the banner for. And every time I talk with physical or network infrastructure folks I run into people who both fearlessly save the day and who put in place the most careful operational processes I have seen.

Frontline ops engineers are forged from the harshest environment of anyone, creating the strongest operators. I've worked with many support engineers who have converted to software developer roles or solution architects, and they bring a level of customer obsession and operational excellence that nobody else on the team has.

Keeping the best while avoiding the worst

DuneOps is all about understanding the environment that you're forged from, so you can take the parts that naturally improve you, and compensate for the parts of the environment that make some skill atrophe. Here are some things to consider to compensate for each operational model:

• DevOps works best when you incentivize organizational altruism, making tools that you actively share with other teams to make their lives better too. Cross-team altruism needs to be part of the job description so that managers know that they can still put together a high performance rating or promotion case for someone who spends a lot of their time on it. No two software developers are the same; some spend time on ops automation, others focus on testing, and others focus on micro-optimization. But it's easy to have a role guideline talk about "shipping features" and forget to encourage altruism.
• Platform Engineering works best when it is connected directly to what's real. That means actively "tool harvesting". If a team spent their precious cycles to build some automation, that means they've solved a real problem that could apply broadly, and should be amplified or brought into the fold. It also means encouraging mobility so that people who lean toward doing that force multiplying can do some kind of rotation into the platform engineering team to bring in direct understanding of what it's like to build using the platform engineering team's stuff.
• SRE works best when you aren't carrying the pager for other teams' crappy code and writing contracts around how much pain is acceptable before service teams fix their shit code. And it works best when the team has license to see things through and not spend all their time writing Greasemonkey bandaids and one-off scripts. They need to be able to be a Platform Engineering team and tool provider for other teams to use too or else they're stuck doing only the crap work.
• Frontline ops works best when their extreme expertise is sought out regularly and issues addressed urgently. These engineers have the best understanding of how things actually work and fail, and how customers are disappointed. It's easy for teams to settle in and ignore this, but watching it is essential.

DuneOps

DuneOps can emerge from any of these operational models - DevOps or SRE or Frontline ops. In Dune, the most powerful fighters are forged by harsh environments. And so in ops, the best operators are born from harsh, real-world operations. They need to be able to make mistakes and learn, see what works and what doesn't, and find out what actually matters by being accountable for the outcomes.

No matter what operational model you go with, I find it's important that dev and ops teams:

1. Stay grounded in reality. The closer you can put the responsibility for building central solutions to the place where the real ops is happening, the more complete the outcome.
2. Put operational ownership with service ownership. The people whose software is broken should have their pager's go off first, not second.
3. Incentivize organizational altruism. Build a culture where people share their solutions to improve others' ops as well as theirs.
4. Keep blame focused on structure and not people. We didn't talk about it in this post but it's super important and is covered in this talk if you're interested.

Epilogue

I'm sure folks will disagree with parts of this. I'm happy to chat about it (see social media links) and keep updating this as I collect more perspectives. And please know and believe that I'm not trying to offend anyone with any of these takes or colorful language. All of this stuff is extremely hard, and I respect everyone who been involved with any of this kind of stuff. After all, we are all brothers and sisters of the Fremen, forged out of the harsh desert of Dune.

I think it came out well, but I didn't realize until mid-day after recording that I was coming down with a cold. I went home and took a 4 hour nap, so hopefully I used all my energy up on the podcast and still had a nice discussion.

Listen on your platform of choice:

• Apple Podcasts
• Spotify
• Amazon Music
• Audible

New York Times reporter Clive Thompson worked on a story where he interviewed more than 70 coders about how AI was changing the profession. A few of us at Amazon also chatted with him.

I brought a slightly different angle, around how AI is changing ops, which is tightly connected to software engineering. Clive liked seeing my physical pager that I had on my desk, and apparently I gave off a "vibe", because he described me hilariously as, "...a trim man of 42 with a gray beard and jittery intensity."

It was super fun showing Clive how I was using AI for ops, so I can imagine I was bouncing off the walls a bit showing everything that I'm so excited and passionate about.

One funny thing is that this interview was during the preview for AWS DevOps Agent, which solved the operational issue I was demoing in 15 minutes. Now that we're GA and added things like learned skills, DevOps Agent solves that same issue in 4 minutes.

Some funny behind the scenes: for this re:Invent, I took a big risk of attending a close friend's wedding in New Delhi right before re:Invent. As in, I flew from India straight to Vegas. That was some serious jet lag, but I think I managed to stay coherent for things like this podcast interview.

I made this hype video showing AWS DevOps Agent in my everyday, but wasn't sure if we'd use it in the talk. I had no idea what Bill Fine was going to say in the intro part before the hype video so I was pretty embarrassed when he talked me up so much. But I trusted Bill's speaking instincts. After all when we decided we were co-presenting this, I said something like, "Hey we're going to need to put this together well because I like to work hard on these talks and get great ratings. I usually end up top 5 in the track." To which Bill replied, "Oh yeah last year I got the top score." Fair enough!

Take a look in this talk for an intro of the DevOps Agent's capabilities from launch, and some demos showing it in action. Well, recorded, but in action.