What Happens When No One Understands the System Anymore?

2026-09-14

TLDR

Introduction

This is going to feel like an anti-AI post but I promise you it is not. At this point, not using AI to speed up development almost feels like handcuffing yourself.

And honestly, it’s a great feeling when some random 500 error shows up in your service, you throw it at an AI tool, and a minute later you have the fix.

But I’ve also seen the other side of it. A major issue happens and AI doesn’t have the answer, or more likely, it gives you something that sounds right but isn’t actually correct.

Right now, that’s usually survivable because there are still people around who understand the system well enough to figure out what’s really happening.

My concern is what happens when that context starts to disappear.

The real risk isn’t that AI will occasionally be wrong. It’s that we may slowly stop building the knowledge needed to recognize when it’s wrong, understand why a system is failing, or know where to look when the obvious answers don’t work.

A lot of the foundational knowledge engineers used to build out of necessity can now be skipped. You can get surprisingly far without really understanding what is happening underneath the code you’re writing. At the same time, teams are being pushed for more output, so there’s less room to slow down, dig in, and learn as you go.

I think AI can make software engineers dramatically better. But only if we use it in a way that helps people build context instead of replacing the need for it.

The Junior Engineer Problem

The ironic part of calling this a problem is that junior engineers are probably producing far more code than junior engineers were 10 years ago. But producing more code was never the hard part. The hard part was producing one really good change.

Expectations around speed are getting higher every day. I’m not arguing that engineers shouldn’t move faster with AI, they should. The problem is that time spent going back and forth with AI agents, reviewing more code, and dealing with much larger PRs leaves less time to actually learn and understand what is being built.

Getting something working has never been the hardest part of software engineering. With a little coding knowledge, you can get a prototype up surprisingly quickly. The hard part is understanding the tradeoffs, the edge cases, the architecture, and how your system actually works. That is what turns something that works into something you can safely run in production.

When you have that context, you naturally start asking different questions. Is this secure? Will this be easy to extend six months from now? Are we adding unnecessary complexity? Is this the right technology for the problem? How does this change affect the rest of the system?

Without that knowledge, the workflow can easily become: give AI a vague prompt, make sure the feature appears to work, and open a PR.

That is the part I don’t think is sustainable.

It also pushes more of the burden onto other engineers. Where a junior engineer may have opened a PR every few days before, they might now be opening two or three a day. That means more code to review, more subtle problems to catch, and less time for the people reviewing it to pass along the context that actually helps someone grow.

And I want to be very clear: this is not a junior engineer problem. I don’t blame junior engineers for using AI as much as possible when they are being asked to move at a pace that practically requires it. The problem is the environment we are creating around them.

What this looks like in the Real World

I'd be remiss to sit here and talk about all these problems without calling out a couple real-world examples I've run into over the last few years.

In one system, we had a major bottleneck in our data pipeline that was preventing us from getting an updated dataset into our backend. We were processing terabytes of data, and I spent way too much time throwing the problem at AI and trying the solutions it gave me. Eventually I stepped back and decided I needed to actually understand what was happening. We were using Solr on the backend, and I eventually found two problems: Solr performs segment merges that can consume a lot of the same processing power needed for indexing, and our documents varied drastically in size. Some were tiny, while others were much larger. That led to two relatively simple changes: adjusting the Solr configuration so merges happened more consistently instead of building up toward the end, and changing how many ECS tasks we ran depending on which dataset we were processing. What was interesting was that neither fix was really a code change. AI kept pushing me toward changing the indexing code, tuning heap settings, changing batch sizes, or making larger infrastructure changes. What finally got me there was reading the Solr documentation and Designing Data-Intensive Applications. The section on Lucene helped me understand segments and merges well enough to realize they might be the bottleneck. Once I had that context, I went back to AI, and suddenly it became much more useful.

I ran into something similar with a memory leak, which I know is a little ironic given the name of the blog. Before doing much debugging myself, I asked Claude Code to look at the problem, and its initial suggestion was essentially to move a couple global variables to local ones. That wasn't it. I started digging into the system and noticed we logged whenever connections opened and closed, with an ID associated with each one. I used Claude to help query our OpenSearch logs and find connections that had opened but never closed, and there were a lot of them. At that point we finally had something concrete. I asked Claude to help reproduce the issue from the code, but none of the suggested reproductions actually triggered it. I ended up playing QA for a while, testing every path I could think of until I finally found a reliable way to reproduce the leak. Once we knew what was actually happening, Claude became very helpful again and helped work through the fix, including adding a watchdog timer so connections would eventually close even in the worst case.

That's the pattern I keep coming back to. In both cases, AI absolutely helped me solve the problem, but it became dramatically more useful once I had enough context to understand the system, narrow down the failure, and recognize which suggestions actually made sense. Without that context, I could have kept spinning in circles for a very long time.

The Bigger Problem: Losing System Context

Now take both of the examples above and imagine that instead of diving into the lower-level details, we just kept throwing the problems at AI tools. Maybe we eventually get to a solution that works, but at what cost? The issue isn't just understanding the systems we're building. It's also understanding the technologies those systems depend on at a deep enough level to know when something doesn't make sense.

This isn't strictly a junior engineer problem either. Say you ask AI to build a new service and it chooses DynamoDB for the database, but you've only ever worked with relational databases. How much time are you going to spend understanding how DynamoDB actually works, where its tradeoffs are, and how it needs to be tuned as the system grows? That's the part I think needs more attention. The answer isn't "AI tools are bad, don't use them." It's that AI becomes much more valuable once you have enough context to evaluate what it's giving you. We still need to give engineers time to dive deep into technologies, understand why they work the way they do, and build the kind of intuition that lets them spot problems before they show up in production.

AI can probably fix a missing connection pool, bad null handling, or any number of other issues after they surface. But there's a big difference between fixing those problems after the fact and having enough understanding to avoid them in the first place. It might feel slower to stop and build that context, but I think it makes engineers much faster over time. Understanding how a system works and being able to reason through architectural tradeoffs is going to become an even more important skill as AI gets better at writing the code itself. Once you have that foundation, AI starts to feel less like magic and more like a superpower.

How we use AI without Losing Understanding

I'm not completely confident we can get there with the current state of the industry, because a lot of this is going to come down to leadership. If expectations around output keep increasing, engineers are going to spend more and more of their time prompting AI agents to write code, review code, debug problems, and write documentation. At some point, organizations have to decide whether the goal is simply more output or whether they also care about building engineers who understand the systems they are responsible for.

If I were leading a technology organization today, I would start prioritizing junior engineers again and treat them as an investment in the future talent pool. I would absolutely want them using AI, but I would design that experience differently. I could imagine specific agents or skills for junior engineers that focus just as much on teaching as they do on completing the task. If an agent writes some code, it could also explain the pattern being used, point to the relevant documentation, and call out the parts of the system the engineer should understand. They might move a little slower, but so what? Junior employees in every industry are expected to spend time learning. I don't see why software engineering should suddenly be different just because we found a way to generate code faster.

I would also put more emphasis back on pair programming. It feels like that has become less common as AI has taken over the role of the "pair programmer," but some of the most valuable learning I've had came from sitting with someone more experienced and hearing why they approached a problem a certain way. AI can still be part of that process, but it doesn't have to replace the human part. We already spend plenty of time waiting for agents to finish tasks. That could be a great opportunity for a senior engineer to explain an architectural decision, walk through a tradeoff, or provide context that won't necessarily show up in the generated code.

The other piece is simply giving engineers time to learn. That might mean reading a book all the way through, watching a deep technical series, digging through documentation, or building something on their own without worrying about how fast they can finish it. It's going to be difficult to build real context if AI is performing a hundred tasks a day on your behalf. Giving engineers time to learn might cost some short-term speed, but I think the long-term payoff is better engineers, better systems, and people who know what to do when the AI doesn't have the answer.

Conclusion

AI is not going away, and I don't think it should. It is already making engineers faster and more capable, and that will only continue. The challenge is making sure we don't optimize so heavily for speed that we slowly lose the knowledge needed to understand the systems we're building.

The best engineers in an AI-driven world won't be the ones who avoid these tools. They'll be the ones who know enough to use them well, question them when necessary, and still know what to do when they don't have the answer.