“>

Condition 1: Agents have to be reliable at the length of task a junior actually gets

The best measurement we have here is METR’s time-horizon work. They time human experts on real software tasks, then find the task length at which a model succeeds 50% of the time.

The main result is that this horizon doubled roughly every seven months from 2019 to 2025. METR’s updated Time Horizon 1.1 expanded the task suite by 34% and doubled the count of tasks running eight hours or longer. Independent readings of the 2024 to 2026 window suggest the doubling has since accelerated. The live leaderboard now puts frontier horizons in the hours.

That sounds decisive. Read the methodology and it stops being decisive.

Two things are important:

• First, 50% is not a bar you can staff against. Kwa et al. also report an 80% horizon, and at any given moment it is dramatically shorter than the 50% figure. In their data, frontier systems are near-perfect on tasks a human finishes in under four minutes and succeed less than 10% of the time on tasks that take a human more than four hours.
• Second, and this is the part almost nobody quotes: METR says its tasks are deliberately self-contained and well-specified. Their own framing is that a two-hour task should be read as what someone with no prior context could do in two hours, not what an experienced engineer familiar with the codebase could do.

That is precisely the wrong shape. A junior engineer’s first six months are almost entirely context acquisition. Which service owns this. Why that abstraction exists. Who to ask. The benchmark measures the one part of the job that has been stripped of the thing that makes it hard.

Condition 2: The benchmark has to measure the job

In February 2026, OpenAI stopped reporting SWE-bench Verified and recommended others do the same.

Their reasoning is worth reading in full, but two findings stand out. They audited a 27.6% subset of the dataset and found that at least 59.4% of the audited problems had flawed test cases that reject functionally correct solutions. And they found contamination: frontier models could reproduce exact gold patches and verbatim problem details, indicating training exposure.

State of the art had moved from 74.9% to 80.9% over six months. The question OpenAI asked was whether the remaining failures reflected model limits or dataset properties. The answer was mostly dataset properties.

Move to a harder, less contaminated set and scores fall off a cliff. SWE-bench Pro was built for exactly this, and frontier performance on it sits far below the Verified figures the launch posts advertise. Newer suites like Terminal-Bench and long-horizon evolution benchmarks are being built for the same reason.

I want to be careful here. This is not “benchmarks are useless.” It is narrower and more damaging: the specific number that has been used for two years to argue juniors are obsolete was retired by the lab that created it, for reasons that make the number look better than reality.

Condition 3: The cost of verifying agent output has to fall below the cost of delegating to a person

This is the condition I think gets ignored most, and it is the one with the cleanest experimental evidence.

METR ran a randomized controlled trial with 16 experienced open-source developers across 246 real tasks in their own repositories. AI allowed or disallowed at random. Screen recordings. Real work.

Developers forecast a 24% speedup. Afterwards they estimated they had been 20% faster. They were 19% slower.

Two caveats, because I would rather you trust the rest of this piece. The tools were early-2025. The sample is small and specific: experienced developers on mature codebases they know well. This is not a universal productivity estimate and METR does not claim it is.

But the perception gap is the durable finding. People were wrong about the direction of their own productivity, under measurement.

The wider data points the same way. Stack Overflow’s 2025 survey of more than 49,000 developers found 84% using or planning to use AI tools, while 46% actively distrust the accuracy of the output against 33% who trust it. Only 3% report high trust. Among experienced developers, high distrust runs at 20%.

Google’s DORA research surveyed around 5,000 professionals and found 90% using AI at work and over 80% believing it lifted their productivity, while 30% report little or no trust in AI-generated code. DORA’s throughput finding improved from the prior year. Delivery instability did not. Their conclusion is that AI is an amplifier: it magnifies what the organization already is.

Put it together. Generation got cheap. Verification did not. Review capacity is now the constraint, and review capacity is senior engineer time.

Condition 4: Firms have to be willing to break their own senior pipeline

Here is the uncomfortable part.

Conditions 1 through 3 describe whether the substitution works. Condition 4 describes whether firms will attempt it anyway. They are not the same question, and the second one is already answered.

Stanford’s Digital Economy Lab tracks ADP payroll data covering roughly one in six American workers. Their Canaries work finds that employment for 22 to 25 year olds in the most AI-exposed occupations, software development among them, has diverged sharply from older workers in the same occupations. The shortfall measured 15% at the July 2025 data vintage. As of June 2026 it is 19%.

The live dashboard shows the adjustment running through reduced hiring rather than separations. Nobody is being fired. The door is closing.

The mechanism the revised paper proposes is the most interesting finding in any of this. Employment fell among young workers in occupations that lean on codified knowledge, the kind you can learn from documentation and standardized procedure. It rose among experienced workers in occupations that lean on tacit knowledge, acquired through practice, mentorship and repeated exposure to real situations.

Stanford is careful that these are descriptive patterns, not causal estimates. Take that seriously.

But if the mechanism holds, notice what it implies. Codified knowledge is what a junior arrives with. Tacit knowledge is what a junior is supposed to acquire, by doing the codified work under supervision until the tacit part sinks in.

We are automating the apprenticeship and keeping the requirement for what the apprenticeship produced.

What I actually think

Agentic coding is not replacing junior engineers. It is replacing the tasks we used to hand junior engineers, which is a different thing with worse consequences.

The bottleneck was never generation. It is verification, context and judgment, and every measurement we have says the frontier is furthest from exactly those three.

Meanwhile hiring decisions are being made on benchmark numbers that the lab which created them has publicly retired.

The firms that will look smart in three years are the ones running the boring experiment: keep hiring juniors, give them agents on day one, and measure whether they reach senior judgment faster than the previous cohort. My guess is that they will, substantially. Nobody is funding that study, because it does not produce a number for an earnings call.

What would change my mind

I would rather be falsifiable than clever. Here is what I am watching:

• An 80% reliability horizon that clears a full working day on tasks with prior context, not self-contained ones.
• A frontier score above 60% on an uncontaminated, privately-authored long-horizon benchmark.
• A replication of the METR trial where measured time and perceived time point the same direction.
• DORA delivery instability falling for two consecutive years while AI adoption holds.
• The Stanford 22-to-25 employment gap narrowing while AI-exposure scores keep rising.

If three of those land, I will write the opposite of this piece and link back here.

Key Takeaways

• METR’s main horizon is a 50% success measure on context-free tasks; juniors work at neither.
• OpenAI retired SWE-bench Verified after finding flawed tests in most of an audited failure subset.
• Generation got cheap, verification did not; senior review time is now the real constraint.
• Stanford’s data shows a 19% employment gap for AI-exposed 22-to-25 year olds, driven by hiring freezes.
• We are automating the apprenticeship while still requiring what the apprenticeship produced.