Every engineer running Claude Code is already writing a diary of where the SDLC breaks: every stalled task, every abandoned plan, every loop they gave up on and finished by hand. Nobody uses that diary, because using it means a human reads a colleague’s session, and that is a trust violation no team will accept. Here is how to turn that diary into a working pipeline anyway, without a single person ever reading a transcript.
Author: James Phoenix | Date: July 2026
The dataset already exists, and reading it wrong kills the project
Every .jsonl transcript Claude Code writes to disk is a structured record of friction: the plan the agent revised four times, the permission prompt an engineer denied twice before giving up and doing it by hand, the tool call that timed out and never got retried, the task that just stops with no completion message. Multiply that across 100 engineers for a year and you have the single richest dataset an engineering org owns about where its own development process fails.
Nobody automates this today, because the obvious first design is to point a person, a manager, a vendor’s analyst, at someone else’s session log. The moment that happens, the pipeline stops being observability and becomes surveillance, and the org either kills the rollout or gets sanitized behavior instead of real signal. The data was never the problem. The read access is. Fix that one constraint before you write a line of the pipeline, or nothing downstream matters.
The shape of the pipeline
Treat it as four stages, each one a hard boundary the next stage cannot see behind:
- Read every engineer’s
.jsonlhistory, locally, on a schedule. - Classify each candidate event against a fixed failure-mode taxonomy with a confidence score.
- Gate on recurrence across engineers and classification confidence, before anything aggregates.
- Publish only what survives as an anonymized proposal, and let the team vote.
Stage 1: Read locally, air-gapped, never by a person
Run inference on hardware inside the company’s own network, on a weekly batch job that walks every engineer’s .jsonl history. Nothing leaves the building. No transcript, no excerpt, no summary tied to a name, goes to a cloud API, a vendor dashboard, or a manager’s inbox. This is the one constraint you do not get to relax later for convenience: no engineer is ever allowed to read a transcript, only a local model is.
This is the same move I make in Closed-Loop Agent Observability, aimed at people instead of infrastructure: don’t make a human reconstruct context from a noisy feed after the fact, extract the exact signal at the moment it exists and hand over only that. Here the noisy feed is a colleague’s session, and the thing worth protecting is trust, not blast radius.
Stage 2: Classify each event with structured output, not free text
None of the gates in Stage 3 work if the classifier hands back a paragraph you have to regex. The confidence gate needs a number you can threshold. The recurrence gate needs a fixed label you can group by, not five different phrasings of the same failure. So the model call has to return a validated object, not prose: a fixed enum for the failure mode, a numeric confidence, and one sentence of evidence to audit later.
Here is the shape using OpenRouter rather than a provider directly, which is the useful part if you want to swap the underlying model without touching the rest of the pipeline:
import { generateObject } from 'ai'
import { createOpenRouter } from '@openrouter/ai-sdk-provider'
import { z } from 'zod'
const openrouter = createOpenRouter({ apiKey: process.env.OPENROUTER_API_KEY })
const FailureMode = z.object({
failureMode: z.enum(['task_abandoned', 'permission_loop', 'context_exhausted', 'plan_churn', 'none']),
confidence: z.number().describe('Between 0 and 1.'),
evidence: z.string(),
})
const { object } = await generateObject({
model: openrouter('anthropic/claude-haiku-4.5'),
schema: FailureMode,
prompt: `Classify this Claude Code transcript excerpt into the failure-mode taxonomy:\n${transcriptExcerpt}`,
})
Run against a transcript excerpt where an agent proposes the same migration-script edit three times, gets denied permission each time, and quietly abandons the task, this returns:
classification: {
failureMode: 'permission_loop',
confidence: 0.95,
evidence: 'Agent repeatedly proposes the same edit after the user denies permission, then abandons the task rather than finding an alternative.'
}
That confidence is exactly the number Stage 3 thresholds on. That failureMode is exactly the label Stage 3 groups by when it counts recurrence. Swap anthropic/claude-haiku-4.5 for whichever model you are actually running air-gapped in production; the schema and the call do not change.
Full runnable version: 17-structured-output-openrouter.mjs in my AI Engineering Examples repo.
Stage 3: Gate on recurrence and confidence before anything aggregates
A local model classifying transcripts will find false patterns constantly, so nothing from Stage 2 reaches a human until it clears two gates:
Recurrence, not incidence. A failure mode does not surface unless it repeats across a minimum number of distinct engineers, in distinct sessions, over the observation window. One person struggling once is a data point about a person. The same stall pattern across a dozen unrelated engineers is a data point about the process. This threshold also keeps any single engineer from ever being individually identifiable in the output. It is a k-anonymity floor disguised as a quality bar.
Classification confidence, not vibes. Use the confidence field from Stage 2 directly. Drop anything below your threshold before aggregation, not into a human review queue, because a human reviewing borderline cases is exactly the read-access violation this whole pipeline exists to prevent.
Both gates run in the same batch job, before anything is written to a shared store. If a candidate failure mode does not clear recurrence and confidence, it never exists as far as any person in the company is concerned.
Stage 4: Publish survivors as anonymized proposals
What survives Stage 3 becomes one entry on an internal CRUD board the engineering team already owns: a small, anonymized, aggregated observation, phrased as a candidate root cause, not a name attached to a screenshot. Here is roughly what that board looks like in practice:
Engineers read each proposal, discuss it, and vote to approve or disprove it, the same way they would triage any other internally sourced idea. The model never decides anything. It proposes. The people who actually live the friction decide, collectively, whether the proposal is real. Disprove it and it is dropped. Approve it and it becomes a normal backlog item with a normal owner, the same as any process improvement the team would have raised in a retro, except this one was surfaced by evidence instead of whoever complained loudest that week.
This is why the pipeline is signal processing, not analytics. A dashboard reports a number to a manager and asks them to interpret it. This pipeline reports a candidate hypothesis to the people whose behavior generated it and asks them to falsify it. The unit of trust moves from “trust the vendor with your logs” to “trust your own team to grade its own proposals,” which is a much smaller ask.
Complementary to talking to people, not a replacement
This does not replace 1:1s, retros, or a stakeholder actually walking the floor. It fills a gap those channels miss structurally: engineers rarely self-report “I gave up on the agent’s plan and just wrote it by hand,” because admitting that in a retro sounds like a personal failing rather than a process signal. A stalled session in a transcript has no such incentive to under-report. The pipeline surfaces candidate topics a manager would never think to ask about, and hands the final judgment straight back to the people closest to the work.
Why automate this instead of asking people directly
The raw material is already sitting on disk, doing nothing, for any org running Claude Code at scale. Automating the read means no change to how engineers work, no new tool to adopt, and no data leaving the network, which is precisely what makes the design defensible to a compliance-anxious org that would reject a cloud analytics vendor on sight. What you are actually building is not a smarter model, it is a trust architecture: the air gap, the two gates, and the CRUD board that puts the verdict back in the team’s hands.
Where this breaks
Local inference is currently good enough to classify coarse, recurring failure modes and bad at anything requiring deep reasoning about intent, so keep the taxonomy coarse on purpose. The CRUD board only works if leadership actually commits to acting on approved ideas, otherwise it degrades into an ignored suggestion box within a quarter. And the whole pipeline rests on one property that has to be real, not asserted: if the air gap is ever violated, even once, the trust does not degrade gracefully, it collapses, and no amount of architecture afterward buys it back.
Related
- Closed-Loop Agent Observability: IDs + Injected Prompts – Extracting the exact signal at the moment it exists instead of reconstructing context after the fact
- Sentry Errors Should Spawn Agents on Your Own Machine – Running inference on your own hardware against your own environment instead of a vendor’s cloud
- The Session Audit Meta-Prompt – Turning a transcript into structured signal rather than a wall of text
- Systems Thinking & Observability – Treating engineering process as a system with measurable feedback loops
- Own Your Control Plane – Why the air gap has to be owned infrastructure, not a vendor’s promise
