Simulate Users Over Time to Find the Bugs Integration Tests Cannot See

A green integration suite proves the product survives one request. It says nothing about whether the product survives a month.

The Suite Was Green and the Product Was Wrong

The 11+ application I am building has a large integration suite. It is green. In two afternoons a cohort simulation harness found eight product defects it could not see, five of them P2. None were exotic. A parent dashboard said a child’s score was 54 while the student and tutor screens said 60. A roster card read “Overdue” forever once the due date passed, even after every student had finished. A report’s “Predicted pass” said 100 percent beside a dashboard tile saying 80, because two formulas lived under one label.

These are the bugs a parent notices in week two of real use, and no test author would ever think to write them down. I built the harness with Claude Code. This note is what I learned from running it.

Name the Class Before You Hunt It

I started by calling these temporal bugs. That is close but not precise. Time is what exposes them. The cause is that state evolves, gets copied or derived into several places, and those copies stop agreeing. I now call the class temporal state-consistency defects: bugs that only become observable as state moves across time, lifecycle transitions, asynchronous processing, and multiple read models.

The sub-types I found, each with a real instance:

• Transition bugs: correct at day t, wrong at t+1. An assignment card never reached in-progress, and abandoning it left it stuck.
• Stale derived state: valid once, never recomputed. The tutor roster streak read a dead metadata sentinel.
• Cross-view disagreement: two projections of one entity differ. Student 60, parent 54, tutor 60.
• Boundary bugs: correctness flips at a day, week, or timezone edge. Report window dates were UTC slices of London midnights, wrong only through the clock change.
• Dormant contracts: fields that fixtures populate and production never writes, so the read model looks alive under test and dead in use.

A conventional test asserts one step: given S0 and action a, did I get S1? The harness asserts a property of the whole trajectory: an invariant holds at every t, and sometimes a relation between two instants, such as “30 idle days cannot change a historical submitted score.”

Five Structural Reasons the Suite Is Blind

Nobody wrote bad tests. The suite is blind by construction.

1. It observes one instant. Seed, call, assert, tear down, inside a few seconds of one wall-clock moment. Nothing in it ever sees the day after a due date.
2. It asserts one surface at a time. Fixtures are per test and the three dashboards are owned by different files. The test that loads student, parent, and tutor views of the same child and diffs them does not exist.
3. Its fixtures write fields production never writes. Dev seeds and factories populate the derived fields directly, so dead fallbacks look alive.
4. It pins expected values by hand. The author typed the answer, so only the answers the author imagined get checked.
5. It never runs the whole system at once. One API per worktree, rarely a worker, never the real Temporal sweep or the outbox.

Every one of the eight defects needed at least two of those five things to be true at once.

What I Built Instead

The harness is a small synthetic world the product has to live in for weeks:

scenario generator (persona cross)
        ↓
known learner actions, one day at a time
        ↓
real public API only (never the database)
        ↓
API + worker + Temporal + outbox + schedules
        ↓
Postgres-resident virtual clock advances
        ↓
quiescence barrier (outbox empty, task queue drained)
        ↓
independent oracle derives expected state
        ↓
cross-surface invariants at every day boundary

The scenario is a persona cross: six timing personas (diligent, sporadic, abandoner, crammer, weekend-only, night owl) by five accuracy personas, with four parent and five tutor behaviours layered over the families. With 30 learners every cell appears at least once.

Three decisions carry the design.

The clock lives in Postgres, not in the process. Every process that touches the run reads one offset from one table, so the API, the worker, and the trigger bodies all agree on what day it is. The next section shows how.

Nothing is read until the world has settled. The barrier waits for zero pending, processing, and failed outbox rows and no running workflow on the run’s task queue, twice, 500 milliseconds apart. Without it every disagreement is ambiguous: bug, or worker not caught up yet. With it, an abandoned attempt being discarded by the real sweep workflow is a clean assertion.

The oracle is deliberately boring. It records what each learner actually did, on which simulated day, with how many correct, and derives streaks, scores, due labels, and feed order with arithmetic anyone could check. It imports nothing from the product’s domain packages. If the oracle calls the production scoring abstraction, both sides share the same bug and the harness proves nothing.

One surprise: the same seed does not reproduce a run across product changes, because the adaptive product decides what the next action sees. The JSONL trace is the reproduction artefact, not the seed.

How the Virtual Clock Actually Works

Time enters the system in three places: Postgres column defaults and SQL bodies, JavaScript Date in the API and worker, and Effect’s Clock service underneath DateTime.now. Miss one and the day boundary invariants become noise. The harness owns all three from a single row.

CREATE TABLE cohort_sim_a1b2.sim_clock (
  singleton boolean PRIMARY KEY DEFAULT true CHECK (singleton),
  offset_ms bigint NOT NULL
);

CREATE OR REPLACE FUNCTION cohort_sim_a1b2.now() RETURNS timestamptz
LANGUAGE sql STABLE AS $$
  SELECT pg_catalog.now()
       + (SELECT offset_ms FROM cohort_sim_a1b2.sim_clock) * interval '1 millisecond'
$$;

Each run gets its own schema, so the shadow now() lives next to the run’s tables and every process connects with search_path = <schema>, public, pg_catalog. Query-text now() and plpgsql bodies then resolve to the shadow. Column defaults do not.

Postgres binds DEFAULT now() to pg_catalog.now() by function OID when the table is created. A shadow on the search path reaches nothing already built. So after migrations run, the harness finds every default whose expression mentions now() and rewrites it:

const defaults = await client.query(
  `SELECT c.relname, a.attname, pg_get_expr(d.adbin, d.adrelid) AS expr
     FROM pg_attrdef d
     JOIN pg_attribute a ON a.attrelid = d.adrelid AND a.attnum = d.adnum
     JOIN pg_class c ON c.oid = d.adrelid
     JOIN pg_namespace n ON n.oid = c.relnamespace
    WHERE n.nspname = $1
      AND pg_get_expr(d.adbin, d.adrelid) ~ '\\mnow\\(\\)'
      AND c.relname <> 'sim_clock'`, [schema])

for (const row of defaults.rows) {
  const rewritten = row.expr
    .replaceAll('pg_catalog.now()', 'now()')
    .replaceAll(/(?<![.\w])now\(\)/g, `"${schema}".now()`)
  await client.query(
    `ALTER TABLE "${schema}"."${row.relname}" ALTER COLUMN "${row.attname}" SET DEFAULT ${rewritten}`)
}

That covers default(sql`now() + interval '30 days'`) on a session’s expiry as well as plain defaultNow(). Two more sweeps finish the job. Security-definer functions carry a pinned search path, so each one is re-pinned to <schema>, pg_catalog. And trigger bodies that call pg_catalog.now() explicitly are beyond any search path, so the harness pulls their definition with pg_get_functiondef, swaps the call for the shadow, and recreates them in the run schema. Setup prints what it touched, which on this schema is a few hundred defaults and a handful of functions.

Advancing time is one UPDATE:

const setClock = async (rt: Runtime, instant: Date): Promise<void> => {
  rt.simNow = instant
  await setSimClockOffset(rt.admin, rt.schemaName, instant.getTime() - Date.now())
  await sleep(600) // both processes poll every 250 ms
}

The JavaScript side is a preload the harness passes with node --import when it spawns the API and worker. It replaces the global Date and polls the same row:

const RealDate = Date
let offsetMs = simEnv.initialOffsetMs

class SimDate extends RealDate {
  constructor(...args: unknown[]) {
    if (args.length === 0) { super(RealDate.now() + offsetMs) } else { super(...(args as [number])) }
  }
  static override now(): number { return RealDate.now() + offsetMs }
}
Object.defineProperty(globalThis, 'Date', { value: SimDate, writable: true, configurable: true })

setInterval(async () => {
  const result = await client.query<{ offset_ms: string }>('SELECT offset_ms FROM sim_clock LIMIT 1')
  offsetMs = Number.parseInt(result.rows[0]?.offset_ms ?? '0', 10)
}, 250).unref()

The preload cannot run in production because it needs an environment variable only the harness sets, and product code never imports it.

Effect needed nothing extra, and that was a deliberate check rather than an assumption. The API is an Effect application. Its live Clock implements unsafeCurrentTimeMillis as Date.now(), and DateTime.now reads through that same clock, so the patched global is enough. The cleaner Effect answer is a Clock layer that reads the offset and is provided to the runtime, and TestClock is the right tool inside a single test. Neither reaches Drizzle’s defaults, the outbox listener, or the Temporal worker, which is why the offset lives in the database and the process patch is the thin part.

Two things the patch deliberately leaves alone. Temporal’s workflow sandbox ships its own deterministic Date, so workflows are untouched and only activities and API code see the virtual clock. And the offset only changes at a scheduled tick while nothing is in flight, so the 250 millisecond poll window is never observable.

A simulated day is a fixed sequence of those ticks in Europe/London wall-clock time: night owls at 00:30, a triage probe at 00:35, adults at 09:00, the afternoon cohort at 16:00, the late cohort at 23:30, and a boundary at 23:55 where the real stale-attempt sweep workflow is started, the barrier waits for quiescence, and every surface is read. Adults sign in again each morning because access tokens expire in simulated time too.

Finally the harness proves the clock held. At each boundary it checks that no row in eleven timestamp columns, from attempt started_at through domain_events.occurred_at, falls outside the run’s simulated window. A single process that leaked real time would show up as a timestamp weeks in the past, and it never has.

The Agent Layer Discovers Invariants, It Does Not Judge

The oracle only checks fields it has been told to read. At the end of a run the harness keeps the API and web app alive and sends Agent SDK sessions, one per actor across the three persona groups, through the real app with playwright-cli. Each session is grounded with that actor’s expected state and told to report only disagreements or visibly broken things. A skeptic session then tries to refute every candidate.

First sweep: 13 sessions, 10 candidates. Three were real. One was a route in my prompt that did not exist. Two were browser-clock artefacts, since the browser computes “3 days ago” from real time while the API sits hours ahead. Four were fidelity gaps: I was not sending topic ids or time taken the way the real client does.

Half the candidates being the harness’s own fault is still a result. The sweep is the best available test of the simulator’s fidelity to the production client.

The discipline that matters is the ratchet. A candidate is not a finding until a skeptic has failed to refute it, and a confirmed finding becomes a deterministic invariant the same day. The agent is an invariant discovery mechanism, never the oracle. The LLM does not decide whether the product works, and its share of the work shrinks with every sweep as coverage moves into the deterministic layer.

One lesson on skeptics: the in-process Agent SDK skeptic ran out of turns on nine of ten candidates. Proving that surface A reads a live score while surface B reads a stale profile field needs source, schema, and Git history. Browser agents are good at “this looks inconsistent.” Repo-aware agents are good at “here is why.” Keep the roles separate.

What It Costs and What It Found

Eight defects to date, five P2 and three P3, plus one rejected candidate where the UTC streak day was a documented product decision. Round three, still in progress, turned the harness on relationship changes: a tutor removes a student, a parent unlinks a child, a card-free trial lapses. Its first run found that reports a parent saw before unlinking a child still answer 200 to that parent afterwards. No request-level test would have staged that.

What it cannot see is worth naming too: schedule cadence, anything computed from the browser’s own clock, third-party callbacks, LLM-generated report prose, and concurrency. The oracle also cannot make product decisions. It can prove “Predicted pass” means two things on two screens. Picking the right one is a human call.

Where I Am Taking It

• Adversarial traces next to realistic cohorts: start, advance, abandon, advance eight days, submit, generated rather than scheduled.
• Metamorphic properties that need no exact answer: another completed assignment cannot lower the completed count; reordering independent learners cannot change either one’s final state.
• Replay equivalence: same trace, same clock, same initial database, same final state, or I have found nondeterminism.
• Fault injection in the async layer: worker dies after an outbox claim, duplicate delivery, timeout after commit.
• Client fidelity as an invariant: diff simulator request shapes against captured real-client traces so the harness cannot drift into an imaginary client.

I did not build more tests. I built a place where the product has to survive for months, and then read the diary.