Integration Testing Patterns

James Phoenix

Integration Over Unit Tests: Higher Signal-to-Noise for LLM Verification

Summary

For LLM-assisted development, integration tests provide higher signal-to-noise ratio than unit tests. A single integration test verifies an entire feature’s correctness across all layers, while dozens of unit tests verify isolated pieces that may still fail when composed. This inverts the traditional test pyramid: prioritize integration tests that verify end-to-end behavior, use unit tests only for complex logic.

The Problem

Unit tests verify isolated components work correctly, but LLM-generated code often fails when components interact. You can have 100% unit test coverage with all tests passing, yet the feature still doesn’t work because integration points are broken. Reviewing hundreds of unit tests to verify correctness is time-consuming and error-prone.

The Solution

Prioritize integration tests that verify complete features work end-to-end across all layers (API → business logic → database). A single passing integration test confirms the entire feature is correct, providing higher information value per test. Unit tests become secondary, used only for complex algorithmic logic or edge cases that are hard to trigger via integration tests.

The Problem with Unit Tests for LLM Code

The traditional testing pyramid tells us to write many unit tests, some integration tests, and few end-to-end tests. This wisdom comes from an era when:

Humans wrote code: Unit tests caught typos, logic errors, off-by-one bugs
Integration was slow: Database tests took seconds, API tests were flaky
Debugging was hard: Failing integration tests were difficult to diagnose

But in LLM-assisted development, these assumptions break down:

Unit Tests Verify the Wrong Thing

LLMs rarely make typos or simple logic errors. They’re probabilistic code generators trained on millions of examples. When Claude generates a function like calculateDiscount(price, percentage), it will almost certainly:

✅ Get the math right: price * (percentage / 100)
✅ Handle basic edge cases: percentage < 0 or price < 0
✅ Use correct types: number inputs, number output

Unit testing this function provides low signal—it’s verifying something the LLM is already good at.

Where LLMs do fail:

❌ Integration points: Wrong database column names, incorrect API contracts
❌ Type mismatches across boundaries: Frontend expects {id: string}, backend returns {id: number}
❌ Business logic composition: Individual functions work but composed workflow is wrong
❌ Side effects: Function doesn’t properly commit transaction, doesn’t send email

Unit tests don’t catch these failures. Only integration tests do.

The Signal-to-Noise Problem

When reviewing LLM-generated code, you have limited time and attention. What provides more signal?

Option A: 47 unit tests

✅ calculateDiscount returns correct value
✅ calculateDiscount handles negative percentage
✅ calculateDiscount handles zero price
✅ calculateDiscount handles large numbers
✅ validateEmail returns true for valid emails
✅ validateEmail returns false for invalid emails
✅ validateEmail handles null input
... (40 more unit tests)
```text

**Outcome**: All 47 tests pass. You assume the feature works.

**Reality**: The discount code checkout flow is completely broken because:
- The API endpoint expects `discountCode` but frontend sends `promoCode`
- The database constraint fails because `discount_amount` column is `INTEGER` but code sends `DECIMAL`
- The email notification never sends because the email service isn't imported

**Option B: 1 integration test**
```typescript
✅ User can apply discount code and complete checkout
```text

**Outcome**: Test passes → Feature works end-to-end

**Signal-to-noise ratio**:
- Unit tests: 47 tests to verify isolated pieces (high noise, low signal)
- Integration test: 1 test to verify entire feature (low noise, high signal)

## The Solution: Invert the Test Pyramid

### Traditional Test Pyramid (Human Development)

```text
        /\
       /E2E\           Few E2E tests (slow, flaky)
      /------\
     /  Integ \         Some integration tests
    /----------\
   /    Unit    \       Many unit tests (fast, focused)
  /--------------\
```text

**Rationale**: Unit tests are fast, stable, easy to debug. Integration tests are slow, flaky, hard to debug.

### LLM-Optimized Test Pyramid

```text
        /\
       /E2E\           Few E2E tests (still valuable for critical paths)
      /------\
     /        \        
    /  INTEG   \       **MANY integration tests** (high signal)
   /            \
  /    Unit      \     Few unit tests (only for complex logic)
 /----------------\
```text

**Rationale**: LLMs generate correct isolated logic but fail at integration points. Integration tests catch the failures that matter.

### What This Means in Practice

**For every feature, write**:

1. **1-3 integration tests** that verify the feature works end-to-end
2. **0-2 unit tests** for genuinely complex algorithmic logic (if any)
3. **0-1 E2E tests** for critical user journeys (optional)

**Example: User Registration Feature**

**Integration tests** (write these first):
```typescript
describe('User Registration', () => {
  it('successfully registers new user with valid data', async () => {
    const response = await api.post('/auth/register', {
      email: '[email protected]',
      password: 'SecurePass123!',
      name: 'Test User'
    });
    
    expect(response.status).toBe(201);
    expect(response.body.user.email).toBe('[email protected]');
    
    // Verify user exists in database
    const user = await db.users.findOne({ email: '[email protected]' });
    expect(user).toBeDefined();
    expect(user.emailVerified).toBe(false);
    
    // Verify email was sent
    const emails = await testMailbox.getEmails('[email protected]');
    expect(emails).toHaveLength(1);
    expect(emails[0].subject).toContain('Verify your email');
  });
  
  it('rejects registration with duplicate email', async () => {
    await createUser({ email: '[email protected]' });
    
    const response = await api.post('/auth/register', {
      email: '[email protected]',
      password: 'SecurePass123!'
    });
    
    expect(response.status).toBe(409);
    expect(response.body.error).toContain('already exists');
  });
  
  it('validates password strength requirements', async () => {
    const response = await api.post('/auth/register', {
      email: '[email protected]',
      password: 'weak'
    });
    
    expect(response.status).toBe(400);
    expect(response.body.error).toContain('password');
  });
});
```text

**Unit tests** (optional, only if needed):
```typescript
// Only if password validation has complex rules
describe('Password Validator', () => {
  it('requires uppercase, lowercase, number, special char', () => {
    expect(validatePassword('Abc123!@#')).toBe(true);
    expect(validatePassword('alllowercase')).toBe(false);
    expect(validatePassword('ALLUPPERCASE')).toBe(false);
    expect(validatePassword('NoNumbers!')).toBe(false);
    expect(validatePassword('NoSpecial123')).toBe(false);
  });
});
```text

**Notice**: The integration tests verify:
- ✅ API contract (endpoint, request/response format)
- ✅ Database integration (user persisted correctly)
- ✅ Business logic (duplicate detection, validation)
- ✅ Side effects (email sent)
- ✅ Error handling (appropriate status codes and messages)

The unit test only verifies complex password rules that would be tedious to test via API calls.

## Why Integration Tests Have Higher Signal

### Information Theory Perspective

**Unit Test Information Content**:
```typescript
Entropy_unit = Information about single function behavior

Example:
  Test: calculateDiscount(100, 10) === 10
  Information: "calculateDiscount does math correctly"
  Scope: 1 function
  Lines verified: ~5
```text

**Integration Test Information Content**:
```text
Entropy_integration = Information about entire feature behavior

Example:
  Test: POST /checkout with discount code → 200 OK + reduced price
  Information: "Entire discount flow works: API → validation → calculation → database → response"
  Scope: 5-10 functions across 3 layers
  Lines verified: ~50-100
```text

**Information density**: Integration tests verify **10-20x more code** per test.

### Verification Efficiency

When verifying LLM-generated code:

**Unit test approach**:
```text
✅ Review 47 unit tests (30 minutes)
   → All pass
   → Assume feature works
   → Deploy to staging
   → Feature is broken (integration issue)
   → Debug for 2 hours
   → Fix and redeploy
   
Total time: 2.5 hours + broken staging environment
```text

**Integration test approach**:
```text
✅ Review 3 integration tests (5 minutes)
   → Test 2 fails (database column mismatch)
   → LLM fixes issue
   → Re-run tests
   → All pass
   → Deploy to staging
   → Feature works
   
Total time: 15 minutes + working feature
```typescript

**10x efficiency improvement** from prioritizing integration tests.

### Failure Detection Rate

**Common LLM failure modes** and detection rates:

| Failure Type | Unit Test Detection | Integration Test Detection |
|--------------|---------------------|----------------------------|
| Logic error in single function | ✅ 95% | ✅ 95% |
| Type mismatch at API boundary | ❌ 0% | ✅ 100% |
| Database schema mismatch | ❌ 0% | ✅ 100% |
| Missing error handling | ⚠️ 30% | ✅ 90% |
| Side effect not triggered | ❌ 0% | ✅ 100% |
| Business logic composition error | ❌ 10% | ✅ 95% |

**Integration tests catch 85%+ of real LLM failures**. Unit tests catch <40%.

## Practical Implementation

### Step 1: Set Up Integration Test Infrastructure

**Database**: Use test database with real schema
```typescript
// tests/setup.ts
import { beforeEach, afterEach } from 'vitest';
import { db } from '../src/db';

beforeEach(async () => {
  // Run migrations on test database
  await db.migrate.latest();
  // Seed essential data
  await db.seed.run();
});

afterEach(async () => {
  // Clean up test data
  await db.migrate.rollback();
});
```text

**API**: Use real API server in test mode
```typescript
// tests/helpers/api.ts
import { createServer } from '../src/server';

const testServer = createServer({
  database: process.env.TEST_DATABASE_URL,
  email: 'mock', // Mock email service
  payment: 'mock', // Mock payment service
});

export const api = supertest(testServer);
```text

**External Services**: Mock only external APIs
```typescript
// tests/mocks/email.ts
export const mockEmailService = {
  sent: [] as Email[],
  
  async send(email: Email) {
    this.sent.push(email);
  },
  
  getSent(recipient: string) {
    return this.sent.filter(e => e.to === recipient);
  },
  
  reset() {
    this.sent = [];
  }
};
```text

### Step 2: Write Integration Tests First

**Test-Driven Development with Integration Tests**:

```typescript
// 1. Write integration test that defines expected behavior
it('creates campaign and sends to subscribers', async () => {
  // Setup
  const user = await createUser({ email: '[email protected]' });
  await createSubscribers(['[email protected]', '[email protected]'], user.id);
  
  // Execute
  const response = await api
    .post('/campaigns')
    .auth(user.token)
    .send({
      subject: 'Test Campaign',
      body: 'Hello subscribers!',
      sendImmediately: true
    });
  
  // Verify API response
  expect(response.status).toBe(201);
  expect(response.body.campaign.id).toBeDefined();
  expect(response.body.campaign.status).toBe('sent');
  
  // Verify database state
  const campaign = await db.campaigns.findById(response.body.campaign.id);
  expect(campaign.sentAt).toBeDefined();
  
  // Verify side effects
  const emails = mockEmailService.getSent();
  expect(emails).toHaveLength(2);
  expect(emails.map(e => e.to)).toContain('[email protected]');
  expect(emails.map(e => e.to)).toContain('[email protected]');
});

// 2. Test fails (feature doesn't exist yet)
// 3. Ask LLM to implement feature to make test pass
// 4. Test passes → Feature is complete and verified
```text

### Step 3: Add Unit Tests Only for Complex Logic

**When to add unit tests**:

✅ **Do write unit tests for**:
- Complex algorithms (sorting, searching, data transformations)
- Mathematical calculations with edge cases
- String parsing/validation with many rules
- Business rule engines with complex conditions

❌ **Don't write unit tests for**:
- Simple CRUD operations
- Data mapping/transformation (DTO conversions)
- Straightforward validation (required fields, length checks)
- API route handlers (test via integration instead)

**Example: When unit tests add value**

```typescript
// Complex pricing algorithm with many rules
function calculateSubscriptionPrice({
  basePlan,
  addons,
  discounts,
  billingCycle,
  promoCode
}: PricingInput): Money {
  // Complex logic:
  // - Apply base plan price
  // - Add addon costs (some are per-user, some are flat)
  // - Apply percentage discounts (some stack, some don't)
  // - Apply billing cycle discount (annual = 20% off)
  // - Apply promo code (if valid and not expired)
  // - Ensure minimum price ($5/month)
  // ... (50+ lines of complex logic)
}

// This deserves unit tests!
describe('calculateSubscriptionPrice', () => {
  it('applies base plan price correctly', () => { /* ... */ });
  it('adds per-user addon costs', () => { /* ... */ });
  it('stacks compatible discounts', () => { /* ... */ });
  it('prevents stacking incompatible discounts', () => { /* ... */ });
  it('applies annual billing discount', () => { /* ... */ });
  it('validates promo code expiration', () => { /* ... */ });
  it('enforces minimum price floor', () => { /* ... */ });
  // 10-15 unit tests covering edge cases
});
```text

**But also write integration test**:
```typescript
it('creates subscription with correct pricing', async () => {
  const response = await api.post('/subscriptions').send({
    plan: 'pro',
    addons: ['advanced-analytics'],
    billingCycle: 'annual',
    promoCode: 'LAUNCH50'
  });
  
  // Verify final price is correct (integration across all layers)
  expect(response.body.subscription.totalPrice).toBe(4200); // $42/year
});
```text

Unit tests verify the algorithm is correct in isolation. Integration test verifies it works in the real system.

### Step 4: Use Integration Tests for Verification

**Trust But Verify Protocol**:

```typescript
// 1. Ask LLM to implement feature
// "Create user profile update endpoint with image upload to S3"

// 2. Ask LLM to write integration test
// "Write integration test that verifies profile update with image upload"

// 3. Run test
const result = await runTests('profile-update');

if (result.failed.length > 0) {
  // 4. LLM fixes issues based on test failures
  // "Fix the failing test: {failure message}"
  
  // 5. Repeat until tests pass
}

// 6. Review test output (not code)
// ✅ Test passed: User profile updated
// ✅ Test passed: Image uploaded to S3
// ✅ Test passed: Old image deleted from S3
// ✅ Test passed: Returns 400 for invalid image type

// 7. Feature is verified without reading hundreds of lines of code
```text

**Verification efficiency**:
- Reading test output: 30 seconds
- Reading implementation code: 10-20 minutes
- **40x faster verification** with integration tests

## Common Objections

### "Integration tests are slow"

**Myth**: Integration tests take minutes to run.

**Reality**: With modern tooling, integration tests run in seconds.

```typescript
// Slow (traditional approach):
- Spin up full application (30 seconds)
- Create real database (10 seconds)
- Run migrations (20 seconds)
- Run 1 test (2 seconds)
- Teardown (10 seconds)
= 72 seconds per test ❌

// Fast (modern approach):
- Use in-memory SQLite (0.1 seconds)
- Keep test server running (1 second startup, reuse)
- Run migrations once (2 seconds total)
- Run 1 test (0.5 seconds)
= 0.5 seconds per test ✅
```text

**Real benchmarks**:
```bash
# 47 unit tests:
$ npm test:unit
✅ 47 tests passed (0.3s)

# 3 integration tests:
$ npm test:integration
✅ 3 tests passed (1.2s)

Total: 1.5s for complete verification
```text

**Integration tests add <1 second**. Not a meaningful slowdown.

### "Integration tests are flaky"

**Myth**: Integration tests randomly fail.

**Reality**: Flakiness comes from poor test design, not integration testing itself.

**Common causes of flakiness**:

❌ **Shared test database** (race conditions)
```typescript
// Don't: All tests use same database
beforeAll(() => db.migrate.latest());

test('creates user', async () => {
  await api.post('/users').send({ email: '[email protected]' });
  // ❌ Fails if another test already created this user
});
```text

✅ **Isolated test database** (no race conditions)
```typescript
// Do: Each test gets fresh database
beforeEach(() => db.migrate.rollback().then(() => db.migrate.latest()));

test('creates user', async () => {
  await api.post('/users').send({ email: '[email protected]' });
  // ✅ Always works, database is clean
});
```typescript

❌ **Timing dependencies** (async race conditions)
```typescript
test('sends email after signup', async () => {
  await api.post('/signup').send({ email: '[email protected]' });
  expect(mockEmailService.sent).toHaveLength(1); // ❌ Email might not be sent yet (async)
});
```typescript

✅ **Explicit async handling** (deterministic)
```typescript
test('sends email after signup', async () => {
  await api.post('/signup').send({ email: '[email protected]' });
  await waitFor(() => mockEmailService.sent.length === 1); // ✅ Wait for async operation
  expect(mockEmailService.sent).toHaveLength(1);
});
```text

**Properly written integration tests are not flaky**.

### "Integration tests are hard to debug"

**Myth**: When integration tests fail, you don't know where the problem is.

**Reality**: Modern test frameworks provide excellent debugging.

**Debugging failed integration test**:

```typescript
test('creates campaign', async () => {
  const response = await api.post('/campaigns').send({ /* ... */ });
  
  // Test fails: expect(response.status).toBe(201)
  // Actual: 500
  
  // Debugging info available:
  console.log(response.status); // 500
  console.log(response.body);   // { error: "Database constraint violation" }
  console.log(response.headers); // Full headers
  
  // Check database state:
  const campaigns = await db.campaigns.findAll();
  console.log(campaigns); // Empty (campaign wasn't created)
  
  // Check logs:
  // Error: null value in column "user_id" violates not-null constraint
});
```text

**Root cause identified in 30 seconds**: Missing `user_id` in request.

**LLM fix**:
```typescript
// Before:
await api.post('/campaigns').send({ subject: 'Test' });

// After:
await api.post('/campaigns').auth(user.token).send({ subject: 'Test' });
//                           ^^^^^^^^^^^^^^^^^ Added authentication
```text

**Integration tests are easy to debug** with proper tooling.

### "I need 100% code coverage"

**Myth**: Unit tests are necessary for code coverage.

**Reality**: Integration tests provide code coverage too.

**Code coverage from integration test**:

```typescript
// Single integration test:
test('creates user and sends welcome email', async () => {
  const response = await api.post('/users').send({
    email: '[email protected]',
    password: 'SecurePass123!'
  });
  
  expect(response.status).toBe(201);
});

// Code coverage from this one test:
✅ POST /users route handler (10 lines)
✅ Request validation middleware (15 lines)
✅ Password hashing service (8 lines)
✅ User creation service (20 lines)
✅ Database repository (12 lines)
✅ Email service (10 lines)
✅ Welcome email template (25 lines)

Total: 100 lines covered by 1 integration test
```text

Compare to unit testing:
```typescript
// Need 7 separate unit tests to cover same code:
1. test('validates request body')
2. test('hashes password')
3. test('creates user in database')
4. test('sends email')
5. test('renders email template')
6. test('handles validation errors')
7. test('handles database errors')

Total: 7 unit tests to cover 100 lines
```text

**Integration tests provide better coverage per test**.

## When to Use Unit Tests

Unit tests still have value in specific scenarios:

### 1. Complex Algorithmic Logic

**Example**: Markdown parser with many edge cases

```typescript
function parseMarkdown(input: string): AST {
  // 200+ lines of complex parsing logic
  // - Headers (6 levels)
  // - Lists (nested, ordered, unordered)
  // - Code blocks (inline, fenced, indented)
  // - Links (inline, reference)
  // - Images
  // - Emphasis (bold, italic, strikethrough)
  // - Escape sequences
  // ... many edge cases
}

// Unit tests make sense here:
describe('parseMarkdown', () => {
  it('parses headers', () => { /* ... */ });
  it('parses nested lists', () => { /* ... */ });
  it('parses code blocks with language', () => { /* ... */ });
  it('parses inline code with backticks', () => { /* ... */ });
  it('handles escape sequences', () => { /* ... */ });
  // 50+ unit tests for edge cases
});
```text

**Why**: Too many edge cases to test via integration. Parser is isolated, testing it directly is efficient.

### 2. Mathematical/Financial Calculations

**Example**: Tax calculation with complex rules

```typescript
function calculateTax(income: Money, deductions: Deduction[]): Tax {
  // Complex tax brackets
  // - Federal tax (progressive brackets)
  // - State tax (varies by state)
  // - Deductions (standard vs itemized)
  // - Credits (child tax credit, etc.)
  // - Alternative minimum tax (AMT)
  // ... very complex logic
}

// Unit tests verify correctness:
describe('calculateTax', () => {
  it('applies federal tax brackets correctly', () => { /* ... */ });
  it('calculates AMT when applicable', () => { /* ... */ });
  it('applies child tax credit', () => { /* ... */ });
  // 20+ unit tests for tax scenarios
});
```text

**Why**: Tax calculations have well-defined inputs/outputs. Testing in isolation is clearer than through API.

### 3. Security-Critical Functions

**Example**: Cryptographic utilities

```typescript
function hashPassword(password: string, salt: string): string {
  // Uses bcrypt with specific rounds
  // Must be constant-time to prevent timing attacks
  // Must use cryptographically secure random for salt
}

// Unit tests verify security properties:
describe('hashPassword', () => {
  it('produces different hashes for same password with different salts', () => { /* ... */ });
  it('produces consistent hash for same password and salt', () => { /* ... */ });
  it('uses minimum 10 rounds', () => { /* ... */ });
  it('is constant-time (no timing attacks)', () => { /* ... */ });
});
```text

**Why**: Security requires precise verification of cryptographic properties.

### 4. Property-Based Testing

**Example**: URL slug generation

```typescript
function generateSlug(title: string): string {
  // Converts title to URL-safe slug
  // - Lowercase
  // - Replace spaces with hyphens
  // - Remove special characters
  // - Handle unicode
}

// Property-based test:
import { fc, test } from '@fast-check/vitest';

test.prop([fc.string()])("generateSlug produces valid URL slugs", (title) => {
  const slug = generateSlug(title);
  
  // Properties that should always hold:
  expect(slug).toMatch(/^[a-z0-9-]*$/); // Only lowercase alphanumeric and hyphens
  expect(slug).not.toMatch(/^-/);        // Doesn't start with hyphen
  expect(slug).not.toMatch(/-$/);        // Doesn't end with hyphen
  expect(slug).not.toMatch(/--/);        // No consecutive hyphens
});
```text

**Why**: Property-based tests generate thousands of random inputs. More efficient as unit tests.

## Measuring Success

### Metrics to Track

**1. Test-to-Code Ratio**

Target: 1:10 or better (1 test verifies 10+ lines of code)

```typescript
// Calculate:
Test-to-Code Ratio = Lines of Production Code / Number of Tests

// Example:
Production code: 1,000 lines
Unit tests: 200 tests (ratio = 5:1)  ❌ Too many tests
Integration tests: 50 tests (ratio = 20:1) ✅ Efficient
```text

**2. Failure Detection Rate**

Target: >90% of real bugs caught by tests

```typescript
// Track bugs that escape to production:
Bugs Found in Production: 5
Bugs Caught by Tests: 45
Detection Rate: 45 / (45 + 5) = 90% ✅

// Break down by test type:
Unit tests caught: 5 bugs (10%)
Integration tests caught: 40 bugs (80%) ← Highest value
E2E tests caught: 5 bugs (10%)
```text

**3. Verification Time**

Target: <5 minutes to verify entire feature

```typescript
// Measure time from "feature complete" to "verified"
Unit test approach: 30 min (read code + review tests)
Integration test approach: 2 min (run tests + review output)

15x faster with integration tests ✅
```text

**4. Test Maintenance Burden**

Target: <10% of development time spent updating tests

```typescript
// When refactoring code:
Unit tests broken: 30 (need updates)
Integration tests broken: 3 (need updates)

Maintenance time:
  Unit: 2 hours
  Integration: 15 minutes
  
8x less maintenance with integration tests ✅
```text

## Integration with Other Patterns

### Trust But Verify Protocol

Integration tests are the "verify" in Trust But Verify:

```typescript
// 1. Trust: LLM generates code
const code = await llm.generate('Create user profile endpoint');

// 2. Verify: LLM generates integration test
const test = await llm.generate('Write integration test for profile endpoint');

// 3. Run test
const result = await runTest(test);

// 4. If pass → Trust is validated ✅
// 5. If fail → LLM fixes code based on failure
```text

See: [Trust But Verify Protocol](./trust-but-verify-protocol.md)

### Quality Gates as Information Filters

Integration tests are high-value quality gates:

```typescript
State Space Reduction:

S₀ = All syntactically valid programs (1,000,000)
S₁ = After type checking (100,000)           // Removes 90%
S₂ = After linting (50,000)                  // Removes 50%
S₃ = After integration tests (100)           // Removes 99.8% ← Highest reduction!
S₄ = After E2E tests (10)                    // Removes 90%

Integration tests provide the largest state space reduction.
```text

See: [Quality Gates as Information Filters](./quality-gates-as-information-filters.md)

### LLM Recursive Function Model

Integration tests in the Verify phase:

```typescript
AI_Agent = fn(Verify(Generate(Retrieve())))

// Verify phase:
function verify(code: Code): Result {
  const integrationTestResults = runIntegrationTests(code);
  
  if (integrationTestResults.allPass) {
    return Ok(code); // ✅ Feature verified
  } else {
    return Err(integrationTestResults.failures); // ❌ Recurse with errors
  }
}
```text

See: [LLM as Recursive Function Generator](./llm-recursive-function-model.md)

### Test-Based Regression Patching

Integration tests prevent regressions:

```typescript
// Bug found: Discount codes don't work for annual plans

// 1. Write integration test that reproduces bug:
test('applies discount code to annual plan', async () => {
  const response = await api.post('/checkout').send({
    plan: 'pro',
    billingCycle: 'annual',
    discountCode: 'SAVE20'
  });
  
  expect(response.body.total).toBe(9600); // $96 ($120 - 20%)
  // ❌ Test fails (bug reproduced)
});

// 2. LLM fixes bug
// 3. Test passes ✅
// 4. Test permanently prevents regression
```text

See: [Test-Based Regression Patching](./test-based-regression-patching.md)

## Conclusion

For LLM-assisted development, **integration tests provide higher signal-to-noise ratio** than unit tests:

**Key Insights**:

1. **LLMs fail at integration points**, not isolated logic → Integration tests catch real failures
2. **1 integration test verifies 10-100 lines** of code → Higher information density
3. **Integration tests run in <1 second** with modern tooling → No performance penalty
4. **Verification time drops from 30 min to 2 min** → 15x faster
5. **Maintenance burden drops by 8x** → Fewer brittle tests

**Practical Recommendations**:

- ✅ **Write 1-3 integration tests per feature** (verify end-to-end behavior)
- ✅ **Write 0-2 unit tests per feature** (only for complex algorithms)
- ✅ **Use integration tests for verification** (Trust But Verify)
- ✅ **Run integration tests in CI** (quality gate before deploy)
- ❌ **Don't write unit tests for simple CRUD** (waste of time)
- ❌ **Don't skip integration tests** (highest-value verification)

**The Inverted Test Pyramid**:

```text
For LLM-assisted development:
  Few E2E tests (critical user journeys)
  MANY integration tests (every feature)
  Few unit tests (complex logic only)
  
This maximizes signal-to-noise ratio and verification efficiency.
```text

## Mathematical Foundation

$$\text{Signal-to-Noise Ratio} = \frac{\text{Lines Verified}}{\text{Number of Tests}} \quad \text{where} \quad \text{Integration} \approx 10-20 \times \text{Unit}$$

## Understanding Test Signal-to-Noise Ratio

The formula **Signal-to-Noise Ratio = Lines Verified / Number of Tests** measures the information efficiency of your test suite.

Let's break it down:

### **Signal-to-Noise Ratio** - Information efficiency metric

This measures **how much code verification** you get per test written.

**Higher ratio = Better**: Each test verifies more code, less test maintenance burden.

**Lower ratio = Worse**: Many tests for little verification, high maintenance cost.

### **Lines Verified** - Scope of code tested

**Lines Verified** = The number of production code lines executed by a test.

Examples:
- **Unit test**: Tests 1 function (5-10 lines) → Lines Verified ≈ 5-10
- **Integration test**: Tests entire feature (50-100 lines) → Lines Verified ≈ 50-100
- **E2E test**: Tests full user journey (200-500 lines) → Lines Verified ≈ 200-500

### **Number of Tests** - Test suite size

**Number of Tests** = How many test cases you wrote.

Examples:
- **Unit testing approach**: 47 unit tests for 1 feature
- **Integration testing approach**: 3 integration tests for same feature

### **The Comparison**

**Unit test signal-to-noise**:

Lines Verified per test: 5-10
Number of Tests: 47
Ratio: 5-10 / 47 ≈ 0.1-0.2 lines per test


**Integration test signal-to-noise**:

Lines Verified per test: 50-100
Number of Tests: 3
Ratio: 50-100 / 3 ≈ 16-33 lines per test

Udemy Bestseller

Learn Prompt Engineering

My O'Reilly book adapted for hands-on learning. Build production-ready prompts with practical exercises.

★ 4.5/5 rating

306,000+ learners

View Course


**Integration tests have 10-20x better signal-to-noise ratio** than unit tests.

### **Why This Matters**

When verifying LLM-generated code:

**Low signal-to-noise (unit tests)**:
- Review 47 test results
- Each test tells you about 5 lines of code
- Total: 47 data points to interpret
- Time: 20-30 minutes
- **High cognitive load** ❌

**High signal-to-noise (integration tests)**:
- Review 3 test results  
- Each test tells you about 50-100 lines of code
- Total: 3 data points to interpret
- Time: 2-5 minutes
- **Low cognitive load** ✅

### **The Formula in Practice**

Calculate for your own test suite:

```typescript
// Example calculation:
Production code: 1,000 lines
Unit tests: 200 tests
Integration tests: 25 tests

Unit test ratio:
  1,000 lines / 200 tests = 5 lines per test
  
Integration test ratio:
  1,000 lines / 25 tests = 40 lines per test
  
Improvement: 40 / 5 = 8x better signal-to-noise ✅

Target ratio: Aim for >20 lines verified per test through integration testing.

Information Theory Connection

Information content of a test = How much uncertainty it reduces about code correctness.

Unit test:

Reduces uncertainty about 1 function
Information ≈ log₂(possible_bugs_in_5_lines) ≈ 3-4 bits

Integration test:

Reduces uncertainty about entire feature
Information ≈ log₂(possible_bugs_in_100_lines) ≈ 7-8 bits

Integration tests provide 2x more information per test through broader verification scope.

Summary

The signal-to-noise formula shows that:

Integration tests verify more code per test (50-100 lines vs 5-10)
Fewer integration tests needed (3 vs 47) for same coverage
Better verification efficiency (10-20x ratio improvement)
Lower maintenance burden (fewer tests to update)

This mathematical advantage makes integration tests the optimal verification strategy for LLM-assisted development.

References

Test Pyramid – Martin Fowler’s classic article on the test pyramid (we invert it for LLMs)
Integration Testing Best Practices – Comprehensive guide to integration testing approaches

The Verification Ladder – Integration tests as Level 4 in the verification hierarchy
Verification Sandwich Pattern – Establish baseline before and after integration testing
Property-Based Testing – Combine with integration tests for comprehensive coverage
Test-Driven Prompting – Write integration tests before generating code
Stateless Verification Loops – Prevent state accumulation in integration tests
Trust But Verify Protocol – AI-generated integration tests for verification

Integration Testing Patterns

Integration Over Unit Tests: Higher Signal-to-Noise for LLM Verification

Summary

The Problem

The Solution

The Problem with Unit Tests for LLM Code

Unit Tests Verify the Wrong Thing

The Signal-to-Noise Problem

Learn Prompt Engineering

Information Theory Connection

Summary

References

Related

More Insights

Thought Leaders

Systems Thinking & Observability