Error Messages as Training Data: Building Persistent Memory for LLMs

Summary

LLMs lack persistent memory and repeat the same mistakes across sessions. Maintain an ERRORS.md file documenting common errors with symptoms, bad patterns, fixes, prevention strategies, and frequency tracking. Include relevant sections in context before similar tasks to prevent recurring mistakes and build institutional knowledge.

The Problem

LLMs are stateless and have no memory across sessions, causing them to repeat the same mistakes (missing await, incorrect schemas, type mismatches) even after being corrected multiple times. Teams waste time debugging identical errors that could have been prevented if the LLM ‘remembered’ previous corrections.

The Solution

Create an ERRORS.md file documenting every recurring error with structured entries: symptom, bad pattern (code example), fix (correct code), root cause, prevention strategy (linting rules, type guards), and frequency count. Include relevant sections from ERRORS.md in context when starting similar tasks. Review monthly to identify high-frequency errors and implement automated prevention (custom linting rules, type guards, CI checks).

The Problem

LLMs are stateless. They don’t remember previous conversations or learn from past mistakes. Every chat session starts from scratch.

This creates a frustrating pattern:

Week 1: LLM makes Error X
→ You correct it
→ LLM fixes it

Week 2: LLM makes Error X again
→ You correct it again
→ LLM fixes it again

Week 3: LLM makes Error X AGAIN
→ You're frustrated
→ Time wasted on the same bug

Common recurring errors:

1. Missing await on Promises → UnhandledPromiseRejectionWarning
2. Incorrect Zod schemas → Runtime validation fails
3. Type mismatches → undefined errors in production
4. Missing null checks → Crashes on edge cases
5. Incorrect async patterns → Race conditions
6. Wrong database types → Query failures
7. Missing error handling → Unhandled exceptions

The cost:

• Time waste: 5-15 minutes per repeated error
• Cognitive load: Context switching to debug
• Frustration: Why doesn’t it remember?
• Lost productivity: Could be building features instead

For a team of 5 developers:

5 developers × 3 recurring errors/week × 10 min each = 150 min/week wasted
= 10 hours/month = 120 hours/year = $12,000 in lost productivity

The Solution

Create persistent memory for your LLM through documentation: an ERRORS.md file that serves as a training dataset.

Core Concept

Every time the LLM makes a mistake:

1. Document it in ERRORS.md
2. Include it in context for future similar tasks
3. Track frequency to identify patterns
4. Implement prevention for high-frequency errors

The LLM can’t remember, but your documentation can.

The ERRORS.md Structure

# Common Errors & Solutions

Last Updated: 2025-11-03
Total Errors Documented: 23

## Error: Missing await on Promises

**Frequency**: 12 occurrences (Jan-Oct 2025)
**Severity**: High (causes production crashes)
**Last Occurrence**: 2025-10-15

**Symptom**: 
- `UnhandledPromiseRejectionWarning` in logs
- Function returns `Promise { <pending> }` instead of actual value
- `undefined` errors when trying to access properties

**Bad Pattern**:
```typescript
// ❌ Missing await - Promise not resolved
const user = getUserById(id);
console.log(user.email); // undefined - user is a Promise!

if (user.role === 'admin') { // Always false
  // This never runs
}

Correct Fix:

// ✅ Await the Promise
const user = await getUserById(id);
console.log(user.email); // Works correctly

if (user.role === 'admin') {
  // This works as expected
}

Root Cause:
Forgetting that database calls, API requests, and file I/O return Promises that must be awaited.

Prevention Strategy:

1. Enable @typescript-eslint/no-floating-promises ESLint rule
2. Use @typescript-eslint/require-await for async functions
3. Add type checking to catch Promise misuse
4. Include this example in context when working with async code

Related Files:

• src/services/user-service.ts (fixed 2025-10-15)
• src/api/auth.ts (fixed 2025-09-20)

---

Error: Incorrect Zod schema for database types

Frequency: 8 occurrences (Jan-Oct 2025)
Severity: Medium (runtime validation fails)
Last Occurrence: 2025-09-28

Symptom:

• Runtime error: Expected string, received object
• Validation fails for database timestamps
• Type coercion missing

Bad Pattern:

// ❌ Schema expects string, but DB returns Date object
const UserSchema = z.object({
  id: z.string(),
  email: z.string(),
  created_at: z.string(), // Wrong! DB returns Date
});

const user = await db.query.users.findFirst();
const validated = UserSchema.parse(user); // ❌ Fails!

Correct Fix:

// ✅ Use coerce.date() for database timestamps
const UserSchema = z.object({
  id: z.string(),
  email: z.string(),
  created_at: z.coerce.date(), // Handles Date objects
});

const user = await db.query.users.findFirst();
const validated = UserSchema.parse(user); // ✅ Works!

Root Cause:
Misunderstanding how database clients return date/time types (usually Date objects, not ISO strings).

Prevention Strategy:

1. Document database type mappings in CLAUDE.md
2. Create reusable schema patterns for common DB types
3. Add integration tests that validate schema against actual DB responses
4. Use z.coerce.date() by default for all timestamp fields

Related Files:

• src/db/schema.ts (fixed 2025-09-28)
• src/api/posts.ts (fixed 2025-08-15)

---

Error: Missing null checks cause production crashes

Frequency: 15 occurrences (Jan-Oct 2025)
Severity: Critical (production outages)
Last Occurrence: 2025-10-20

Symptom:

• Cannot read property 'X' of null in production logs
• 500 errors for edge cases
• Crashes when optional data is missing

Bad Pattern:

// ❌ No null check - crashes if user not found
const user = await getUserById(id);
return {
  name: user.name, // Crash if user is null!
  email: user.email,
};

Correct Fix:

// ✅ Always check for null/undefined
const user = await getUserById(id);

if (!user) {
  return {
    success: false,
    error: 'User not found',
  };
}

return {
  success: true,
  data: {
    name: user.name,
    email: user.email,
  },
};

Root Cause:
Assuming database queries always return data, ignoring the possibility of null/undefined.

Prevention Strategy:

1. Use TypeScript strict null checks (strictNullChecks: true)
2. Return Result types that encode success/failure
3. Never allow null to propagate – handle at boundaries
4. Add custom ESLint rule to detect missing null checks

Related Files:

• src/api/users.ts (fixed 2025-10-20)
• src/services/posts.ts (fixed 2025-10-12)
• src/utils/helpers.ts (fixed 2025-09-30)

## Implementation

### Step 1: Create ERRORS.md

Start with a template:

```markdown
# Common Errors & Solutions

Last Updated: [DATE]
Total Errors Documented: 0

## How to Use This File

1. When LLM makes a mistake, document it here
2. Include relevant sections in context for similar tasks
3. Update frequency counts when errors recur
4. Review monthly to implement prevention strategies

## Error Template

**Frequency**: X occurrences
**Severity**: Critical/High/Medium/Low
**Last Occurrence**: YYYY-MM-DD

**Symptom**: What you observe
**Bad Pattern**: Code that causes the error
**Correct Fix**: Code that solves it
**Root Cause**: Why this happens
**Prevention Strategy**: How to prevent automatically
**Related Files**: Where this occurred

---

Step 2: Document Errors as They Occur

When the LLM makes a mistake:

# 1. LLM generates code with bug
# 2. Tests fail or you catch it in review
# 3. Document it:

## Error: [Descriptive name]

**Frequency**: 1 occurrence
**Severity**: [Critical/High/Medium/Low]
**Last Occurrence**: $(date +%Y-%m-%d)

**Symptom**: [What happened]

**Bad Pattern**:
```text
[Code that was wrong]

Correct Fix:

[Code that fixed it]

Root Cause: [Why this happened]

Prevention Strategy: [How to prevent]

Related Files: [Where this occurred]

### Step 3: Include in Context

Before starting a task, identify relevant error patterns:

```markdown
Task: Implement user authentication API endpoint

Relevant context from ERRORS.md:

---

## Error: Missing await on Promises
[Full error documentation]

## Error: Missing null checks
[Full error documentation]

## Error: Incorrect error handling patterns
[Full error documentation]

---

Now, implement the authentication endpoint, avoiding these documented errors.

Step 4: Track Frequency

Each time an error recurs:

Udemy Bestseller

Learn Prompt Engineering

My O'Reilly book adapted for hands-on learning. Build production-ready prompts with practical exercises.

★ 4.5/5 rating

306,000+ learners

View Course

## Error: Missing await on Promises

**Frequency**: 12 occurrences → 13 occurrences
**Last Occurrence**: 2025-10-15 → 2025-11-03

[Rest of documentation...]

Step 5: Implement Prevention (Monthly Review)

Review ERRORS.md monthly:

# Generate frequency report
grep -A 1 "Frequency:" ERRORS.md | sort -t: -k2 -nr

# Output:
# Missing null checks: 15 occurrences
# Missing await: 12 occurrences  
# Incorrect Zod schemas: 8 occurrences
# Type mismatches: 6 occurrences

For high-frequency errors (>5 occurrences), implement automated prevention:

Example 1: Custom ESLint Rule

// .eslint/rules/no-missing-null-check.js
module.exports = {
  meta: {
    type: 'problem',
    docs: {
      description: 'Require null checks after database queries',
    },
  },
  create(context) {
    return {
      // Detect: const x = await db.query.X.findFirst()
      // Require: if (!x) { ... }
      AwaitExpression(node) {
        if (isDbQuery(node)) {
          const nextNode = getNextStatement(node);
          if (!isNullCheck(nextNode)) {
            context.report({
              node,
              message: 'Add null check after database query (see ERRORS.md)',
            });
          }
        }
      },
    };
  },
};

Example 2: Type Guard Utility

// src/utils/type-guards.ts
/**
 * Forces null checks at compile time.
 * Documented in ERRORS.md: "Missing null checks"
 */
export function assertDefined<T>(
  value: T | null | undefined,
  message: string
): asserts value is T {
  if (value === null || value === undefined) {
    throw new Error(`Assertion failed: ${message}`);
  }
}

// Usage (prevents "Missing null checks" error):
const user = await getUserById(id);
assertDefined(user, 'User not found'); // Type error if missing
// Now TypeScript knows user is not null
console.log(user.email);

Example 3: CI Check

# .github/workflows/prevent-common-errors.yml
name: Prevent Common Errors

on: [pull_request]

jobs:
  check-errors:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      
      - name: Check for floating promises
        run: |
          # From ERRORS.md: "Missing await" occurs 12 times
          npm run lint -- --rule @typescript-eslint/no-floating-promises:error
      
      - name: Check for missing null checks
        run: |
          # From ERRORS.md: "Missing null checks" occurs 15 times
          npm run lint -- --rule @custom/no-missing-null-check:error

Workflow Integration

Before Starting a Task

1. Identify task category: API, database, UI, etc.
2. Search ERRORS.md: Find related errors
3. Include in prompt:

Task: Implement post creation API endpoint

Relevant errors to avoid (from ERRORS.md):

1. Missing await on database calls
2. Missing null checks on user lookup
3. Incorrect Zod schema for timestamps
4. Missing transaction handling

Implement the endpoint following best practices and avoiding these documented errors.

After Completing a Task

Run tests and checks:

# Run tests
npm test

# If tests fail with recurring error pattern:
# 1. Fix the error
# 2. Update ERRORS.md frequency count
# 3. Ask: "Can we prevent this automatically?"

Monthly Review Process

# Monthly Error Review Checklist

## 1. Generate Frequency Report
- [ ] Sort errors by frequency
- [ ] Identify top 3 most common errors

## 2. Implement Prevention
For each high-frequency error (>5 occurrences):
- [ ] Can we add an ESLint rule?
- [ ] Can we add a type guard?
- [ ] Can we add a CI check?
- [ ] Can we improve documentation?

## 3. Update Documentation
- [ ] Add prevention strategies to ERRORS.md
- [ ] Update CLAUDE.md with new patterns
- [ ] Share learnings with team

## 4. Measure Impact
- [ ] Compare this month vs last month
- [ ] Calculate time saved
- [ ] Identify remaining gaps

Real-World Example

Week 1: First Occurrence

Error: Missing await on database query

// Bug introduced
const user = getUserById(id); // Missing await
if (user.role === 'admin') { // Always false - user is Promise
  return { admin: true };
}

Action: Document in ERRORS.md

## Error: Missing await on Promises

**Frequency**: 1 occurrence
**Last Occurrence**: 2025-01-15

[Full documentation...]

Week 3: Second Occurrence

Error: Same mistake, different file

Action: Update frequency

## Error: Missing await on Promises

**Frequency**: 1 occurrence → 2 occurrences
**Last Occurrence**: 2025-01-29

Week 8: Fifth Occurrence

Error: Happened 5 times total

Action: Implement prevention

// .eslintrc.json
{
  "rules": {
    "@typescript-eslint/no-floating-promises": "error"
  }
}

Result: Linter now catches this automatically

Week 12: Review

Before prevention: 5 occurrences in 8 weeks (0.6/week)
After prevention: 0 occurrences in 4 weeks (0/week)

Time saved: 5 min/error × 0.6 errors/week × 4 weeks = 12 minutes

Next steps: Implement prevention for next highest-frequency error

Measuring Success

Key Metrics

1. Error Frequency Over Time

Jan 2025: 23 total errors
Feb 2025: 18 total errors (-22%)
Mar 2025: 12 total errors (-33%)
Apr 2025: 8 total errors (-33%)

Target: 30% monthly reduction through prevention

2. Prevention Coverage

Total documented errors: 23
Errors with automated prevention: 8
Coverage: 35%

Target: 50%+ coverage for high-frequency errors

3. Time Saved

Average time per error: 10 minutes
Errors prevented per month: 15
Time saved: 150 minutes/month = 2.5 hours/month = 30 hours/year

For team of 5: 150 hours/year = $15,000 saved

4. Recurrence Rate

Before ERRORS.md: 60% of errors recur within 30 days
After ERRORS.md: 15% of errors recur within 30 days

Reduction: 75% fewer recurring errors

Best Practices

1. Document Immediately

Don’t wait – document errors as soon as you catch them:

✅ Good: Document within 5 minutes of fixing
❌ Bad: "I'll document it later" (you won't)

2. Include Concrete Examples

Always show actual code:

✅ Good:
**Bad Pattern**:
```typescript
const user = getUserById(id);

❌ Bad:
“Missing await” (too vague)

### 3. Track Root Cause

Understand **why** the error happened:

```markdown
✅ Good:
**Root Cause**: Database client returns Date objects, not ISO strings

❌ Bad:
**Root Cause**: Schema was wrong

4. Link to Related Files

Help identify patterns across files:

**Related Files**:
- src/api/users.ts (2025-01-15)
- src/api/posts.ts (2025-02-20)
- src/api/comments.ts (2025-03-10)

**Pattern**: All API endpoints have this issue

5. Set Severity Levels

Prioritize prevention efforts:

Critical: Production outages → Prevent immediately
High: Runtime errors → Prevent within 1 week
Medium: Test failures → Prevent within 1 month
Low: Style issues → Document only

6. Review Monthly

Set a recurring calendar reminder:

1st of every month: Review ERRORS.md
- Generate frequency report
- Implement prevention for top 3 errors
- Update documentation
- Share learnings with team

Integration with Other Patterns

Combine with Institutional Memory (LEARNING.md)

ERRORS.md focuses on mistakes, while LEARNING.md focuses on successes:

ERRORS.md: What NOT to do (anti-patterns)
LEARNING.md: What TO do (patterns)

Both create persistent memory for LLMs

See: Institutional Memory Through LEARNING.md

Combine with Test-Based Regression Patching

Every documented error should have a test:

## Error: Missing null checks

**Prevention Strategy**:
1. Document in ERRORS.md ✓
2. Add test case:

```typescript
it('should handle null user gracefully', () => {
  const result = processUser(null);
  expect(result.success).toBe(false);
  expect(result.error).toContain('User not found');
});

**See**: [Test-Based Regression Patching](test-based-regression-patching)

### Combine with Custom ESLint Rules

Convert frequent errors into linting rules:

ERRORS.md frequency report:

1. Missing await: 12 occurrences
2. Missing null checks: 15 occurrences

Convert to ESLint rules:

1. @typescript-eslint/no-floating-promises
2. @custom/no-missing-null-check

**See**: [Custom ESLint Rules for Determinism](custom-eslint-rules-determinism)

## Common Pitfalls

### ❌ Pitfall 1: Too Vague

**Bad**:

```markdown
## Error: Code doesn't work

**Fix**: Made it work

Good:

## Error: Missing await causes UnhandledPromiseRejection

**Symptom**: `UnhandledPromiseRejectionWarning` in logs
**Bad Pattern**: [Actual code]
**Fix**: [Actual fix with code]

❌ Pitfall 2: Not Including Code Examples

Bad: “User lookup failed” (what code?)

Good: Show exact code that failed and exact fix

❌ Pitfall 3: Forgetting to Update Frequency

# Easy to forget:
**Frequency**: 3 occurrences # Never updated!

# Set reminder:
**Frequency**: 3 occurrences (last updated: 2025-03-01)

❌ Pitfall 4: Not Implementing Prevention

Documenting errors is good, preventing them is better:

✅ Document → Prevent → Never happens again
❌ Document → Document → Document → Still happening

❌ Pitfall 5: Ignoring Low-Frequency Errors

Even 1-2 occurrences are worth documenting:

Low-frequency today = High-frequency tomorrow
Documentation is cheap, debugging is expensive

Conclusion

LLMs can’t remember, but your documentation can.

ERRORS.md creates persistent memory:

1. Document every mistake the LLM makes
2. Include relevant errors in context for similar tasks
3. Track frequency to identify patterns
4. Prevent high-frequency errors through automation
5. Review monthly to measure impact

The result:

• 75% fewer recurring errors
• 30% monthly reduction in total errors
• 150+ hours saved per year per team
• Institutional knowledge that survives team changes

Start today:

# Create ERRORS.md
touch ERRORS.md

# Add template
echo "# Common Errors & Solutions\n\nLast Updated: $(date +%Y-%m-%d)\nTotal Errors Documented: 0" > ERRORS.md

# Next time LLM makes a mistake, document it

Your future self (and your team) will thank you.

Related Concepts

• Institutional Memory Through LEARNING.md – Document successes, not just failures
• Test-Based Regression Patching – Convert errors into permanent tests
• Custom ESLint Rules for Determinism – Automate prevention of frequent errors
• Quality Gates as Information Filters – How prevention strategies reduce entropy
• Five-Point Error Diagnostic Framework – Categorize errors by root cause before documenting
• Context Debugging Framework – ERRORS.md provides Layer 1 context for systematic debugging
• Clean Slate Trajectory Recovery – Use documented errors as constraints when starting fresh sessions

References

• TypeScript ESLint Rules – Collection of TypeScript-specific ESLint rules for preventing common errors
• Custom ESLint Rules Guide – Official guide for creating custom ESLint rules to prevent project-specific errors