James Phoenix

Agentic Tool Detection: Preventing LLM Tool-Use Failures

James Phoenix
James Phoenix

The Problem

AI coding agents are powerful because they can use tools—executing shell commands, reading files, making API calls, running tests, and more. But there’s a critical failure mode:

Agents assume tools are available when they’re not.

Common Failure Scenarios

Scenario 1: Missing CLI Tools

Agent: "I'll use ripgrep to search the codebase"
> rg "function authenticate"
Error: command not found: rg

Agent: "Let me try again with rg..."
> rg "function authenticate"
Error: command not found: rg

Agent: "Hmm, still trying rg..."
> rg --json "function authenticate"
Error: command not found: rg

Cost: 3+ failed attempts, wasted tokens, no progress

Scenario 2: Unavailable API Features

Agent: "I'll use the GitHub API to create a pull request"
> gh pr create --title "Fix bug"
Error: gh: command not found

Agent: "Let me authenticate first..."
> gh auth login
Error: gh: command not found

Cost: Multi-step workflow fails midway through, partial work wasted

Scenario 3: Environment-Specific Limitations

Agent: "I'll deploy using gcloud CLI"
> gcloud run deploy api-service
Error: gcloud: command not found

User: "We're on AWS, not GCP"

Agent: "Oh, let me use gcloud differently..."
> gcloud --project=... run deploy
Error: gcloud: command not found

Cost: Wrong deployment approach, confusion, manual intervention needed

Why This Happens

LLMs are trained on vast codebases where tools like git, npm, docker, gh, gcloud, etc. are commonly available. They assume standard tools exist without checking.

Additionally:

  1. No built-in capability detection: LLMs don’t probe environment before using tools
  2. Optimistic execution: Agents try commands first, discover failures later
  3. Poor error recovery: After tool failures, agents often retry the same approach
  4. Context loss: By the time tool unavailability is clear, significant tokens are spent

The Cost

For individual workflows:

  • 3-5 failed attempts before giving up
  • 2,000-5,000 wasted tokens per failure
  • 2-5 minutes of developer time debugging

For teams (100 workflows/week):

  • 20-30 tool-related failures weekly
  • 100,000-150,000 wasted tokens monthly
  • 5-10 hours of cumulative debugging time

The Solution

Systematic tool detection: Probe for required capabilities before starting workflows.

Core Pattern

// 1. Define tool requirements
const requiredTools = [
  { name: 'git', check: 'git --version' },
  { name: 'node', check: 'node --version' },
  { name: 'npm', check: 'npm --version' },
];

const optionalTools = [
  { name: 'gh', check: 'gh --version', alternatives: ['hub', 'manual PR creation'] },
  { name: 'docker', check: 'docker --version', alternatives: ['podman'] },
];

// 2. Detect capabilities
const capabilities = await detectTools([...requiredTools, ...optionalTools]);

// 3. Fail fast if critical tools missing
if (!capabilities.git || !capabilities.node) {
  throw new Error(
    `Missing required tools: ${getMissingTools(capabilities, requiredTools).join(', ')}\n` +
    `Please install: https://docs.project.com/setup`
  );
}

// 4. Provide capability context to agent
const context = `
Available tools:
${formatCapabilities(capabilities)}

Use ONLY the available tools listed above.
For missing optional tools, use listed alternatives.
`;

// 5. Execute workflow with capability awareness
await runAgentWorkflow({ context, capabilities });

Implementation

Step 1: Define Tool Registry

Create a registry of all tools your workflows might use:

// tools-registry.ts
export interface ToolDefinition {
  name: string;
  check: string; // Command to verify availability
  required: boolean;
  alternatives?: string[];
  installDocs?: string;
}

export const TOOL_REGISTRY: ToolDefinition[] = [
  // Version control
  {
    name: 'git',
    check: 'git --version',
    required: true,
    installDocs: 'https://git-scm.com/downloads',
  },

  // Package managers
  {
    name: 'npm',
    check: 'npm --version',
    required: true,
    alternatives: ['yarn', 'pnpm'],
  },
  {
    name: 'yarn',
    check: 'yarn --version',
    required: false,
    alternatives: ['npm', 'pnpm'],
  },

  // Code search
  {
    name: 'ripgrep',
    check: 'rg --version',
    required: false,
    alternatives: ['grep', 'ag (silver searcher)'],
  },

  // GitHub CLI
  {
    name: 'gh',
    check: 'gh --version',
    required: false,
    alternatives: ['hub', 'manual PR creation via web UI'],
    installDocs: 'https://cli.github.com/',
  },

  // Cloud CLIs
  {
    name: 'gcloud',
    check: 'gcloud --version',
    required: false,
    alternatives: ['GCP Console', 'Terraform'],
  },
  {
    name: 'aws',
    check: 'aws --version',
    required: false,
    alternatives: ['AWS Console', 'Terraform'],
  },

  // Container tools
  {
    name: 'docker',
    check: 'docker --version',
    required: false,
    alternatives: ['podman', 'finch'],
  },

  // Testing tools
  {
    name: 'playwright',
    check: 'npx playwright --version',
    required: false,
    alternatives: ['cypress', 'selenium'],
  },
];

Step 2: Implement Detection Logic

// tool-detector.ts
import { exec } from 'child_process';
import { promisify } from 'util';

const execAsync = promisify(exec);

export interface ToolCapability {
  name: string;
  available: boolean;
  version?: string;
  alternatives?: string[];
}

export interface CapabilityReport {
  tools: Record<string, ToolCapability>;
  missingRequired: string[];
  missingOptional: string[];
}

export async function detectToolCapability(
  tool: ToolDefinition
): Promise<ToolCapability> {
  try {
    const { stdout } = await execAsync(tool.check, {
      timeout: 2000, // 2 second timeout
      shell: '/bin/bash',
    });

    return {
      name: tool.name,
      available: true,
      version: stdout.trim(),
      alternatives: tool.alternatives,
    };
  } catch (error) {
    return {
      name: tool.name,
      available: false,
      alternatives: tool.alternatives,
    };
  }
}

export async function detectAllCapabilities(
  tools: ToolDefinition[]
): Promise<CapabilityReport> {
  // Run checks in parallel
  const results = await Promise.all(
    tools.map((tool) => detectToolCapability(tool))
  );

  // Build capability map
  const capabilityMap: Record<string, ToolCapability> = {};
  for (const result of results) {
    capabilityMap[result.name] = result;
  }

  // Identify missing tools
  const missingRequired = tools
    .filter((t) => t.required && !capabilityMap[t.name].available)
    .map((t) => t.name);

  const missingOptional = tools
    .filter((t) => !t.required && !capabilityMap[t.name].available)
    .map((t) => t.name);

  return {
    tools: capabilityMap,
    missingRequired,
    missingOptional,
  };
}

Step 3: Generate Agent Context

Create clear context for the LLM agent:

// capability-formatter.ts
export function formatCapabilitiesForAgent(
  report: CapabilityReport
): string {
  const available = Object.values(report.tools)
    .filter((t) => t.available)
    .map((t) => `  ✅ ${t.name} (${t.version})`)
    .join('\n');

  const unavailable = Object.values(report.tools)
    .filter((t) => !t.available)
    .map((t) => {
      const alts = t.alternatives
        ? ` → Use instead: ${t.alternatives.join(' or ')}`
        : '';
      return `  ❌ ${t.name}${alts}`;
    })
    .join('\n');

  return `
# Available Tools

## ✅ Available (you can use these)
${available}

## ❌ Unavailable (do NOT attempt to use)
${unavailable}

**IMPORTANT**: Only use tools marked with ✅. For tools marked with ❌, use the listed alternatives.
`;
}

Step 4: Integrate with Agent Workflows

// agent-workflow.ts
import { detectAllCapabilities } from './tool-detector';
import { formatCapabilitiesForAgent } from './capability-formatter';
import { TOOL_REGISTRY } from './tools-registry';

export async function runAgentWorkflow(task: string) {
  // 1. Detect capabilities upfront
  console.log('🔍 Detecting available tools...');
  const capabilities = await detectAllCapabilities(TOOL_REGISTRY);

  // 2. Fail fast if required tools missing
  if (capabilities.missingRequired.length > 0) {
    throw new Error(
      `Missing required tools: ${capabilities.missingRequired.join(', ')}\n` +
      `Install them before running this workflow.`
    );
  }

  // 3. Warn about missing optional tools
  if (capabilities.missingOptional.length > 0) {
    console.warn(
      `⚠️  Optional tools unavailable: ${capabilities.missingOptional.join(', ')}`
    );
    console.warn('   Workflow will use alternatives where possible.');
  }

  // 4. Generate capability context for agent
  const capabilityContext = formatCapabilitiesForAgent(capabilities);

  // 5. Execute workflow with capability awareness
  const prompt = `
${capabilityContext}

---

# Task
${task}

Remember: Use ONLY the tools marked as available above.
`;

  return await executeAgentTask(prompt, capabilities);
}

Best Practices

1. Detect Early, Fail Fast

Run detection before starting expensive operations:

// ✅ Good: Detect before starting
async function deployToProduction() {
  const capabilities = await detectCapabilities();

  if (!capabilities.gcloud.available) {
    throw new Error('gcloud CLI required for deployment');
  }

  // Now run deployment workflow
  await runDeployment();
}

// ❌ Bad: Discover mid-deployment
async function deployToProduction() {
  await buildDockerImage(); // Takes 5 minutes
  await runTests(); // Takes 3 minutes
  await exec('gcloud run deploy...'); // FAILS: command not found
}

2. Cache Detection Results

Tool availability rarely changes during a session:

// Cache capabilities for session
let cachedCapabilities: CapabilityReport | null = null;

export async function getCapabilities(): Promise<CapabilityReport> {
  if (cachedCapabilities) {
    return cachedCapabilities;
  }

  cachedCapabilities = await detectAllCapabilities(TOOL_REGISTRY);
  return cachedCapabilities;
}

// Invalidate cache if needed
export function invalidateCapabilityCache() {
  cachedCapabilities = null;
}

3. Provide Actionable Error Messages

// ❌ Bad: Vague error
if (!capabilities.gh.available) {
  throw new Error('gh not found');
}

// ✅ Good: Actionable error
if (!capabilities.gh.available) {
  throw new Error(
    `GitHub CLI (gh) is required for this workflow.\n\n` +
    `Install it:\n` +
    `  macOS: brew install gh\n` +
    `  Linux: https://cli.github.com/manual/installation\n` +
    `  Windows: winget install GitHub.cli\n\n` +
    `Or use alternative: Create PR manually via GitHub web UI`
  );
}

4. Support Graceful Degradation

Use alternatives when optional tools are missing:

if (capabilities.ripgrep.available) {
  // Fast: use ripgrep
  await exec('rg "pattern" --json');
} else if (capabilities.ag.available) {
  // Fallback: use ag (silver searcher)
  await exec('ag "pattern" --json');
} else {
  // Last resort: use grep (slower but universal)
  await exec('grep -r "pattern" .');
}

5. Document Tool Requirements

Make requirements discoverable:

# Project Setup (README.md)

## Required Tools

- **git** - Version control
- **node** (v18+) - Runtime
- **npm** or **yarn** - Package management

## Optional Tools (recommended)

- **gh** - GitHub CLI for PR creation
  - Install: `brew install gh` (macOS) or `https://cli.github.com`
  - Alternative: Manual PR creation via web UI

- **ripgrep** - Fast code search
  - Install: `brew install ripgrep` (macOS) or `https://github.com/BurntSushi/ripgrep`
  - Alternative: Uses `grep` if unavailable

Real-World Examples

Example 1: PR Creation Workflow

async function createPullRequest(options: PROptions) {
  const capabilities = await getCapabilities();

  if (capabilities.tools.gh.available) {
    // Best: Use GitHub CLI
    console.log('✅ Using GitHub CLI to create PR');
    await exec(`gh pr create --title "${options.title}" --body "${options.body}"`);
  } else if (capabilities.tools.hub.available) {
    // Fallback: Use hub CLI
    console.log('⚠️  gh unavailable, using hub instead');
    await exec(`hub pull-request -m "${options.title}\n\n${options.body}"`);
  } else {
    // Manual fallback
    console.log('❌ No CLI available. Manual steps required:');
    console.log('1. Push your branch: git push origin ' + options.branch);
    console.log('2. Visit: https://github.com/owner/repo/compare/' + options.branch);
    console.log('3. Click "Create pull request"');
    console.log('4. Title: ' + options.title);
    console.log('5. Body: ' + options.body);

    // Return manual instructions instead of failing
    return {
      success: false,
      manual: true,
      instructions: [...],
    };
  }
}

Example 2: Deployment Workflow

async function deployApplication(env: 'staging' | 'production') {
  const capabilities = await getCapabilities();

  // Determine deployment method based on available tools
  if (capabilities.tools.gcloud.available) {
    console.log('✅ Deploying with gcloud (Google Cloud Run)');
    return await deployWithGCloud(env);
  } else if (capabilities.tools.aws.available) {
    console.log('✅ Deploying with aws CLI (AWS ECS)');
    return await deployWithAWS(env);
  } else if (capabilities.tools.docker.available) {
    console.log('✅ Building with docker, manual deploy needed');
    await exec('docker build -t app:latest .');
    console.log('Image built. Deploy manually to your platform.');
    return { success: false, manual: true };
  } else {
    throw new Error(
      'No deployment tools available. Install gcloud, aws, or docker.'
    );
  }
}

Example 3: Code Search Workflow

async function searchCodebase(pattern: string) {
  const capabilities = await getCapabilities();

  // Use fastest available search tool
  if (capabilities.tools.ripgrep.available) {
    // Fastest (100x faster than grep)
    return await exec(`rg "${pattern}" --json`);
  } else if (capabilities.tools.ag.available) {
    // Fast (10x faster than grep)
    return await exec(`ag "${pattern}" --json`);
  } else {
    // Fallback to universal grep (slow but always available)
    console.warn('⚠️  Using grep (slow). Install ripgrep for 100x speedup.');
    return await exec(`grep -r "${pattern}" .`);
  }
}

Integration with Claude Code

Pattern: Add Capability Context to CLAUDE.md

# CLAUDE.md

## Available Tools

This project uses the following tools. Check availability before use:

### Required
- git (version control)
- node v18+ (runtime)
- npm (package manager)

### Optional
- gh (GitHub CLI) - for automated PR creation
  - Alternative: Manual PR via web UI
- ripgrep - for fast code search
  - Alternative: grep (slower)
- docker - for containerization
  - Alternative: Manual builds

**Before using any tool, verify it's available with `which <tool>` or `<tool> --version`.**

Pattern: Claude Code Hooks

Use hooks to detect capabilities automatically:

Udemy Bestseller

Learn Prompt Engineering

My O'Reilly book adapted for hands-on learning. Build production-ready prompts with practical exercises.

★ 4.5/5 rating
306,000+ learners
View Course 
// .claude/hooks.json
{
  "pre-task": "node scripts/detect-capabilities.js"
}

// scripts/detect-capabilities.js
const { detectAllCapabilities } = require('./tool-detector');
const { TOOL_REGISTRY } = require('./tools-registry');

async function main() {
  const report = await detectAllCapabilities(TOOL_REGISTRY);

  if (report.missingRequired.length > 0) {
    console.error('❌ Missing required tools:', report.missingRequired);
    process.exit(1);
  }

  if (report.missingOptional.length > 0) {
    console.warn('⚠️  Missing optional tools:', report.missingOptional);
  }

  console.log('✅ Tool check passed');
}

main();

Measuring Success

Key Metrics

  1. Tool-Related Failure Rate

    Before: 20-30% of workflows fail due to missing tools
After: <5% failure rate
Improvement: 80%+ reduction

  2. Time to First Failure

    Before: 30-60 seconds (after significant token usage)
After: <5 seconds (immediate detection)
Improvement: 90% faster failure detection

  3. Wasted Token Count

    Before: 2,000-5,000 tokens per tool failure
After: <100 tokens (early detection)
Improvement: 95%+ reduction

  4. Developer Debugging Time

    Before: 2-5 minutes per failure
After: <30 seconds (clear error messages)
Improvement: 80%+ time savings

Common Pitfalls

Pitfall 1: Assuming Tools Are Available

Problem: Agent assumes gh is available without checking

Solution: Always detect before use

Pitfall 2: Poor Error Messages

Problem: “Command not found” without context

Solution: Provide installation instructions and alternatives

Pitfall 3: Re-detecting Every Time

Problem: Running detection for every operation (slow)

Solution: Cache results for session duration

Pitfall 4: No Graceful Degradation

Problem: Hard failure when optional tools missing

Solution: Implement fallback alternatives

Pitfall 5: Incomplete Tool Registry

Problem: Registry missing tools actually used by workflows

Solution: Audit workflows to build complete registry

Advanced Patterns

Pattern: Feature Flags Based on Capabilities

const features = {
  autoCreatePR: capabilities.gh.available,
  fastSearch: capabilities.ripgrep.available,
  containerBuilds: capabilities.docker.available,
};

// Adjust workflow based on available features
if (features.autoCreatePR) {
  await createPRAutomatically();
} else {
  await generatePRInstructions();
}

Pattern: Progressive Enhancement

// Start with basic capabilities
let searchSpeed = 'basic';

if (capabilities.ripgrep.available) {
  searchSpeed = 'fast';
} else if (capabilities.ag.available) {
  searchSpeed = 'medium';
}

console.log(`Code search speed: ${searchSpeed}`);

Pattern: Capability-Aware Prompts

const prompt = `
Available capabilities:
${formatCapabilities(capabilities)}

Task: ${userTask}

Plan your approach using ONLY available tools.
For unavailable tools, use listed alternatives.
`;

Conclusion

Agentic tool detection is a high-impact, low-effort pattern that prevents 80%+ of tool-related workflow failures.

Key Principles:

  1. Detect early: Check capabilities before starting workflows
  2. Fail fast: Immediate errors for missing required tools
  3. Clear messages: Actionable errors with installation instructions
  4. Graceful degradation: Fallback alternatives for optional tools
  5. Cache results: Avoid redundant detection checks

The Result: Reliable agent workflows that fail predictably with actionable feedback, not mysteriously mid-execution.

Related Concepts

Topics
Agent ReliabilityAgentic WorkflowsCapability DetectionClaude CodeError PreventionLlm AgentsReliabilityTool DetectionTool Use

More Insights

Cover Image for Thought Leaders

Thought Leaders

People to follow for compound engineering, context engineering, and AI agent development.

James Phoenix
James Phoenix
Cover Image for Systems Thinking & Observability

Systems Thinking & Observability

Software should be treated as a measurable dynamical system, not as a collection of features.

James Phoenix
James Phoenix