LAP Benchmark v2: Measuring
API Documentation Compression
Efficacy for AI Coding Agents

0 Abstract

1 Key Findings

2 Methodology

5-Tier Documentation System

Each API specification is compiled into five documentation tiers that span the spectrum from no documentation to the most verbose original format:

Scoring Rubric

Each run is scored on a 0-1 scale using three weighted components:

Total score = 0.6 * endpoint + 0.3 * params + 0.1 * code

Confounding Variable Mitigations

The benchmark applies several controls to isolate documentation quality as the independent variable:

• Neutral filenames: All documentation tiers are delivered as api_docs.txt to prevent tier/format leakage into the agent context
• Business-language tasks: All 100 task descriptions are phrased in domain language without endpoint-revealing technical terms (see Appendix B)
• No-doc baseline (none tier): Establishes agent prior knowledge without documentation
• Python-only mandate: Prompt enforces Python code output, eliminating language choice as a variable
• No library hints: Prompt specifies "appropriate libraries" without naming specific SDKs
• Agent isolation: Each run uses double-UUID nested temp directories to prevent cross-run contamination
• Timeout handling: All 500 runs completed successfully with a 360-second execution limit per run

Spec Selection - Full Benchmark (50 specs)

The full benchmark covers 50 real-world production APIs, 10 per format, chosen for diversity of size, domain, and schema complexity:

Run Statistics

The full benchmark executed 500 runs (50 specs x 5 tiers x 2 tasks). All 500 runs completed successfully (100%) with a 360-second execution time limit per run. This complete dataset provides full coverage across all spec-tier-task combinations with no missing data points.

3 Results - Tier Comparison

Average scores per documentation tier across all 500 completed runs.

Average Score per Tier

4 Results - Compression Analysis

How much does each tier compress the documentation compared to the pretty (original) baseline? Doc tokens are the tokens in the documentation file delivered to the agent. Score efficiency = avg score / (avg doc tokens / 1000). Averages across all 50 specs and both tasks.

Documentation Size by Tier (avg doc tokens, all 50 specs)

5 Results - Cost and Efficiency

Per-run cost, execution time, and token consumption by tier. Savings are calculated relative to the pretty (original format) baseline. The highlighted row (LAP-Lean) represents the recommended production tier. Total benchmark cost across all 500 completed runs: $130.68.

Average Cost per Run by Tier

6 Results - Wall Time Analysis

Execution wall time per run by tier and format. Lower times indicate faster agent completion. Wall time includes the full agent execution cycle (prompt processing, inference, code generation).

Average Wall Time per Run by Tier

Average Wall Time by Format and Tier

7 Results - Spec-Level Score Heatmap (50 Specs)

Average score per (spec, tier) pair, averaged across t1 and t2 tasks. Color coding indicates performance level. Missing cells ("-") indicate no completed runs for that combination (timed out).

8 Results - Format Comparison (5 Formats)

Performance comparison across all five API specification formats. Each format contains 10 specs. The benchmark covers OpenAPI (REST), AsyncAPI (event-driven), GraphQL (query language), Postman (collection format), and Protobuf (binary protocol).

OpenAPI

AsyncAPI

GraphQL

Postman

Protobuf

9 Results - Task Difficulty Comparison (t1 vs t2)

Each spec has two tasks (t1 and t2), both phrased in business language to avoid endpoint-revealing technical terms. This section examines whether task difficulty varies systematically between the two task slots across all 50 specs and 5 tiers.

10 Results - Code Quality Analysis

Code quality (10% of total score) measures whether the agent produced executable Python code containing the correct endpoints and parameters. Agents with documentation consistently produce better-structured code across all 50 specs.

Code Quality Score by Tier

11 Results - Score Distribution

Score range, central tendency, and variability per tier across all 500 completed runs. "% Perfect" = runs scoring exactly 1.0. "% Good" = runs scoring 0.7 or above. The sparkline shows a sample of individual run scores sorted ascending.

12 Statistical Analysis

12.1 Descriptive Statistics

With approximately 100 runs per tier (50 specs x 2 tasks), standard error and 95% confidence intervals are provided for transparency.

12.2 Paired t-Tests: Task Score

Paired t-tests (df=49, two-tailed) comparing mean score per spec across tiers. Each spec contributes one data point (mean of t1 and t2). Cohen's d measures effect size.

12.3 Paired t-Tests: Wall Time

Paired t-tests for wall time (seconds). Positive values mean the second tier is slower.

12.4 Paired t-Tests: Cost (USD)

Paired t-tests for per-run cost. Negative values mean the second tier is cheaper.

12.5 Paired t-Tests: Total Tokens

Paired t-tests for total tokens consumed per run. Negative values mean the second tier uses fewer tokens.

12.6 Documentation Lift (Tier Score - None Score)

How much does each tier improve over the no-documentation baseline, per spec? This isolates documentation's contribution from prior knowledge.

13 Results - Documentation-Sensitive Subset Analysis

Some APIs score ≥ 0.9 across all tiers including the no-documentation baseline, indicating the model has strong prior knowledge. These "ceiling-effect" specs contribute 1.0 to every tier's average, compressing observed differences and measuring API familiarity rather than documentation quality. This section excludes 5 such specs to isolate the documentation signal.

Excluded specs (none-tier ≥ 0.9): github-rest, plaid, slack, spotify, twilio

Full Set vs Documentation-Sensitive Subset (45 specs)

Paired t-Tests on Sensitive Subset

14 Limitations

This benchmark has several methodological limitations that should be considered when interpreting results:

15 Discussion

LAP-Lean: Best Efficiency Tier

LAP-Lean achieves the numerically highest average score of 0.851 compared to the pretty baseline (0.825), though this difference (+0.027) does not reach statistical significance (paired t-test, p > 0.05). The significant finding is efficiency: LAP-Lean delivers approximately 88% documentation token reduction and ~35% inference cost savings per run (p < 0.001 for both wall time and token reduction). For production AI coding workflows where agents are invoked at scale, LAP-Lean offers equivalent quality at significantly lower cost and latency. This result holds across all 5 formats and 50 APIs.

No-Doc Baseline: Documentation Is Essential

The no-documentation baseline scores 0.404 on average - substantially below all documented tiers across all 5 formats tested. The primary failure mode is endpoint identification (EP score): without documentation, agents cannot reliably identify the correct API endpoint path for unfamiliar or domain-specific APIs. This confirms that documentation is not merely helpful but essential for reliable AI agent performance. The none-tier effect is especially pronounced for Protobuf, AsyncAPI, and less well-known APIs where the model has limited training signal.

Minification Provides Limited Benefit Over Pretty

The minified tier achieves 0.835 vs 0.825 for pretty -- a negligible difference that does not reach statistical significance. Despite removing whitespace, minification of YAML/JSON/GraphQL does not meaningfully reduce semantic token count because structural keywords, field names, and values remain unchanged. LAP-format representations are fundamentally different: they reorganize information into a denser, agent-optimized structure rather than simply removing whitespace.

LAP-Standard: Intermediate Trade-off

LAP-Standard achieves 0.840 - essentially identical to LAP-Lean at 0.851 - while using approximately 82% fewer tokens than the pretty baseline. The near-identical scores between LAP-Standard and LAP-Lean suggest that for the tested task types, type information alone (LAP-Lean) is sufficient for correct task completion. However, LAP-Standard may be preferable for agents working with unfamiliar APIs where natural language endpoint descriptions provide additional disambiguation context, or for more complex multi-step tasks.

Cross-Format Robustness

The LAP format compression benefit is consistent across all 5 tested specification formats: OpenAPI (REST), AsyncAPI (event-driven), GraphQL (query language), Postman (collection), and Protobuf (binary protocol). This cross-format robustness is significant because each format has a fundamentally different structure and information organization. The LAP converter successfully normalizes all formats into a common endpoint-centric representation that preserves the information an AI agent needs for task completion.

16 Conclusion

This full benchmark (500 runs across 50 APIs, 5 formats, 5 tiers, 2 tasks) yields two clear findings:

1. Documentation is essential. Providing any form of API documentation dramatically improves agent performance (mean documented score ~0.84 vs none-tier 0.40, p << 0.001). This holds across all 5 specification formats. Agents cannot reliably identify API endpoints from training knowledge alone, particularly for domain-specific or less-prominent APIs.

2. Compression preserves quality while significantly reducing cost. No statistically significant quality difference was detected between any documented tier (paired t-tests, all p > 0.05). LAP-Lean achieved the numerically highest score (0.851 vs pretty 0.825), but this difference is directional only. The statistically significant advantages of LAP-Lean are efficiency: 88% documentation token reduction, 35% cost savings (p < 0.001), and 29s faster per run (p < 0.001).

Recommended Production Configuration

• Use LAP-Lean for high-volume, cost-sensitive AI coding agent deployments -- equivalent quality at significantly lower cost and latency
• Use LAP-Standard when working with complex or ambiguous APIs where descriptions provide additional disambiguation context
• Avoid relying on minification alone -- it provides minimal token savings compared to LAP format restructuring
• Always provide some documentation -- the no-doc baseline demonstrates a significant quality gap across all formats

Future Work

• Repetition trials: n >= 3 per condition to enable per-spec statistical significance testing
• Multiple models: Compare results across GPT-4o, Gemini Pro, Claude Opus, and local LLMs
• Randomized tier order: Eliminate potential ordering bias within spec runs
• Precision scoring: Add F1-based endpoint metric (penalize false positives, not just recall)
• Complex task types: Multi-step workflows, error handling, pagination, authentication
• Context window scaling: Test performance degradation curves as doc size approaches context limits

A Appendix A: Individual Runs

All 500 individual runs, sorted by format, spec, tier, task. All runs completed successfully.