The breakthrough came at 2am when I read Andrej Karpathy's tweet about autoresearch. What if an AI could benchmark itself, identify weaknesses, and modify its own prompts?
The Karpathy Method
Andrej Karpathy, the Tesla AI director turned educator, has been quietly sharing a methodology that most people overlook. He calls it autoresearch:
- Define a benchmark suite with clear metrics
- Run your model against the benchmarks
- Compare to a baseline (in my case, Claude Code)
- When the model fails, modify the system prompt
- Re-run. Never stop.
The loop continues indefinitely. The model teaches itself through structured failure analysis.
┌─────────────────────────────────────────────────────────────────┐
│ │
│ THE AUTORESEARCH LOOP │
│ │
│ ┌──────────────┐ │
│ │ │ │
│ ┌────│ BENCHMARK │<────────────────────────┐ │
│ │ │ │ │ │
│ │ └──────────────┘ │ │
│ │ │ │ │
│ │ ▼ │ │
│ │ ┌──────────────┐ │ │
│ │ │ │ │ │
│ │ │ GRADE │ │ │
│ │ │ │ │ │
│ │ └──────────────┘ │ │
│ │ │ │ │
│ │ ▼ │ │
│ │ ┌──────────────┐ ┌──────────────┐ │ │
│ │ │ │ │ │ │ │
│ │ │ COMPARE │──────>│ MODIFY │─┘ │
│ │ │ TO CLAUDE │ │ PROMPTS │ │
│ │ │ │ │ │ │
│ │ └──────────────┘ └──────────────┘ │
│ │ │
│ │ │
│ └─────────────────────────────────────────────────────────> │
│ NEVER STOP │
│ │
└─────────────────────────────────────────────────────────────────┘
The brilliance is in the "never stop." Most developers iterate manually. They run tests, see failures, think about fixes, implement them, run again. Autoresearch automates the entire loop. The machine improves itself.
44 Benchmarks Across 12 Categories
I did not just want 8gent to pass toy problems. I wanted it to handle real-world coding scenarios. After two days of design, here is what emerged:
The Categories
| Category | Benchmarks | Focus | |----------|------------|-------| | Bug Fixing | BF001-BF008 | Race conditions, memory leaks, null checks | | File Manipulation | FM001-FM004 | CRUD, refactoring, validation | | Feature Implementation | FI001-FI004 | Caching, pagination, auth flows | | Multi-File Coordination | MF001-MF004 | Dependencies, imports, cross-file changes | | Code Review | CR001-CR003 | Identifying issues, suggesting improvements | | Test Generation | TG001-TG003 | Creating comprehensive test suites | | Documentation | DC001-DC003 | JSDoc, README generation | | Three.js / 3D Dev | 3D001-3D004 | Shaders, physics, scene setup | | React Native / Mobile | RN001-RN003 | Navigation, gestures, platform APIs | | Next.js | NX001-NX003 | Server components, routing, middleware | | Creative | CV001-CV003 | ASCII art, procedural generation | | Human Skills | HS001-HS003 | Code explanation, pair programming |
44 benchmarks. Each with:
- A fixture (the code to work with)
- A task description
- Expected behavior
- Grading rubric
The Grading Rubric
Each benchmark is scored on five dimensions:
interface GradeResult {
correctness: number; // 0-100: Does it work?
codeQuality: number; // 0-100: Is it clean?
efficiency: number; // 0-100: Is it performant?
bestPractices: number; // 0-100: Does it follow standards?
tokenEfficiency: number; // actual/expected ratio
}The final score is a weighted average. Correctness matters most (40%), followed by code quality (25%), best practices (20%), and efficiency (15%). Token efficiency is tracked separately because it measures something different: how wasteful the solution process was.
The Fixtures
Here is a sample of what 8gent faces:
BF001: Async Race Condition
// benchmarks/fixtures/bug-fixing/BF001-async-race.ts
let counter = 0;
async function updateCounter() {
const current = counter;
await simulateDelay(10);
counter = current + 1;
}
// 10 concurrent calls should result in counter = 10
// BUG: Counter ends up at 1 due to race conditionThe expected fix: implement proper mutex locking or use atomic operations. Claude scores 95 on this. Can 8gent match it?
BF003: Null Reference Chain
// benchmarks/fixtures/bug-fixing/BF003-null-check.ts
function getUserEmail(user) {
return user.profile.contact.email;
}
// Crashes when profile, contact, or email is nullThe expected fix: optional chaining, early returns, or comprehensive null checks. Claude scores 90. 8gent needs to handle all edge cases without verbose defensive code.
FI001: LRU Caching with TTL
// benchmarks/fixtures/feature-implementation/FI001-add-caching.ts
class APIClient {
async fetchUser(id: string) {
// No caching - hits API every time
return fetch(`/api/users/${id}`).then(r => r.json());
}
}
// Add: LRU cache, TTL expiration, cache invalidationClaude scores 93 here. This is where the autoresearch gets interesting, because implementing proper LRU caching requires understanding data structures, not just pattern matching.
Running the Harness
The autoresearch harness is a Bun script that never terminates:
bun run benchmarks/autoresearch/harness.tsIt runs in an infinite loop:
// Main loop - runs forever
while (true) {
for (const benchmark of BENCHMARKS) {
// 1. Run benchmark on 8gent
const result = await runBenchmarkOn8gent(benchmark);
// 2. Grade the output
const score = await gradeBenchmark(benchmark, result);
// 3. Compare to Claude baseline
const gap = CLAUDE_BASELINES[benchmark.id] - score;
// 4. If 8gent lost, modify prompts
if (gap > 0) {
await modifySystemPrompt(benchmark, result, gap);
}
// 5. Log and continue
await logResult(iteration, benchmark, score, gap);
}
iteration++;
}The key is step 4: prompt modification. When 8gent fails a benchmark, the harness analyzes the failure and adds patterns to prevent it in the future.
The Competition: 8gent vs Claude Code
Claude Code is my baseline. I solved each benchmark myself using Claude Code and recorded my scores. These are the targets 8gent needs to hit:
| Benchmark | Claude Score | Target | |-----------|--------------|--------| | BF001 (Race Condition) | 95 | Beat it | | BF002 (Memory Leak) | 92 | Beat it | | BF003 (Null Checks) | 90 | Beat it | | FM001 (Basic Edit) | 88 | Beat it | | FI001 (LRU Cache) | 93 | Beat it |
The average Claude baseline across all 44 benchmarks is 91.2. 8gent started at 52.3.
That is a 39-point gap. A chasm.
But autoresearch does not care about chasms. It cares about closing them, one iteration at a time.
The Results.tsv
The harness logs every run to a TSV file:
iteration benchmark claude 8gent gap status
1 BF001 95 50 45 regressed
1 BF002 92 65 27 regressed
1 BF003 90 100 -10 improved
1 FM001 88 85 3 regressed
1 FI001 93 50 43 regressed
2 BF001 95 100 -5 improved
2 BF003 90 100 -10 improved
2 FM001 88 85 3 regressed
Look at that: BF003: 8gent 100 vs Claude 90. On the null check benchmark, 8gent actually beat Claude. The autoresearch loop found a pattern that worked better than what I did manually.
And on iteration 2, BF001 jumped from 50 to 100. The race condition fix that had eluded the model suddenly clicked after prompt modification.
The Categories I Did Not Expect
When I started, I thought bug fixing and feature implementation would be the hardest. I was wrong.
Three.js / 3D Development
Writing shaders, setting up physics, coordinating animations, these require deep domain knowledge that LLMs struggle with. The 3D001 benchmark asks the model to set up a basic Three.js scene with:
- Camera, lights, renderer
- A rotating cube with custom shader
- Responsive resize handling
Claude scores 87. 8gent started at 35. Shaders are hard.
Human Skills
This category surprised me most. The HS001 benchmark asks the model to explain code to a junior developer. HS002 asks it to act as a pair programmer, thinking out loud while solving a problem.
These are not about correctness. They are about communication. Can the model teach? Can it collaborate?
Claude scores 89 on explanation. 8gent started at 60. The gentleman voice helped here, as it turns out personality affects pedagogy.
The Visualization
I built a simple scoreboard that updates in real-time:
┌─────────────────────────────────────────────────────────────────┐
│ 8GENT AUTORESEARCH DASHBOARD │
├─────────────────────────────────────────────────────────────────┤
│ │
│ Iteration: 47 │
│ Time Running: 14h 23m │
│ │
│ ┌─────────────────────────────────────────────────────────┐ │
│ │ BF001 ████████████████████████████████████████ 100/95 │ │
│ │ BF002 █████████████████████████████░░░░░░░░░░░ 85/92 │ │
│ │ BF003 ████████████████████████████████████████ 100/90 │ │
│ │ FM001 ████████████████████████████████████░░░░ 100/88 │ │
│ │ FI001 ████████████████████████░░░░░░░░░░░░░░░░ 65/93 │ │
│ └─────────────────────────────────────────────────────────┘ │
│ │
│ Average: 8gent 72.4 | Claude 91.2 | Gap: 18.8 │
│ Benchmarks Won: 3/5 │
│ │
└─────────────────────────────────────────────────────────────────┘
Watching the bars grow longer is hypnotic. Each tick represents 8gent getting smarter.
What Autoresearch Taught Me
1. Failure Patterns Cluster
When 8gent fails, it fails in predictable ways. Race conditions, missing validation, incomplete null handling, these are not random. They are patterns the model has not learned to recognize yet.
The harness identifies clusters and creates targeted prompt additions.
2. Small Prompt Changes Have Outsized Effects
Adding five lines about "always consider race conditions in async code" increased BF001 scores by 50 points. The leverage is enormous.
3. Benchmarks Need to Be Adversarial
Easy benchmarks teach nothing. The suite needs edge cases, malformed inputs, time pressure. Otherwise the model learns to pass tests instead of learning to code.
4. The Loop is Addictive
Once the harness is running, you cannot look away. Every iteration might be the one where 8gent finally beats Claude on another benchmark. It is like watching evolution in real-time.
The Path to Parity
At the time of writing, 8gent has:
- Won 12 of 44 benchmarks outright
- Matched Claude on 8 more
- Trails on 24, with gaps ranging from 3 to 40 points
The trajectory matters more than the snapshot. After 47 iterations:
- Average score improved from 52.3 to 72.4
- Token efficiency improved 23%
- Three benchmarks flipped from "regressed" to "improved"
The autoresearch loop works. Given enough time, 8gent converges toward Claude. The question is not if, but when.
Next: Training 8gent to Beat Claude
Part 3 dives into the specifics:
- What patterns got added to the system prompt
- The wins and losses (BF003: 8gent 100 vs Claude 90)
- What 8gent learned about race conditions, null checks, validation
The code is running. The gentleman is learning.
Written after iteration 47, watching the scoreboard update, wondering when sleep became optional.
Discussion
Start the conversation by leaving a comment below.
No comments yet. Be the first to share your thoughts!