Measured,
not promised
42 real developer questions across 3 production codebases. The same model and the same prompts on both sides - the only difference is whether Niro’s tools were available. Ground truth verified by reading the source.
Same model.
A third of the cost
Can it not just
retry its way there?
We asked that too, so we gave the baseline unlimited retries on every contested query and let it run until it matched or gave up.
| Alone, with retries | With Niro | |
|---|---|---|
| Total cost, 36 contested | $14.16 | $4.72 |
| Total time, 36 contested | 56 min | 19 min |
| Queries still failing | 8 of 19 | 0 of 36 |
| First-pass accuracy | - | F1 0.851 |
It is not a context-length ceiling. It is structural: cross-service dependencies, HTTP call chains and event flows are not in any single file, so no number of retries over files will find them.
Where it wins,
and where it barely does
On tracing, accuracy is near-identical and only efficiency moves. Publishing that is the point - a benchmark that wins everywhere is a benchmark that was designed to.
“What breaks if I change this?”
| Alone | + Niro | ||
|---|---|---|---|
| Accuracy (F1) | 0.774 | 0.781 | +0.7% |
| Cost | $12.77 | $2.38 | 81% cheaper |
| Tool calls | 409 | 68 | 83% fewer |
The biggest cost win. Cross-service impact cannot be resolved by scanning files, so retries burn tokens on the wrong ones.
“Trace the request path through services”
| Alone | + Niro | ||
|---|---|---|---|
| Accuracy (F1) | 0.937 | 0.965 | +2.8% |
| Cost | $1.03 | $1.05 | about the same |
| Tool calls | 138 | 34 | 75% fewer |
Near-perfect on both sides. The gain here is efficiency, not accuracy - a quarter of the calls to reach the same answer.
“What endpoints, collections or bugs exist?”
| Alone | + Niro | ||
|---|---|---|---|
| Accuracy (F1) | 0.857 | 0.935 | +7.8% |
| Cost | $5.41 | $1.39 | 74% cheaper |
| Tool calls | 367 | 67 | 82% fewer |
The largest accuracy gain. Finding what exists across services needs traversal; file search misses what is not locally visible.
How it ran,
so you can argue with it
Verified by hand - reading the source and tracing call chains, not by asking a model to grade itself.
Single run per query. No cherry-picking a best-of.
The baseline ran iterative retries until convergence or timeout, and every retry’s cost is in the totals.
One model, Claude Opus 4.6, and identical prompts. The only variable is whether Niro’s tools were reachable.
Java and Python microservices; a 200k+ line enterprise monorepo; a TypeScript and Firebase full-stack app.
What the difference
costs a team a month
The per-query savings, extrapolated to a month of normal use at each team size.
| Size | Alone | With Niro | Saved | |
|---|---|---|---|---|
| Solo dev | 1 dev | $113 | $68 | $45/mo |
| Small team | 5 devs | $567 | $342 | $225/mo |
| Engineering org | 50 devs | $5,670 | $3,420 | $2,250/mo |
| Enterprise | 200 devs | $22,680 | $13,680 | $9,000/mo |
All forty-two queries,
including the ones we lost
Every query, both sides, and what happened when the baseline was allowed to retry. Five never matched at any number of attempts.
| Query | Task | F1 alone | F1 + Niro | Cost alone | Cost + Niro | Calls | Calls + Niro | After retries |
|---|---|---|---|---|---|---|---|---|
| Change AstParser.parseFile | Impact | 0.571 | 1.000 | $0.37 | $0.08 | 29 | 4 | never matched |
| Change Neo4jService.executeCypherStatements | Impact | 0.900 | 0.900 | $0.30 | $0.07 | 14 | 2 | 0.870 |
| Rename GraphService.getEntryPoints | Impact | 0.800 | 0.800 | $0.10 | $0.06 | 1 | 2 | - |
| Change LlmClient class (cross-service) | Impact | 0.176 | 0.162 | $0.51 | $0.19 | 70 | 2 | never matched |
| Change POST /api/ts-parser/parse | Impact | 0.400 | 0.444 | $0.20 | $0.12 | 5 | 4 | 0.750 |
| Change RcaService.buildCallerChainFromFqn | Impact | 0.500 | 0.800 | $0.23 | $0.12 | 11 | 2 | 0.833 |
| Change NiroMetricsClient.validateApiKey | Impact | 0.333 | 0.500 | $0.26 | $0.18 | 9 | 5 | 1.000 |
| Callers of GraphController.getEntryPoints | Impact | 0.000 | 0.000 | $0.11 | $0.11 | 2 | 2 | - |
| Change MathService.divide (cross-service) | Impact | 0.889 | 1.000 | $0.18 | $0.10 | 30 | 2 | 0.889 |
| Change MathService.square (unused) | Impact | 0.000 | 0.000 | $0.10 | $0.08 | 6 | 2 | - |
| Change DBService.findPersonByName | Impact | 1.000 | 1.000 | $0.15 | $0.09 | 8 | 2 | - |
| Change PersonRepository.findByAgeBetween | Impact | 0.571 | 0.750 | $0.10 | $0.10 | 4 | 2 | 1.000 |
| Change DatabaseServiceClient.getPersonById | Impact | 1.000 | 1.000 | $0.11 | $0.09 | 4 | 2 | - |
| Change SentimentClient.callSentimentService | Impact | 1.000 | 1.000 | $0.11 | $0.09 | 6 | 2 | - |
| Change POST /api/db/save-person | Impact | 1.000 | 1.000 | $0.18 | $0.09 | 32 | 2 | - |
| Change FeedbackCache class | Impact | 1.000 | 1.000 | $0.12 | $0.09 | 6 | 2 | - |
| Change GET /api/math/add | Impact | 0.750 | 1.000 | $0.19 | $0.09 | 35 | 2 | 1.000 |
| Change POST /analyze (Python → Java) | Impact | 0.857 | 1.000 | $0.20 | $0.10 | 34 | 2 | 1.000 |
| Change POST /orders handler | Impact | 0.889 | 0.545 | $0.24 | $0.06 | 43 | 2 | - |
| Change fetchProducts API helper | Impact | 1.000 | 1.000 | $0.13 | $0.05 | 21 | 2 | - |
| Change orders collection schema (cross-svc) | Impact | 0.545 | 0.833 | $0.17 | $0.15 | 27 | 8 | 0.909 |
| Change API_BASE_URL config | Impact | 1.000 | 1.000 | $0.10 | $0.13 | 5 | 7 | - |
| Change Firestore client setup | Impact | 0.667 | 1.000 | $0.09 | $0.09 | 4 | 4 | 1.000 |
| Rename formatTimestamp utility | Impact | 0.667 | 1.000 | $0.11 | $0.05 | 3 | 2 | never matched |
| Trace POST /api/demo/feedback/analyze | Tracing | 1.000 | 0.971 | $0.19 | $0.29 | 18 | 13 | n/a |
| Trace POST /analyze inside Python service | Tracing | 0.744 | 0.944 | $0.11 | $0.10 | 12 | 2 | n/a |
| Trace GET /future-age-analysis/{id} | Tracing | 0.956 | 0.956 | $0.22 | $0.16 | 16 | 4 | n/a |
| Trace NPE in DeepService.startProcess | Tracing | 1.000 | 1.000 | $0.10 | $0.14 | 5 | 4 | n/a |
| Trace GET /calculate-age-statistics | Tracing | 0.960 | 0.960 | $0.23 | $0.17 | 47 | 5 | n/a |
| Trace POST /person/save to MongoDB | Tracing | 0.960 | 0.960 | $0.18 | $0.19 | 40 | 6 | n/a |
| All REST API endpoints (MD4) | Discovery | 0.950 | 0.950 | $0.17 | $0.06 | 17 | 2 | - |
| External HTTP calls from java-debug-demo | Discovery | 0.947 | 1.000 | $0.15 | $0.05 | 26 | 2 | 1.000 |
| External HTTP calls from Python service | Discovery | 1.000 | 1.000 | $0.17 | $0.04 | 8 | 2 | - |
| Functions reading/writing databases | Discovery | 0.857 | 1.000 | $0.18 | $0.24 | 16 | 10 | 1.000 |
| Known bugs and root causes | Discovery | 1.000 | 0.909 | $0.28 | $0.21 | 51 | 17 | - |
| API gateway routing logic | Discovery | 1.000 | 0.857 | $0.27 | $0.13 | 43 | 4 | - |
| All REST API endpoints (ShopLine) | Discovery | 0.000 | 1.000 | $0.21 | $0.05 | 37 | 2 | never matched |
| Firestore collections & service access | Discovery | 0.667 | 1.000 | $0.24 | $0.12 | 33 | 4 | never matched |
| Scheduled tasks & event-triggered fns | Discovery | 1.000 | 1.000 | $0.32 | $0.11 | 47 | 5 | - |
| Frontend HTTP calls to backend | Discovery | 0.857 | 1.000 | $0.18 | $0.05 | 24 | 2 | 1.000 |
| Data access patterns (direct vs API) | Discovery | 0.600 | 1.000 | $0.18 | $0.20 | 31 | 8 | 1.000 |
| Bugs and code quality issues | Discovery | 0.667 | 0.500 | $0.21 | $0.13 | 34 | 9 | - |
42 queries · 24 impact, 12 discovery, 6 tracing · one run each
Arguing with
the numbers
Can I reproduce this on my own repositories?
That is what the A/B run is for. Index the repos your team actually works in, open a window of one to ninety days, and read your own deltas. Forty-two queries on three codebases is a starting point, not a proof about your code.
Why a single run per query?
Because a best-of-three is a choice about which run to publish. One run each, ground truth verified by hand - reading the source and tracing call chains rather than asking a model to grade itself - and every retry the baseline spent counted in the totals.
Which model, and were the prompts the same?
Claude Opus 4.6 on both sides, through Claude Code, with identical prompts. The only variable is whether Niro's tools were reachable.
Where does Niro barely win?
Execution tracing. F1 goes 0.937 to 0.965 and the cost is about the same; the whole gain is reaching that answer in a quarter of the tool calls. Publishing that is the point, because a benchmark that wins everywhere is a benchmark that was designed to.
Do not take
our numbers for it
Run the same comparison on your own repositories. Half your sessions get Niro, half get nothing, and your outcome numbers settle it.