Half your sessions get Niro.
The other half don’t.
A controlled experiment on your own repositories: same agent, same work, one arm without a single Niro tool. Your numbers decide, not ours.
One side of the split
has no Niro in it
A comparison is only worth anything if one side is genuinely without. In the off bucket the tool list is filtered down to one, so the agent works exactly as it did before Niro existed.
On: the full surface. Off: mark_task_complete alone, kept so the session still reports how it went. Everything else disappears from tools/list.
A second check rejects any other tool call while the session is in the off bucket, so a session that began before the window cannot leak past its assignment.
Both buckets record whether the task succeeded, partially succeeded or failed. That is the signal the two sides are compared on.
Deterministic,
not probabilistic
A session id is hashed with SHA-256 and the parity of the first byte picks the arm. The same id always lands in the same bucket, so a reconnect cannot quietly move a session mid-measurement. This runs the real calculation.
The delta,
on your own sessions
This is the shape of it. The figures below are an illustration - yours come from your own sessions.
The report computes raw deltas. It does not test statistical significance, and a fortnight of one team’s sessions is a signal rather than a proof. It is your signal though, from your repos and your work, which is worth more than a number from ours.
Set it up once,
then leave it alone
Nothing to instrument. The sessions your team already runs are the experiment.
One to ninety days, named. One active run per account, long enough to catch a normal week rather than a good afternoon.
Every MCP session is bucketed once at initialize and stamped for its lifetime. Nothing re-rolls mid-session.
Force-enable arms your next session into the on bucket for five minutes, so you can watch the difference rather than wait for the dice.
Ending a run marks it ended. Nothing is deleted, so every past run stays queryable and comparable.
Questions
before you start one
Does the control arm slow my team down?
It works exactly as it did before Niro existed - in the off bucket the tool list is filtered down to mark_task_complete alone, kept so the session still reports how it went. That is the cost of a real control. If someone needs the tools right now, force-enable arms their next session into the on bucket for five minutes.
Are the results statistically significant?
No, and the report does not pretend otherwise. It computes raw deltas on three figures: sessions, median session duration and median tool calls per session. A fortnight of one team's sessions is a signal rather than a proof - but it is your signal, from your repos and your work.
Can a session move between arms?
No. A session id is hashed with SHA-256 and the parity of the first byte fixes the arm at initialize, for that session's lifetime. A second gate rejects any other tool call while a session sits in the off bucket, so a reconnect cannot quietly move a session mid-measurement.
What happens to a run when it ends?
Ending a run marks it ended and nothing is deleted, so every past run stays queryable and comparable against the next one. One active run per account, over a window of one to ninety days.
Stop taking
our word for it
Index the repos your team actually works in, open a window, and read your own numbers in a fortnight.