Collect real work
The proxy records requests, context, tool actions, outcomes, and usage while the model does the job.
LEARN THE TASK. REUSE THE PROGRAM.
Turn repeated
tasks into code.
Spot a pattern. Write a program.
Learn when to use it.
Autotoken watches model traffic for repeated work, tries writing a program for it, and tests that program on new tasks. A decision model routes future requests to tested code or the main model. Each outcome feeds the next round of learning.
Open-source research · Built in the open
AUTOTOKEN / PROPOSED DESIGN↗
01 — INCOMING REQUEST
“Sort these numbers ascending.”
↓
Decision modelDoes a tested program fit this request?
↗MATCHRun codeRun a known program
ABSTAINCall the modelHandle the request
DETERMINISTIC OUTPUT
[1, 2, 9]The program sorts a new array.
OBSERVE → WRITE → TEST → ROUTE → LEARNIllustration, not a live model
01 / Install and try it
Run the proxy
on your machine.
You need Node.js 22.19 or later. Install from GitHub, then try the offline demo without an API key. Docker and Pi are not required.
The proxy records traffic. Installing it does not turn on automatic model-call replacement. Live requests use your provider account and its normal charges.
Install from GitHub
npm install -g github:hexapode/autotoken autotoken demo
The demo uses fixed examples to show the idea. It does not call a model or measure real savings.
1 / Configure and start
autotoken init # Edit .autotoken.env: # set LLM_API_KEY to your provider key autotoken proxy --provider openai
init creates a local configuration file with a random PROXY_TOKEN and adds the config and recording directory to .gitignore. Keep your provider key in that local file.
2 / Point your client at the proxy
Base URL: http://127.0.0.1:3001/v1 API key: your local PROXY_TOKEN Model: your usual provider model ID
Use the token from .autotoken.env as the client’s API key. The proxy uses your provider key upstream. Keep the client’s existing model choice and API format.
Anthropic, compatible providers, and Ollama
Use the provider’s native API format; the proxy does not translate between APIs.
# Anthropic (LLM_API_KEY or ANTHROPIC_API_KEY) autotoken proxy --provider anthropic # Anthropic SDK base URL: http://127.0.0.1:3001 # The SDK appends /v1/messages. # OpenAI-compatible provider autotoken proxy --provider compatible --upstream https://YOUR_PROVIDER/v1 # Local Ollama autotoken proxy --provider ollama
Chat Completions, Responses, and Anthropic Messages are supported. The model ID remains your client’s choice. See the README for setup details and the coding experiment guide for Pi and repository tests.
↳ The hypothesis
We want to use models to find repeated work and write programs for it. Then we need to check whether those programs work on new requests.
02 / The learning loop we want to build
Spot a pattern.
Write, test, route, repeat.
The system learns two things: how to do a recurring task with code, and when that code applies. We now have recording, a restricted program compiler, a shadow decision API, and a first router trained on traces. Reliable automatic deployment is still the goal.
Find a recurring task
Compare traces for repeated procedures across different inputs. A sequence of routine checks may be more useful than one repeated command.
Try a small program
A model writes a candidate with defined inputs, outputs, and conditions for use. For example: collect the patch, check whitespace, and report repository status.
Make it earn traffic
Test edge cases and new issues. Run it alongside the normal path in an isolated copy, then compare task outcomes, cost, and time. Reject programs that do not help.
Choose a program or model
A decision model sees the request and eligible programs. It returns a program ID and arguments, or falls back to the main model. Code checks the arguments and conditions before execution.
Use the outcome
Record whether the choice worked and what it cost. Use verified outcomes to improve programs and the router. Keep checking new versions on separate examples before promoting them.
Every outcome becomes another example. Failures suggest missing conditions or new tasks to automate. Successful matches teach the router when a program fits. A program can also be retired when it stops helping.
Start with a decision API
A small choice, with a fallback.
One endpoint receives the request, relevant context, and a shortlist of tested programs. It returns a structured choice. A model API can make this decision first; a local classifier can later serve the same interface.
{ "route": "program", "program": "patch_review_v1",
"arguments": { "scope": "current_patch" } }
{ "route": "model" }The router chooses from a registry. It does not invent executable commands. Invalid arguments, unmet conditions, or failed checks return control to the main model.
Then improve the decision
Learning does not require a fine-tune on every request.
Start by updating the program catalog and adding verified examples to the router’s prompt. Once we have enough labeled outcomes, train a small classifier and compare it with the API router.
The prototype can update a candidate classifier after each labeled replay outcome; the serving policy stays frozen. A candidate router still needs evaluation before replacing the active version. A model’s confidence or a command exiting successfully is not proof that the task was solved.
Measure the full bill: routing, program execution, fallback, discovery, and evaluation. A router only helps if those costs are lower at comparable task quality.
A small, concrete example
Two requests can use the same code.
“Sort these numbers.” “Put these values in ascending order.” A sorting program can handle either request with a new array.
The decision still requires care: “sort by absolute value” is a different task.
EXAMPLE: SORT AN ARRAY
intent sort numbers ascending
input [9, 2, 1]
program numeric sort · ascending
output [1, 2, 9]
New input, same procedure. No answer-model call.
Learning when to use code
A program does the work.
A model decides when it fits.
Finding repeated work is only half the problem. The system also needs to recognize when a new request can use the same program.
Start with a decision API
The router receives a request and a shortlist of tested programs. It chooses a program with valid inputs or sends the request to the main model.
For a coding assistant, a small program might collect the current patch, check whitespace, and report repository status in one step.
Learn from what happens
Verified outcomes become examples for a smaller router. New versions are tested on separate tasks before they can handle requests.
The prototype supports a model API or a local classifier. Learning from recorded outcomes is available; automatic deployment of learned coding workflows is still under development.
We are testing whether this reduces total cost while preserving task quality. Results are preliminary. Read the research notes ↗
From traces to programs
Which steps could
code take over?
We look for recurring work in recorded coding sessions, build a bounded program, and test whether it helps on a complete task.
Search with the source attached
A search result often leads straight to another request to read the surrounding code. Our program returns both together.
8 → 5 model calls · 8.8% fewer tokens in one live comparison. Both fixes passed repository tests and the same bug-specific check; the original code failed that check.
This was one previously seen issue, with one run per approach. Consistent savings across new tasks remain unproven. A fixed rule selected the program; the learned router is still in shadow mode.
See the loop in action
Follow a request from a repeated pattern to a program, its checks, and a routing decision. The search program executes in your browser. The routing choices are illustrative.
No typing, installation, or API key needed.
Three workflows to test next
We mined 300 public trajectories from three harnesses. These counts show when a program predicted the next recorded action from prior output. They are not measured saved calls.
Repair an oversized read
The editor reports that a requested range extends past the end of a file. A program uses that reported length to retry the read once.
42 of 49 next reads matched
34 exact requests and 8 equivalent reads to EOF. All 42 returned the requested source lines.
Go to the only match
When a search finds one line in the currently open file, the next navigation step can be determined from the search result.
22 of 23 next actions matched exactly
A related file-opening rule matched 39 of 62 next actions. Each matched read displayed the expected file.
Write and run a script
The agent writes a Python reproduction script, then asks to run it. An explicit write-and-run operation could combine those steps.
469 of 584 next commands matched exactly
Other follow-ups included setup or different interpreters. The prototype plans a run; it does not execute scripts automatically.
Three bounded planners pass 37 fixture checks. These are discovery results: matching a tool action does not prove that removing the model turn preserves its reasoning or fixes the issue. Read the methods and results ↗
Replay estimate · 300 trajectories
How many turns could code replace?
We checked four programs against 11,525 recorded assistant turns. There are 34 exact, command-only matches (0.30% of turns). Including equivalent reads and assuming accompanying reasoning can be omitted gives 572 potential replacements (4.96%).
The broader scenario includes 469 write-and-run steps, 61 navigation steps and 42 read repairs. Automatic script execution would require explicit opt-in. These are opportunities in discovery traces, not demonstrated saved calls.
Counted with tiktoken
What about tokens?
Counting visible text with cl100k_base gives roughly 1.02 million tokens for the strict scenario and 6.98 million for the broader scenario—0.42% and 2.87% of the estimated corpus total.
Most of that is context that would not need to be sent for a skipped request. This assumes full conversation history on each request. Actual usage, context truncation, caching and routing costs are unavailable; these are gross estimates, not billed savings.
18 program candidates · five models
Classify thousands of traces.
2,498 trajectories, 500 issues and 129,846 recorded responses. The classifiers run locally, with no model calls.
About 36 seconds for a full pass · 4 CPU workers
| Program | Matches / proposals | Potential tokens avoided |
|---|---|---|
| Write a script, then run it | 438 / 2,238 | 7,017,304 |
| Read a unique Python definition | 133 / 2,353 | 2,142,923 |
| Status after a syntax check | 42 / 382 | 1,113,702 |
| Syntax check after an edit | 30 / 1,574 | 949,349 |
| Continue a source read | 38 / 12,335 | 518,334 |
Combined: 0.59% of recorded calls / 0.40% of recorded tokens. The deduplicated total is 760 calls and 13.09 million tokens. These are conditional replay estimates. Removing model reasoning and preserving task results still need live testing; low proposal match rates rule out broad activation.
The runner evaluates each program independently and caches results by trace and source hashes. Wider-context opportunities stay separate because extra output costs are unknown. Run it yourself ↗ · All 18 candidates and percentages ↗ · Full batch report ↗ · Sources ↗
Programs with several steps
Can code carry a workflow further?
Each program keeps state between tool results and stops when its next step is uncertain. We tested three sequences across the same 2,498 coding runs.
| Workflow | First-step matches | Complete sequences |
|---|---|---|
| Find definition → read → expand a cut-off expression | 133 / 2,353 | 0 |
| Write reproduction → run → read failure context | 350 / 1,339 | 0 |
| Edit → compile → rerun a registered test | 1 / 83 | 0 |
133 new call matches / 2.14 million recorded tokens. All come from reading a unique Python definition. That adds 0.10 percentage points of call coverage and 0.07 points of token coverage. The reproduction and verification first steps overlap existing programs and add no new coverage.
The longer sequences remain unproven: none matched both steps. The investigation program proposed 31 expansions after detecting incomplete Python syntax, but none matched the next response. Of 350 reproduction runs, 149 hit import or environment failures—useful evidence for testing recovery with environment metadata.
These are local planners, not automatically enabled workflows. Exact agreement measures a potential opportunity; live routing, extra tool output and task outcomes determine actual savings. Counts, percentages and overlap audit ↗ · Full 18-program replay ↗
Earlier round: four workflow programs
Four new programs · same 2,498 traces
How much does another round add?
One agent explored each program, then tested its fixed rule on the larger corpus. Percentages below use all 3.24 billion recorded tokens as the denominator.
| Program | Matches / proposals | Corpus tokens potentially avoided |
|---|---|---|
| Status after a syntax check | 42 / 382 | 0.0343% |
| Syntax check after an edit | 30 / 1,574 | 0.0293% |
| Read the only matching small file | 31 / 4,191 | 0.0103% |
| Repair a missing search command | 9 / 11 | 0.0011% |
All eight programs: 0.44% of calls and 0.31% of tokens. The previous four covered 0.35% and 0.23%. This round adds 112 exact matches and 2.43 million tokens; the combined total is 571 calls and 9.91 million tokens across 240 issues. No targets overlap in this replay.
The missing-command repair matches 82% of its proposals but is rare. The other new rules match 0.7–11%, so enabling them broadly could add unnecessary work. These are candidates for selective routing, not proven net savings. Filesystem checks, reasoning and task outcomes still need live validation.
All eight are available in the same batch runner through --program. A full pass takes about 29 seconds with four workers. Percentages and overlap audit ↗ · Full batch report ↗ · Run a classifier ↗
Earlier round: source reads and patch review
Source reads and patch review
Three more workflows tested.
| Program | Matches / proposals | Corpus tokens potentially avoided |
|---|---|---|
| Continue a source read | 38 / 12,335 | 0.0160% |
| Review the edited file’s diff | 4 / 1,574 | 0.0038% |
| Read traceback source context | 0 / 862 exact | 0% |
All 11 programs: 0.47% of calls / 0.33% of tokens. This round adds 42 exact matches and 640,294 tokens, bringing the total to 613 calls and 10.55 million tokens. These are conditional gross estimates. Both matching rules predict the next call in fewer than 1% of proposals; broad activation would add unnecessary work.
Wider-context matches remain separate: 51 traceback reads and 489 read continuations could cover the requested lines, but extra context costs are unmeasured. All 11 classifiers run in about 33 seconds with four workers. Percentages and overlap audit ↗ · Full batch report ↗
Earlier round: search fallback and environment checks
Error recovery
Search fallback and environment checks.
| Program | Eligible matches / proposals | Corpus tokens potentially avoided |
|---|---|---|
| Broaden an empty search | 14 / 784 | 0.0122% |
| Retry a missing pytest command | 0 / 106 | 0% |
| Repair a missing file path | 0 supported proposals | 0% |
| Finish after registered verification | 0 supported proposals | 0% |
This round adds 14 exact matches and 396,689 tokens. All 12 observed module retries after a missing pytest command still lacked pytest, so they receive no repair credit. Missing paths had no unique correction in prior observations. No finish candidate met the strict registered-verification conditions; a passing test alone never implies completion.
Empty-search fallback matched only 1.8% of proposals. Its runtime scope limits remain unverified, so it needs selective routing and live validation. Read the accounting and zero results ↗
Structured test results · prototype tested
In 66 discovery turns, the agent shortened test output through a shell pipeline. Fifteen outputs contained failure evidence even though the outer shell reported success; some failures were intentional. Our program preserves the actual test-process status and returns bounded failure details. Local fixture checks pass. Model-call savings are unproven.
Guarded edits · prototype tested
A bounded edit program checks the file fingerprint and expected match count before replacing text. It supports the shape of 127 recorded replacement operations across 56 discovery turns. Refusal and successful-edit checks pass, but historical fingerprints remain unverified. Its explicit JSON inputs were 12.2% larger than the scripts in this comparison, so this is a reliability result, not token savings.
Original-versus-patched checks · prototype tested
We found ten comparison turns across six discovery issues. The program puts identical regression tests in separate original and patched copies. A synthetic bug fails before the fix and passes afterward, while the source worktree stays unchanged. Historical test overlays have not been reconstructed, and no saved calls are established.
Compact reads · smaller tool arguments
A tuple-based read operation used 24.9% fewer argument bytes across 24 supported discovery turns: 2,589 → 1,945 bytes. That is 644 bytes total. The model still chooses what to read; tool-schema costs, output overhead, and token savings have not been measured. Nine fixture checks pass.
Find related tests · lightweight filename hints
Using only previously mentioned source files and the original repository tree, a program suggested test filenames on 61 turns. On fifteen of those turns, a suggestion matched a file requested next. The hints total 2,343 bytes, but they do not replace reading tests or deciding what to change. Thirteen fixture checks pass.
Failure traceback context · low priority
Returning source around traceback locations produced 28 bounded plans, but covered no complete following turn in the discovery traces. Those plans could add up to 2,720 lines of context. The planner passes its checks; the evidence does not support automatic injection yet.
What we ruled out in the earlier local sample
Strict replay found no duplicate read ranges to remove. An audit of 140 Python snippets found only two unrelated inspection examples, too little for a reusable helper. A failed-test planner produced 22 proposals, but none matched the next recorded test selection. These checks found no additional savings in the local sample. The broader public corpus is evaluated separately above.
Beyond one harness
Our discovery corpus now includes a public sample of 300 trajectories and 11,525 assistant turns from SWE-agent, OpenHands, and mini-SWE-agent-plus, spanning 259 repositories.
These are imported research traces, not new runs or measured savings. Known local issues and reserved evaluation groups are excluded from the sample.
Different tools, different patterns
OpenHands often uses a structured editor; the other samples mostly use text commands. Comparing them helps us look beyond patterns caused by a single tool interface.
Each candidate needs an independent task comparison that includes program overhead, routing cost, and correctness. Methods and candidate evidence ↗
03 / First completed experiment
What happened when we
ran checks automatically?
We ran the same two coding tasks three ways: normally, with an automatic diff check, and with an automatic test-and-diff sequence.
Pi coding agent→Autotoken proxy→gpt-6.1-sol
Explore recorded runs
6 recorded runs · no live API callsChoose a task and an approach to see what was actually recorded. The differences below compare separate runs; they do not prove that these commands caused the change.
In each run, the coding agent had to fix a real bug in a fresh repository checkout. The approaches below change how it checks its patch.
Astroid: keep the code analyzer from crashing
Astroid examines Python code to infer what values expressions might produce. An expression such as "{:d}".format(None) made the analyzer itself crash.
A correct fix: handle formatting it cannot infer without crashing. Python’s formatting rules stay the same.
Our check: analyze two invalid formatting expressions without raising an exception, then run the repository tests.
pydicom: export valid data despite an invalid field
pydicom converts medical-image metadata to JSON. A malformed integer field could stop the export even with suppress_invalid_tags=True.
A correct fix: let export finish with suppression enabled and preserve valid fields. Without suppression, the error should still surface.
Our check: export a dataset containing an invalid integer and a valid patient-name field, then run the repository tests.
Recorded model calls133 fewer calls than baseline (18.8%)
Reported total tokens64,98322,406 fewer tokens than baseline (25.6%)
Baseline tokens87,389
Workflow tokens64,983
Commands added by the workflow
python -m pytest -q tests # Astroid python -m pytest -q pydicom/tests # pydicom git diff --check git diff --stat
The fixed rule triggered once in each task after successful tests on a changed checkout. It returned these results to the model, with a reminder not to repeat unchanged checks.
Tokens include reported input and output tokens; cached input is already included. Changing these controls only reads the six results stored in this page.
Two independent development issues: Astroid #1866 and pydicom #1694. Three approaches, one run of each per issue. Every final patch passed repository tests and an independent public-issue regression in a fresh checkout. These are local checks, not official SWE-bench scores.
| Approach | Model calls | Reported tokens |
|---|---|---|
| Baseline | 16 | 87,389 |
| Single diff check | 13 | 77,905 |
| Validation workflow | 13 | 64,983 |
The workflow did not beat the single check on call count and was slower on both issues in wall time including setup and evaluation. There is only one run of each approach per issue, and both modified approaches included a reminder to avoid repeated checks. We cannot isolate how much of the difference came from the added commands. Read the protocol and results ↗
What we changed
After a test command passes and the repository has changes, a fixed rule runs either git diff --check or the full suite plus diff checks and statistics. The agent receives those results with its tool output. If a check fails or cannot finish, the model gets the result and continues.
We chose this rule ourselves and added it to the tool executor. A learned model did not select it. The normal proxy still records requests without replacing model calls.
From a fixed rule to a learned choice
This example uses a rule we wrote ourselves. The next step is to discover useful programs from traffic and learn when to choose them.
We evaluate the complete task, including routing and fallback costs. Consistent savings across new tasks have not yet been established.
04 / State of the project
What works today,
and what we are still testing.
The sorting prototype and coding experiment test different parts of the idea. A result in one does not show that the other works.
Small data tasks
The first prototype can propose and route to small programs for tasks such as sorting. Programs must pass tests and checks on new inputs before activation. The demo uses fixed examples, not live models.
A router that can learn
The decision API can use a model or a local classifier. Recorded outcomes can train a candidate router while the current policy stays unchanged. Automatic activation is still under development.
Automatic test commands
Coding experiments explore whether a short program can handle routine verification steps. We compare it with the usual agent workflow on separate issues.
Could the decision model learn as requests arrive?
Yes. A small online classifier could learn which program applies from verified examples. A contextual bandit could learn which eligible route gives the best outcome and cost. It observes only the route actually taken, so we also need held-out tests and controlled comparisons; missing outcomes are not evidence that another route would fail. The prototype now supports incremental classifier updates from replay labels. A contextual-bandit policy and automatic deployment are not implemented. How contextual bandits work ↗
Do we need a special decision API?
No specific service is required. We can put an ordinary model API behind our own routing endpoint and request a structured program choice. Structured output makes the result parseable; our tests and runtime checks still decide which programs are eligible. Example: structured outputs with Claude ↗
Is this just a response cache?
The goal is to reuse a procedure across new inputs, rather than replay an answer to a previous request. A sorting program can handle an array it has never seen. Its applicability still needs to be checked.
Does it eliminate model calls?
An active program can avoid an answer-model call, but the decision model still uses inference. Discovery and shadow evaluation add costs too. The small paired workflow trial used fewer model queries than baseline, but its fixed rule is not a learned router. General net savings have not been established.
Can generated code run anything it wants?
No. The data prototype can only use a small set of operations, such as sorting and counting. Coding-task proposals select from a fixed list of read-only commands. The workflow experiment runs a configured test suite and diff checks. The coding agent itself can still run local commands; local execution is not a sandbox.
What would count as a convincing result?
We would need to fix the programs before testing them on new tasks, repeat both the ordinary and modified runs, and preserve task success. We would also need lower total cost or time after counting the work of discovering programs and deciding when to use them.