For teams that build agent harnesses and frameworks
Find the request that broke your prompt cache.
One reordered tool list, or a timestamp in the system prompt, and every later request pays full price. Nothing reports it. prefixlens names the call, the block that changed, what it cost and the fix, and fails the test in CI before it reaches the bill.
Open source, MIT, version 0.6.0. No account, no telemetry, no runtime dependencies. Reads Anthropic Messages, OpenAI Chat Completions and OpenAI Responses requests, from your own harness or through a framework.
- read from the cache
- written to the cache
- paid again after a break
The same session, as a CI step
The check fails for what your code did: a changed tool list, system prompt, history, model or cache-relevant parameter, or a prefix that changed without being declared.
It passes what your code did not do: cache expiry, a miss on the provider's side, a first call, and a break you declared, such as a compaction.
prefixlens check: run.trace.jsonlFAIL: 1 of 5 calls turn 4: avoidable miss [tools_changed], tool list changed (+1 tool: 3 โ 4)lost 49,000 of 49,000 cached tokens (~$0.24) ยท vs turn 3changed block: 4 (system part 1, now tool "web_search")+ web_searchfix: keep one tool list for the whole session (refuse a disallowed tool when it is called rather than removing it), or change it only at a deliberate breakpoint such as a compaction. 5 calls: 1 avoidable miss, 1 cold start, 3 ok
Hand it to your agent
Paste these instructions into the coding agent that works in your harness's repository. It reads how your requests are built, adds the matching test with your real system prompt, tools and memory settings, runs it, and tells you about any cache break it finds.
- Finds whether you build the request yourself or a framework does, and whether the code is JavaScript at all.
- Adds
prefixlensas a dev dependency. - Writes the test in your own test runner, beside your other tests.
- Runs it. A failure is a real cache break: it fixes the small safe ones and shows you the rest.
- Checks that your CI picks the test up.
The test makes no network call and needs no API key: a stand-in answers every model call. Read the instructions before you paste them. They are all here, and the same text is at /agent.md.
# Add a prompt-cache regression test with prefixlens This repository sends requests to an LLM API, directly or through a framework. Add a test that fails when a code change breaks the provider's prompt cache. Use prefixlens: the npm package `prefixlens`, version 0.6.0 or later (https://github.com/parthg1901/prefixlens). Why it matters: a provider caches the start of a request. When anything early in the request differs from the previous call of the same conversation (the tool list or its order, the system prompt, an earlier message), everything after that point is billed at full price, and nothing reports it. prefixlens compares consecutive requests and names the first block that changed. ## 1. Find how requests are built Read the code that builds or sends model requests, and decide which case this is: - **Case A: the project builds the request body itself.** There is a function that returns the body for the Anthropic Messages, OpenAI Chat Completions or OpenAI Responses API. - **Case B: a framework or an SDK builds it.** The Vercel AI SDK, Mastra, LangChain, or a provider's SDK called from an agent loop. - **Case C: the project is not JavaScript or TypeScript.** The test helpers are JavaScript, so the project's own tests capture the request bodies and a small Node script checks them. When A and B are both true, do case B on the main agent loop: it covers more. ## 2. Install Add `prefixlens` as a dev dependency with the package manager the project already uses. It needs Node.js 22 or later, or Bun, and has no dependencies of its own. In case C the project has no JavaScript package manager: step 3 says where it goes. ## 3. Write the test Use the project's own test runner and put the test beside its other tests. Build the agent the way production does: the real system prompt, tools, memory and history settings. A simplified copy protects nothing. Case A, two consecutive requests of one conversation: ```ts import { assertPrefixStable } from "prefixlens/test" const first = buildRequest(conversation) // the project's own request builder const second = buildRequest([...conversation, ...nextStep]) // the same conversation, one step later assertPrefixStable(first, second) ``` Cover a second user turn, a tool call with its result, and anything the project adds during a session (reminders, memory, changed instructions). Case B, a whole run with a stand-in provider: ```ts import { callTools, recordRequests } from "prefixlens/test" // The session answers every model call itself: no network, no API key, no cost. const session = recordRequests({ reply: callTools([{ name: "<one of the agent's tools>", args: {} }]) }) // Give session.fetch to the model provider the agent uses, for example: // createAnthropic({ apiKey: "test", fetch: session.fetch }) Vercel AI SDK, Mastra // new OpenAI({ apiKey: "test", fetch: session.fetch }) a provider's SDK // Then run the real agent in one conversation for at least eight user turns, // so that a history window or a compaction, if there is one, is reached. session.assertStable() ``` Case C, request bodies captured by the project's own tests: 1. In the project's test runner, write a test that collects the JSON bodies of consecutive model requests of one conversation, in the order they are sent and exactly as they go to the provider. Either call the project's request builder for each step, as in case A, or run the agent against the fake provider or the HTTP mock its tests already use and capture each body, as in case B, for at least eight user turns. 2. Add the checker in a directory of its own, so that it stays apart from the project's code: ``` tools/prefixlens/package.json { "private": true, "type": "module", "devDependencies": { "prefixlens": "^0.6.0" } } tools/prefixlens/check.mjs import { readFileSync } from "node:fs" import { assertSessionStable } from "prefixlens/test" assertSessionStable(JSON.parse(readFileSync(process.argv[2], "utf8"))) ``` 3. Install it with `npm install --prefix tools/prefixlens`, and keep `tools/prefixlens/node_modules` out of version control. 4. In the test, write the bodies to a temporary file as one JSON array, run `node tools/prefixlens/check.mjs <file>`, and fail the test when it exits with anything but 0, showing what it printed. That output is prefixlens's message. When the tests cannot produce request bodies without a real provider, stop and tell the user. The other route records a real run through a local proxy (`npx prefixlens record`, then `npx prefixlens check` on the trace); it makes real model calls, so set it up only when the user asks for it. ## 4. Run it - **It passes.** Go to step 5. - **It fails.** The message names the call, the first changed block, a diff and a fix. That is a real cache break in this project, not a problem with the test. Do not loosen the test, shorten the run or change the agent's settings to make it pass. - When the fix is small and safe (a timestamp or an id in the system prompt, tools sent in an unstable order), make it and say what you changed. Keep what the model needs: fix a value once per session or send it in a new message, and delete it only when nothing uses it. - When the break is deliberate (a compaction, a history window the team chose), stop and show the user the message. If they confirm it, allow it at that call only: `session.assertStable({ allowBreakAt: { 7: "compaction" } })`, or the same option as the second argument of `assertSessionStable` in case C. - Otherwise stop and show the user the message. Do not commit a failing test without telling them. ## 5. Make sure CI runs it Check that the command CI already runs picks the new test up. Add a workflow only when the project has none that runs tests. In case C, CI also needs Node.js 22 or later and the `npm install --prefix tools/prefixlens` step before the tests. ## 6. Report Say which case it was, the files you added or changed, the test's output, any cache break you found with prefixlens's message for it, and what the test does not cover. ## Rules - Send no real model requests and use no real API key. - Change production code only for a fix described in step 4, or to let a test hand the model provider a `fetch` function when the code has no way to do that yet. Keep that change as small as it can be. - Commit no trace files: a trace holds prompts. - More detail, when you need it: https://github.com/parthg1901/prefixlens/blob/main/docs/assertions.md
Or wire it yourself
Three ways in, by where your request is built. Each of these was run before it was put here.
Your harness has a function that builds the request body. Build it for two consecutive steps of one conversation and assert that the second keeps the first one's prefix.
import { test } from "node:test" // or bun:test, vitest, jest import { assertPrefixStable } from "prefixlens/test" import { buildRequest } from "../src/request.ts" // your harness's own request builder test("a new step extends the previous request", () => { const history = [{ role: "user", content: "Fix the failing test." }] const first = buildRequest(history) const second = buildRequest([...history, { role: "assistant", content: "Reading it now." }, { role: "user", content: "Go on." }]) // Nothing is sent to a provider: two request bodies are compared. assertPrefixStable(first, second) })
The request is built inside the SDK, so give its provider a session's fetch. The session answers every model call itself and compares each request with the one before it in the same conversation. Run with ai 7 and @ai-sdk/anthropic 4.
import { createAnthropic } from "@ai-sdk/anthropic" import { generateText, stepCountIs, tool, type ModelMessage } from "ai" import { test } from "node:test" // or bun:test, vitest, jest import { callTools, recordRequests } from "prefixlens/test" import { z } from "zod" test("the agent keeps its prefix across turns and tool calls", async () => { // The session stands in for the provider: after each user message the model calls read_file once, then answers. const session = recordRequests({ reply: callTools([{ name: "read_file", args: { path: "src/app.ts" } }], "Done.") }) const model = createAnthropic({ apiKey: "test", fetch: session.fetch })("claude-sonnet-5-5") const tools = { read_file: tool({ description: "Read a file.", inputSchema: z.object({ path: z.string() }), execute: async () => "export const app = 1" }), } // Your agent loop goes here. This one keeps one history and sends all of it. const messages: ModelMessage[] = [] for (const question of ["Where is the bug?", "Fix it.", "Add a test."]) { messages.push({ role: "user", content: question }) const result = await generateText({ model, system: "You are a coding agent.", tools, messages, stopWhen: stepCountIs(5) }) messages.push(...result.responseMessages) } session.assertStable() })
The same session, given to the model of a Mastra agent with memory. Run with @mastra/core 1.83 and @mastra/memory 1.41.
import { createAnthropic } from "@ai-sdk/anthropic" import { Agent } from "@mastra/core/agent" import { LibSQLStore } from "@mastra/libsql" import { Memory } from "@mastra/memory" import { test } from "node:test" // or bun:test, vitest, jest import { recordRequests } from "prefixlens/test" test("the agent keeps its prefix through a long thread", async () => { const session = recordRequests() const model = createAnthropic({ apiKey: "test", fetch: session.fetch })("claude-sonnet-5-5") const agent = new Agent({ id: "support", name: "support", instructions: "You are a support agent.", model, memory: new Memory({ storage: new LibSQLStore({ id: "test", url: ":memory:" }) }), }) // Long enough to pass any history window: 8 questions are 16 stored messages. for (let turn = 1; turn <= 8; turn++) { await agent.generate(`Question ${turn}`, { memory: { thread: "thread-1", resource: "user-1" } }) } session.assertStable() })
With Mastra's default memory this test fails at the seventh call. The default keeps the last ten messages, so from then on the oldest ones leave on every request:
PrefixBreakError: prefix not stable at call 7 (lane main, compared with call 6): 2 oldest messages removed from the start of history (9 kept, 2 appended)
first changed block: 2 of 12 (message #1, user); block 1 still matches, everything from block 2 on is re-read
fix: a window that slides on every call re-reads the whole history each time: drop history in chunks at a saved boundary (cut rarely, and replay from the same cut until the next one), or keep history append-only.
if this change is intentional, pass { allowBreakAt: { 7: { reason: "..." } } }
1 more call changes the prefix: call 8 (2 oldest messages removed from the start of history (9 kept, 2 appended))This is a trade-off Mastra documents, not a bug: its docs point long threads to observational memory, which keeps the prefix stable. The test shows you the call where a setting starts to cost you, whichever you choose.
No test to write, and it works for a harness in any language. Record a real run through a local proxy, then read the report or gate on it. The proxy forwards your credentials and never writes them.
# 1. Record: a local proxy forwards every request unchanged and writes one record per model call. npx prefixlens record --provider anthropic -o run.trace.jsonl # or --provider openai # 2. Point your agent at it and run a task as usual, in any language. export ANTHROPIC_BASE_URL=http://127.0.0.1:8787 # openai: OPENAI_BASE_URL=http://127.0.0.1:8787/v1 # 3. Read it, or gate on it. npx prefixlens report run.trace.jsonl --html # every call: tokens read, written and lost, and the cause of each miss npx prefixlens check run.trace.jsonl # exits 1 when your harness broke its own cache
Read a report
For every call: the tokens read from the cache, written and lost, the hit rate and the cost. For every miss: the first changed block, a diff and what it cost. This is the report of the session at the top of the page, exactly as prefixlens report --html writes it: one file, nothing loaded from elsewhere.
What it tells apart
Fails Your harness changed its own prefix
- The tool list gained, lost or reordered a tool, or a definition changed.
- The system prompt was edited, or carries a clock, an id or a relative date that is rendered again on each call.
- History was rewritten: a message edited, inserted, removed or reordered, tool output rendered again, a window that slides, a rewind.
- The model or a parameter that is part of the cached prefix changed.
- A prefix changed with nothing cached yet: a break waiting for a cache to hit.
Passes Not something your code did
- Expiry: the prefix was intact and the cache entry had run out.
- The provider's side: the prefix was intact and in time, and the provider read nothing.
- A first call, cold or warm.
- A declared break: a compaction, a revert or a settings change your harness says it meant, counted with its cost, apart from the rest.
What it does not do
- It sees the request your code sends, not what the provider does with it. Eviction and routing show up as a miss with the prefix intact.
- Costs are list prices from a bundled table, not your bill. Discounts, batch tiers and subscriptions are not applied.
- The test helpers are JavaScript and TypeScript. From another language, your tests hand the request bodies to a four-line Node script, which the instructions above set up, or you record through the proxy, or write the trace format.
- Gemini's native API and WebSocket transports are not recorded.
- It is before 1.0: the JSON output may change in a minor version. What has been checked on real traffic is written down.