For teams that build agent harnesses and frameworks
Find the request that broke your prompt cache.
One reordered tool list, or a timestamp in the system prompt, and every later request pays full price. Nothing reports it. prefixlens names the call, the block that changed, what it cost and the fix, and fails the test in CI before it reaches the bill.
Open source, MIT, version 0.6.0. No account, no telemetry, no runtime dependencies. Reads Anthropic Messages, OpenAI Chat Completions and OpenAI Responses requests, from your own harness or through a framework.
- read from the cache
- written to the cache
- paid again after a break
The same session, as a CI step
The check fails for what your code did: a changed tool list, system prompt, history, model or cache-relevant parameter, or a prefix that changed without being declared.
It passes what your code did not do: cache expiry, a miss on the provider's side, a first call, and a break you declared, such as a compaction.
prefixlens check: run.trace.jsonlFAIL: 1 of 5 calls turn 4: avoidable miss [tools_changed], tool list changed (+1 tool: 3 → 4)lost 49,000 of 49,000 cached tokens (~$0.24) · vs turn 3changed block: 4 (system part 1, now tool "web_search")+ web_searchfix: keep one tool list for the whole session (refuse a disallowed tool when it is called rather than removing it), or change it only at a deliberate breakpoint such as a compaction. 5 calls: 1 avoidable miss, 1 cold start, 3 ok
Hand it to your agent
Paste these instructions into the coding agent that works in your harness's repository. It reads how your requests are built, adds the matching test with your real system prompt, tools and memory settings, runs it, and tells you about any cache break it finds. It asks before it changes your own code, your CI or your repository's layout.
- Finds whether you build the request yourself or a framework does, and whether the code is JavaScript at all.
- Adds
prefixlensas a dev dependency. - Writes the test in your own test runner, beside your other tests.
- Runs it. A failure is a real cache break: it shows you the message and a fix, and applies the fix only when you say so.
- Checks that your CI picks the test up, and commits or pushes nothing unless you ask.
The test makes no network call and needs no API key: a stand-in answers every model call. Read the instructions before you paste them. They are all here, and the same text is at /agent.md.
# Add a prompt-cache regression test with prefixlens This repository sends requests to an LLM API, directly or through a framework. Add a test that fails when a code change breaks the provider's prompt cache. Use prefixlens: the npm package `prefixlens`, at exactly version 0.6.0 (https://github.com/parthg1901/prefixlens). Why it matters: a provider caches the start of a request. When anything early in the request differs from the previous call of the same conversation (the tool list or its order, the system prompt, an earlier message), everything after that point is billed at full price, and nothing reports it. prefixlens compares consecutive requests and names the first block that changed. ## 1. Find how requests are built Read the code that builds or sends model requests, and decide which case this is: - **Case A: the project builds the request body itself.** There is a function that returns the body for the Anthropic Messages, OpenAI Chat Completions or OpenAI Responses API. - **Case B: a framework or an SDK builds it.** The Vercel AI SDK, Mastra, LangChain, or a provider's SDK called from an agent loop. - **Case C: the project is not JavaScript or TypeScript.** The test helpers are JavaScript, so the project's own tests capture the request bodies and a small Node script checks them. When A and B are both true, do case B on the main agent loop: it covers more. ## 2. Install Add `prefixlens` as a dev dependency at exactly version 0.6.0, with the package manager the project already uses (for example `npm install --save-dev --save-exact prefixlens@0.6.0`), and keep the lockfile change. It needs Node.js 22 or later, or Bun, and has no dependencies of its own. In case C the project has no JavaScript package manager: step 3 says where it goes. ## 3. Write the test Use the project's own test runner and put the test beside its other tests. Build the agent the way production does: the real system prompt, tools, memory and history settings. A simplified copy protects nothing. Writing the test should not need a change to the project's own code. Case A, two consecutive requests of one conversation: ```ts import { assertPrefixStable } from "prefixlens/test" const first = buildRequest(conversation) // the project's own request builder const second = buildRequest([...conversation, ...nextStep]) // the same conversation, one step later assertPrefixStable(first, second) ``` Cover a second user turn, a tool call with its result, and anything the project adds during a session (reminders, memory, changed instructions). Case B, a whole run with a stand-in provider: ```ts import { callTools, recordRequests } from "prefixlens/test" // The session answers every model call itself: no network, no API key, no cost. const session = recordRequests({ reply: callTools([{ name: "<one of the agent's tools>", args: {} }]) }) // Hand session.fetch to the model provider the agent uses. Take the first of these that applies: // 1. The code already accepts a fetch function or a client: pass it. // createAnthropic({ apiKey: "test", fetch: session.fetch }) Vercel AI SDK, Mastra // new OpenAI({ apiKey: "test", fetch: session.fetch }) a provider's SDK // 2. It does not: replace the global fetch for this test only. Set it and the provider's API key // variable ("test") before the agent's module is imported, and put the real fetch back afterwards. // globalThis.fetch = session.fetch // 3. Neither works: stop and ask before changing the project's code (see Rules). // Then run the real agent in one conversation for at least eight user turns, // so that a history window or a compaction, if there is one, is reached. session.assertStable() ``` Case C, request bodies captured by the project's own tests. This adds a small Node.js tool to a project that has none: tell the user what you are about to add (the two files below, and Node.js 22 in CI) and ask before adding it. 1. In the project's test runner, write a test that collects the JSON bodies of consecutive model requests of one conversation, in the order they are sent and exactly as they go to the provider. Either call the project's request builder for each step, as in case A, or run the agent against the fake provider or the HTTP mock its tests already use and capture each body, as in case B, for at least eight user turns. 2. Add the checker in a directory of its own, so that it stays apart from the project's code: ``` tools/prefixlens/package.json { "private": true, "type": "module", "devDependencies": { "prefixlens": "0.6.0" } } tools/prefixlens/check.mjs import { readFileSync } from "node:fs" import { assertSessionStable } from "prefixlens/test" assertSessionStable(JSON.parse(readFileSync(process.argv[2], "utf8"))) ``` 3. Install it with `npm install --prefix tools/prefixlens`. Keep its `package-lock.json`, and keep `tools/prefixlens/node_modules` out of version control. 4. In the test, write the bodies to a temporary file as one JSON array, run `node tools/prefixlens/check.mjs <file>`, and fail the test when it exits with anything but 0, showing what it printed. That output is prefixlens's message. When the tests cannot produce request bodies without a real provider, stop and tell the user. The other route records a real run through a local proxy (`npx prefixlens record`, then `npx prefixlens check` on the trace); it makes real model calls, so set it up only when the user asks for it. ## 4. Run it - **It passes.** Go to step 5. - **It fails.** The message names the call, the first changed block, a diff and a fix. That is a real cache break in this project, not a problem with the test. Do not loosen the test, shorten the run or change the agent's settings to make it pass. - When the fix looks small (a timestamp or an id in the system prompt, tools sent in an unstable order), show it to the user as a diff and ask before applying it. Keep what the model needs: fix a value once per session or send it in a new message, and delete it only when nothing uses it. When the user has already told you to apply fixes, make it, in a commit of its own if you are committing, and say what you changed. - When the break is deliberate (a compaction, a history window the team chose), stop and show the user the message. If they confirm it, allow it at that call only: `session.assertStable({ allowBreakAt: { 7: "compaction" } })`, or the same option as the second argument of `assertSessionStable` in case C. The number is the call's number in the failure message; calls are counted from 1. - Otherwise stop and show the user the message. - Leave the failing test in place, and say that it fails and why. ## 5. Make sure CI runs it Check that the command CI already runs picks the new test up. When the project has no workflow that runs tests, say so and ask before adding one. In case C, CI also needs Node.js 22 or later and `npm ci --prefix tools/prefixlens` before the tests: tell the user, and change a workflow only when they agree. ## 6. Report Say which case it was, the files you added or changed, the test's output, any cache break you found with prefixlens's message for it, anything you are waiting for an answer on, and what the test does not cover. ## Rules - Send no real model requests and use no real API key. - Ask before changing the project's own code. That covers any fix from step 4, and any change made only so that a test can reach the model provider. Show the change as a diff first, and keep it as small as it can be. - Ask before adding a CI workflow, or the `tools/prefixlens/` directory of case C. - When there is nobody to ask, stop at that point and put the question in your report. - Install exactly version 0.6.0, and keep the lockfile change. - Commit no trace files: a trace holds prompts. - In your report, quote only the lines prefixlens names; don't paste full system prompts or anything that looks like a secret. - Do not commit, push or open a pull request unless the user asks you to. - More detail, when you need it: https://github.com/parthg1901/prefixlens/blob/main/docs/assertions.md
Or wire it yourself
Three ways in, by where your request is built. Each of these was run before it was put here.
Your harness has a function that builds the request body. Build it for two consecutive steps of one conversation and assert that the second keeps the first one's prefix.
import { test } from "node:test" // or bun:test, vitest, jest import { assertPrefixStable } from "prefixlens/test" import { buildRequest } from "../src/request.ts" // your harness's own request builder test("a new step extends the previous request", () => { const history = [{ role: "user", content: "Fix the failing test." }] const first = buildRequest(history) const second = buildRequest([...history, { role: "assistant", content: "Reading it now." }, { role: "user", content: "Go on." }]) // Nothing is sent to a provider: two request bodies are compared. assertPrefixStable(first, second) })
The request is built inside the SDK, so give its provider a session's fetch. The session answers every model call itself and compares each request with the one before it in the same conversation. Run with ai 7 and @ai-sdk/anthropic 4.
import { createAnthropic } from "@ai-sdk/anthropic" import { generateText, stepCountIs, tool, type ModelMessage } from "ai" import { test } from "node:test" // or bun:test, vitest, jest import { callTools, recordRequests } from "prefixlens/test" import { z } from "zod" test("the agent keeps its prefix across turns and tool calls", async () => { // The session stands in for the provider: after each user message the model calls read_file once, then answers. const session = recordRequests({ reply: callTools([{ name: "read_file", args: { path: "src/app.ts" } }], "Done.") }) const model = createAnthropic({ apiKey: "test", fetch: session.fetch })("claude-sonnet-5-5") const tools = { read_file: tool({ description: "Read a file.", inputSchema: z.object({ path: z.string() }), execute: async () => "export const app = 1" }), } // Your agent loop goes here. This one keeps one history and sends all of it. const messages: ModelMessage[] = [] for (const question of ["Where is the bug?", "Fix it.", "Add a test."]) { messages.push({ role: "user", content: question }) const result = await generateText({ model, system: "You are a coding agent.", tools, messages, stopWhen: stepCountIs(5) }) messages.push(...result.responseMessages) } session.assertStable() })
The same session, given to the model of a Mastra agent with memory. Run with @mastra/core 1.83 and @mastra/memory 1.41.
import { createAnthropic } from "@ai-sdk/anthropic" import { Agent } from "@mastra/core/agent" import { LibSQLStore } from "@mastra/libsql" import { Memory } from "@mastra/memory" import { test } from "node:test" // or bun:test, vitest, jest import { recordRequests } from "prefixlens/test" test("the agent keeps its prefix through a long thread", async () => { const session = recordRequests() const model = createAnthropic({ apiKey: "test", fetch: session.fetch })("claude-sonnet-5-5") const agent = new Agent({ id: "support", name: "support", instructions: "You are a support agent.", model, memory: new Memory({ storage: new LibSQLStore({ id: "test", url: ":memory:" }) }), }) // Long enough to pass any history window: 8 questions are 16 stored messages. for (let turn = 1; turn <= 8; turn++) { await agent.generate(`Question ${turn}`, { memory: { thread: "thread-1", resource: "user-1" } }) } session.assertStable() })
With Mastra's default memory this test fails at the seventh call. The default keeps the last ten messages, so from then on the oldest ones leave on every request:
PrefixBreakError: prefix not stable at call 7 (lane main, compared with call 6): 2 oldest messages removed from the start of history (9 kept, 2 appended)
first changed block: 2 of 12 (message #1, user); block 1 still matches, everything from block 2 on is re-read
fix: a window that slides on every call re-reads the whole history each time: drop history in chunks at a saved boundary (cut rarely, and replay from the same cut until the next one), or keep history append-only.
if this change is intentional, pass { allowBreakAt: { 7: { reason: "..." } } }
1 more call changes the prefix: call 8 (2 oldest messages removed from the start of history (9 kept, 2 appended))This is a trade-off Mastra documents, not a bug: its docs point long threads to observational memory, which keeps the prefix stable. The test shows you the call where a setting starts to cost you, whichever you choose.
No test to write, and it works for a harness in any language. Record a real run through a local proxy, then read the report or gate on it. The proxy forwards your credentials and never writes them.
# 1. Record: a local proxy forwards every request unchanged and writes one record per model call. npx prefixlens record --provider anthropic -o run.trace.jsonl # or --provider openai # 2. Point your agent at it and run a task as usual, in any language. export ANTHROPIC_BASE_URL=http://127.0.0.1:8787 # openai: OPENAI_BASE_URL=http://127.0.0.1:8787/v1 # 3. Read it, or gate on it. npx prefixlens report run.trace.jsonl --html # every call: tokens read, written and lost, and the cause of each miss npx prefixlens check run.trace.jsonl # exits 1 when your harness broke its own cache
Read a report
For every call: the tokens read from the cache, written and lost, the hit rate and the cost. For every miss: the first changed block, a diff and what it cost. This is the report of the session at the top of the page, exactly as prefixlens report --html writes it: one file, nothing loaded from elsewhere.
What it tells apart
Fails Your harness changed its own prefix
- The tool list gained, lost or reordered a tool, or a definition changed.
- The system prompt was edited, or carries a clock, an id or a relative date that is rendered again on each call.
- History was rewritten: a message edited, inserted, removed or reordered, tool output rendered again, a window that slides, a rewind.
- The model or a parameter that is part of the cached prefix changed.
- A prefix changed with nothing cached yet: a break waiting for a cache to hit.
Passes Not something your code did
- Expiry: the prefix was intact and the cache entry had run out.
- The provider's side: the prefix was intact and in time, and the provider read nothing.
- A first call, cold or warm.
- A declared break: a compaction, a revert or a settings change your harness says it meant, counted with its cost, apart from the rest.
What it does not do
- It sees the request your code sends, not what the provider does with it. Eviction and routing show up as a miss with the prefix intact.
- Costs are list prices from a bundled table, not your bill. Discounts, batch tiers and subscriptions are not applied.
- The test helpers are JavaScript and TypeScript. From another language, your tests hand the request bodies to a four-line Node script, which the instructions above set up, or you record through the proxy, or write the trace format.
- Gemini's native API and WebSocket transports are not recorded.
- It is before 1.0: the JSON output may change in a minor version. What has been checked on real traffic is written down.