Skip to documentation

Reference · LLM fixtures

Available

Last verified

Every fixture field, condition, and placeholder

A fixture pairs AND-combined match rules with a response, a tool-call workflow, or WebSocket events. Enabled fixtures are sorted by priority and the first match wins, independent of which provider route the client called.

In this guide

  • Fixture structure and defaults
  • Match fields and structured conditions
  • Sequences, sessions, and reset
  • Templates, reasoning, structured output, and errors

Prerequisites

  • A hosted LLM DotMock workspace
  • The workspace ID in $API_ID and its runtime URL in $MOCK_URL (SDK base URL $MOCK_URL/v1 for OpenAI-compatible clients)
On this page
  1. Fixture structure
  2. Match fields
  3. Structured conditions
  4. Priority and first match
  5. Sequences, sessions, and reset
  6. Content, usage, and provider errors
  7. Templates and placeholders
  8. Reasoning output
  9. Structured outputs
  10. Sequence behavior with shared state

Fixture structure

Every fixture needs a name and one of response, workflow, or ws. The same JSON is stored by the workspace, accepted by POST /mock-apis/$API_ID/llm-fixtures and the MCP create_llm_fixture tool, and written under fixtures: in dotmock.yaml (where id defaults to a slug of the name, priority to list order, enabled to true, and protocol to both).

  • id, name — stable identifiers; the journal and x-dotmock-fixture-id / x-dotmock-fixture-name headers report them
  • priority — integer; lower is evaluated first
  • enabled — disabled fixtures are skipped
  • protocol — http, ws, or both
  • match — AND-combined rules (below)
  • response — content, reasoning, toolCalls, finishReason, usage, error, responseId, embedding
  • workflow — run.toolCalls followed by a final response (see Tool-call workflows)
  • streaming, latency — timing (see Streaming physics)
  • chaos — per-fixture failure rates (see Chaos testing)
  • ws — scripted WebSocket events
  • stateActions — before and after state mutations shared with other protocols
Fixture
{
  "name": "refund-policy",
  "priority": 20,
  "enabled": true,
  "protocol": "both",
  "match": {
    "model": "gpt-4o*",
    "userMessage": "refund",
    "conditions": [{ "field": "turnIndex", "operator": ">=", "value": "1" }]
  },
  "response": {
    "content": "Refunds are issued within 5 business days.",
    "finishReason": "stop",
    "usage": { "promptTokens": 40, "completionTokens": 11 }
  }
}

Match fields

Every field you set must match. A fixture with an empty match is a catch-all; keep exactly one and give it the highest priority number. Text fields are case-insensitive substrings unless written as /regex/ or /regex/i.

  • userMessage — last user message (substring or /regex/)
  • systemPrompt — first system or developer message (substring or /regex/)
  • model — exact (case-insensitive) or glob such as gpt-4* or claude-*
  • toolName — any tool defined in the request has this name
  • toolCallId — the last tool message answers this call ID
  • hasToolResult — true or false: the request contains any tool result
  • responseFormat — exact response_format.type, for example json_object or json_schema
  • temperature — exact number
  • inputText — embeddings input (substring or /regex/)
  • provider — openai, anthropic, gemini, bedrock, azure, ollama, or cohere
  • endpoint — exact or glob against the normalized endpoint
  • turnIndex — zero-based count of user turns in the conversation
  • context — map of key to substring/regex, read from the request's context or metadata object
  • sequenceIndex — serve on the Nth (zero-based) matching request in a session
  • conditions — structured rules (next section)

Structured conditions

conditions is an AND-only list of { field, operator, value } rules for comparisons the plain fields cannot express. Values are strings; numeric operators parse them. An unknown operator never matches.

  • Fields: userMessage, model, systemPrompt, temperature, provider, endpoint, hasToolResult, turnIndex, toolName (any tool), context.<key>, $.<gjson.path> into the raw request JSON, random, request.time
  • Text operators: contains (default), equals, startswith, endswith, regex, glob
  • Other operators: is (boolean), exists (value true or false), >, >=, <, <=
  • random — a fresh draw in [0, 1) per evaluation; random < 0.25 fires about 25% of the time and is reproducible with X-Dotmock-Seed
  • request.time — operator window with value "HH:MM-HH:MM" in server time; overnight windows such as 22:00-06:00 work
Conditions
"conditions": [
  { "field": "$.metadata.tier", "operator": "equals", "value": "enterprise" },
  { "field": "$.max_tokens", "operator": "<", "value": "256" },
  { "field": "toolName", "operator": "startswith", "value": "search_" },
  { "field": "random", "operator": "<", "value": "0.1" },
  { "field": "request.time", "operator": "window", "value": "09:00-17:00" }
]

Priority and first match

Enabled fixtures are sorted by ascending priority (ties keep list order) and evaluated in that order; the first fixture whose rules all match serves the request. Put narrow scenarios first, broad ones after, and the catch-all last. Reorder with PUT /mock-apis/$API_ID/llm-fixtures/reorder or drag in the fixture list.

Sequences, sessions, and reset

Fixtures that share identical match rules (ignoring sequenceIndex) form one sequence group with one counter. The fixture with sequenceIndex N serves only when the group's counter equals N, so indexes 0, 1, 2 answer the first, second, and third matching request. Sequences do not wrap: after the last index, later fixtures match instead.

Counters are stored per API and per session in Redis (llm:seq:{apiId}:{session}, 24-hour TTL) and fall back to memory when Redis is unavailable. The session is the X-Dotmock-Session request header (up to 128 characters) or default. Use one session per test so parallel runs never advance each other's sequences.

Retry then succeed
[
  { "name": "flaky-1", "priority": 10, "match": { "userMessage": "summarize", "sequenceIndex": 0 },
    "response": { "error": { "status": 503, "type": "server_error", "message": "Overloaded" } } },
  { "name": "flaky-2", "priority": 11, "match": { "userMessage": "summarize", "sequenceIndex": 1 },
    "response": { "content": "Here is the summary." } }
]

Reset counters
# Management credential from DOTMOCK_API_KEY or dotmock login
# Omit --session to reset every session for the API
dotmock llm reset $API_ID --session test-42

Content, usage, and provider errors

content is the assistant text. finishReason accepts provider spellings and is normalized: stop; tool_calls (also tool_use, function_call); length (also max_tokens); content_filter (also safety, refusal). When omitted it is tool_calls if tool calls are present, otherwise stop. usage takes promptTokens, completionTokens, and reasoningTokens; without it tokens are estimated at one per four characters. Tool calls without an id get a provider-style generated ID.

error returns a provider-shaped failure with { status, type, message }. Use it for deterministic 400, 401, 429, 500, or 529 paths; a 429 also sends Retry-After.

Error fixture
{
  "name": "context-too-long",
  "match": { "conditions": [{ "field": "$.max_tokens", "operator": ">", "value": "100000" }] },
  "response": {
    "error": { "status": 400, "type": "invalid_request_error", "message": "maximum context length exceeded" }
  }
}

Templates and placeholders

Response content, tool arguments, and tool results run through the same template processor as REST mocks. Workflow final responses additionally resolve tool placeholders before faker helpers run.

Faker and utility helpers: {{uuid}}, {{now}}, {{date("+1d")}}, {{randomInt}}, {{randomFloat}}, {{randomBool}}, {{randomChoice}}, {{randomString}}, {{randomEmail}}, {{randomName}}, {{randomPhone}}, {{randomAddress}}, {{randomCompany}}, {{lorem}}, {{number.int({"min":1,"max":100})}}, {{animal.dog}}, {{animal.cat}}. Send X-Dotmock-Seed to make their output reproducible.

  • {{userMessage}} — the last user message
  • {{tool.result}} — the last tool message, falling back to the first run call's stored result
  • {{tool.<name>.result}} — the result for that tool, matched by call ID or name, falling back to its stored result
  • {{tool.<name>.args.<path>}} — a gjson path into that tool call's arguments

Reasoning output

Set response.reasoning to model thinking text and optionally usage.reasoningTokens. DotMock renders it as reasoning_content (OpenAI Chat), a reasoning summary item (Responses API), a signed thinking block (Anthropic), thought parts (Gemini), reasoningContent (Bedrock), and thinking (Ollama). Reasoning streams before the answer text, so UIs that show thinking separately can be tested.

Reasoning fixture
{
  "name": "math-with-thinking",
  "match": { "userMessage": "17 * 23" },
  "response": {
    "reasoning": "17 * 20 = 340 and 17 * 3 = 51, so 391.",
    "content": "391",
    "usage": { "promptTokens": 14, "completionTokens": 2, "reasoningTokens": 18 }
  }
}

Structured outputs

Match on responseFormat (json_object or json_schema) and return the exact JSON string your parser should see as content. Returning explicit content is the most deterministic option and the one to use in dotmock.yaml.

In a hosted workspace, a fixture with no content and no tool calls synthesizes a schema-valid answer instead: arguments for a forced tool from its parameter schema, or content from the requested JSON schema (const, then enum, examples, default, then type; $ref, anyOf, oneOf, allOf, and formats are honored), or {} for plain JSON mode.

Structured output fixture
{
  "name": "sentiment",
  "match": { "responseFormat": "json_schema" },
  "response": {
    "content": "{\"sentiment\":\"positive\",\"confidence\":0.93}",
    "finishReason": "stop"
  }
}

Sequence behavior with shared state

For behavior that depends on more than request order, before and after state actions use the same runtime as REST, GraphQL, and webhook workflows. Increment a per-conversation turn, consume a queue of outcomes, or read a workspace-shared customer record, and reference it from content.

handlebars
Turn {{state.increment "conversation-turns" "$body.metadata.sessionId" 1}}