Skip to documentation

Guide · Chaos and failure testing

Available

Last verified

Prove retries, fallbacks, and partial-stream handling

Provider outages rarely arrive as a clean error. Chaos settings make DotMock drop requests, corrupt a stream frame, cut a connection mid-answer, or rate-limit with provider headers, at a rate you choose and with a seed you can replay.

In this guide

  • Four failure actions
  • API, fixture, and request levels
  • Deterministic runs with X-Dotmock-Seed
  • Deterministic errors without chaos

Prerequisites

  • An LLM DotMock workspace, or a local dotmock.yaml served by dotmock serve
  • The workspace ID in $API_ID and its runtime URL in $MOCK_URL (SDK base URL $MOCK_URL/v1 for OpenAI-compatible clients)
On this page
  1. Failure actions
  2. Configure at three levels
  3. Make chaos reproducible with X-Dotmock-Seed
  4. Use deterministic errors for assertions
  5. Scope and limits

Failure actions

Each rate is a probability from 0 to 1. For every matched HTTP request DotMock rolls, in order, drop, rate limit, malformed, and disconnect; the first action that fires is applied and recorded as the journal entry's chaosAction.

  • dropRate — respond immediately with HTTP 500 and error code chaos_drop
  • rateLimitRate — respond with a provider-shaped 429 plus Retry-After, retry-after-ms, x-ratelimit-remaining-requests: 0, and x-ratelimit-reset-requests (Anthropic also gets anthropic-ratelimit-requests-remaining: 0); retryAfterSeconds sets the value, default 1
  • malformedRate — corrupt one frame in the middle of the stream while keeping the framing intact, so the client's parser fails on that event (a non-streamed body is truncated JSON; Bedrock frames fail their CRC)
  • disconnectRate — stream disconnectAfterChunks content chunks (default half of them, at least 1), then kill the connection without a terminal event

Configure at three levels

Request headers override the fixture's chaos object, which overrides the API's settings.chaos. Use API settings for a resilience environment, fixture chaos for one flaky scenario, and headers for a single test case.

  • X-Dotmock-Chaos-Drop, X-Dotmock-Chaos-RateLimit, X-Dotmock-Chaos-Malformed, X-Dotmock-Chaos-Disconnect — rates, clamped to 0–1
  • X-Dotmock-Chaos-Disconnect-After — content chunks before the disconnect
  • X-Dotmock-Chaos-Retry-After — seconds for the 429 Retry-After header
API settings
{
  "chaos": { "dropRate": 0.02, "malformedRate": 0.01, "disconnectRate": 0.01 }
}

Fixture
{
  "name": "long-answer-unstable",
  "match": { "userMessage": "write a report" },
  "response": { "content": "Section 1 …" },
  "streaming": { "chunkSize": 12, "latencyMs": 20 },
  "chaos": { "disconnectRate": 0.5, "disconnectAfterChunks": 4 }
}

Per request
curl -i "$MOCK_URL/v1/chat/completions" \
  -H 'content-type: application/json' \
  -H 'X-Dotmock-Chaos-RateLimit: 1' \
  -H 'X-Dotmock-Chaos-Retry-After: 3' \
  -d '{"model":"gpt-4o-mini","messages":[{"role":"user","content":"hello"}]}'

Expected result: HTTP 429 with Retry-After: 3 and an OpenAI-shaped rate-limit error body.

Make chaos reproducible with X-Dotmock-Seed

Without a seed, every roll is random. Send X-Dotmock-Seed: <integer> and the same request produces the same chaos decision, jitter, random-condition result, generated IDs, and faker values every time. Record the seed in failing test output so a flaky-looking failure can be replayed exactly.

Seeded client
client = OpenAI(
    base_url=f"{MOCK_URL}/v1",
    api_key="dotmock",
    default_headers={"X-Dotmock-Seed": "1234", "X-Dotmock-Chaos-Disconnect": "0.3"},
    max_retries=0,  # let the test observe the failure directly
)

Use deterministic errors for assertions

Chaos answers "does the system survive random failure?". To assert a specific branch, prefer deterministic setups: an error fixture ({ status: 429, type: rate_limit_error, … }), a sequence that fails on sequenceIndex 0 and succeeds on 1, a header rate of 1, or truncateAfterChunks for a clean early stop.

Scope and limits

Chaos applies to HTTP and SSE responses on every provider route, including embeddings and streamed tool calls. Requests that match no fixture still roll the API-level and header drop and rate-limit rates before the unmatched fallback or VCR runs; malformed and disconnect need a matched fixture. Scripted WebSocket fixtures are not affected.