Run LLM tests offline in every pipeline

Commit the fixtures next to the code as dotmock.yaml, start a local mock server in the job, and assert which fixture answered each request. No DotMock account, API key, backend, or Redis is needed in local mode.

Local mode, dotmock serve, the GitHub Action, @dotmock/cli/testing, and pytest-dotmock ship with @dotmock/cli 0.2.0 and are in preview.

In this guide

Describe mocks as config-as-code
Start the server with dotmock serve or Docker
Use the GitHub Action
Assert fixtures from Vitest, Jest, or pytest

Prerequisites

Node.js 18+ for the CLI, or Docker
A dotmock.yaml project file (dotmock init --llm)

On this page

Describe the mock in dotmock.yaml
Start the server with dotmock serve
Use the GitHub Action
Assert fixtures from Vitest or Jest
Assert fixtures from pytest
Test against a hosted workspace instead

Describe the mock in dotmock.yaml

A local project file lists one or more APIs. Each API has a name, a DNS-safe subdomain, a type of llm or openapi, and for LLM APIs an optional defaultModel, settings, and fixtures. Fixture fields are identical to the hosted workspace: id defaults to a slug of the name, priority defaults to list order, and enabled defaults to true.

In local mode the server selects the API from the X-Dotmock-Api header, from a /<subdomain>/ path prefix, or automatically when the file defines exactly one API. Edits hot-reload without restarting.

dotmock.yaml

# Schema: https://dotmock.com/schemas/dotmock-project.schema.json
version: 1
apis:
  - name: Assistant
    subdomain: assistant
    type: llm
    defaultModel: gpt-4o-mini
    settings:
      fallback: { type: none }
    fixtures:
      - name: greeting
        priority: 10
        match: { userMessage: hello }
        response: { content: "Hello!", finishReason: stop }
      - name: fallback
        priority: 1000
        match: {}
        response: { content: "Default mock answer.", finishReason: stop }

Start the server with dotmock serve

dotmock serve runs a dotmock-server binary from PATH (or $DOTMOCK_SERVER_BIN) with DOTMOCK_LOCAL_MODE=true and DOTMOCK_LOCAL_CONFIG pointing at your file. Without a binary it falls back to docker run of ghcr.io/mockitohq/dotmock-server:latest with the config directory mounted read-only. It waits for GET /__dotmock/health, prints each API's base URL, and prints export lines for DOTMOCK_URL, OPENAI_BASE_URL, ANTHROPIC_BASE_URL, and DOTMOCK_LLM_URL.

-c, --config <file> — default dotmock.yaml
-p, --port <port> — default 8080
-d, --detach — run in the background; stop with dotmock serve stop
--runtime auto|binary|docker — default auto
--image <image> — or $DOTMOCK_SERVER_IMAGE
--timeout <seconds> — health wait, default 60

CLI

npx @dotmock/cli init --llm
npx @dotmock/cli serve --detach   # prints the base URLs and export lines
export OPENAI_BASE_URL=http://127.0.0.1:8080/assistant/v1
export ANTHROPIC_BASE_URL=http://127.0.0.1:8080/assistant
npm test
npx @dotmock/cli serve stop

Docker only

docker run --rm -p 127.0.0.1:8080:8080 \
  -v "$PWD":/dotmock:ro \
  -e DOTMOCK_LOCAL_MODE=true \
  -e DOTMOCK_LOCAL_CONFIG=/dotmock/dotmock.yaml \
  -e PORT=8080 \
  ghcr.io/mockitohq/dotmock-server:latest

curl -s http://127.0.0.1:8080/__dotmock/health

Use the GitHub Action

The action runs dotmock serve --detach for the rest of the job and stops it in a post step that always runs, after printing the journal entry count. It exports DOTMOCK_URL, DOTMOCK_LLM_URL, OPENAI_BASE_URL, ANTHROPIC_BASE_URL, DOTMOCK_CONFIG, and DOTMOCK_STATE_DIR to later steps, and sets OPENAI_API_KEY, ANTHROPIC_API_KEY, and GEMINI_API_KEY to dotmock when they are unset.

Inputs: config, port, image, runtime, cli-version, timeout, working-directory, export-dummy-keys
Outputs: url, openai-base-url, anthropic-base-url, apis (JSON)

.github/workflows/test.yml

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v5
      - uses: actions/setup-node@v5
        with: { node-version: 22 }
      - name: Start DotMock
        id: dotmock
        uses: mockitoHQ/dotmock-cli/action@v1
        with:
          config: dotmock.yaml
          port: "8080"
      - run: npm ci
      - run: npm test
      - name: Show which fixtures answered
        if: always()
        run: dotmock llm journal assistant --local

Assert fixtures from Vitest or Jest

@dotmock/cli/testing is an ESM-only export, so Jest must run in ESM mode. startDotmock starts a server from a config path or inline object (or attaches to DOTMOCK_URL when the GitHub Action already started one) and sets the SDK environment variables. resetDotmock clears sequence counters and the journal; getJournal returns entries oldest first; expectFixtureMatched throws an AssertionError that lists the journal when the expectation fails.

agent.test.ts

import OpenAI from "openai";
import { afterAll, beforeAll, beforeEach, expect, it } from "vitest";
import { expectFixtureMatched, resetDotmock, startDotmock, stopDotmock } from "@dotmock/cli/testing";

let client: OpenAI;

beforeAll(async () => {
  const dotmock = await startDotmock({ config: "dotmock.yaml" });
  client = new OpenAI({ baseURL: dotmock.openaiBaseUrl("assistant"), apiKey: "dotmock" });
}, 120_000);
afterAll(() => stopDotmock());
beforeEach(() => resetDotmock());

it("greets the user", async () => {
  const res = await client.chat.completions.create({
    model: "gpt-4o-mini",
    messages: [{ role: "user", content: "hello" }],
  });
  expect(res.choices[0].message.content).toBe("Hello!");
  await expectFixtureMatched("greeting", { times: 1 });
});

Assert fixtures from pytest

pytest-dotmock registers two fixtures. dotmock_server is session-scoped: it starts or attaches once and exports DOTMOCK_URL, OPENAI_BASE_URL, and ANTHROPIC_BASE_URL. dotmock is function-scoped: the same server with counters and journal reset before each test. Configure with --dotmock-config (ini dotmock_config), --dotmock-port, --dotmock-image (ini dotmock_image), and --dotmock-runtime.

test_assistant.py

# pip install pytest-dotmock
from openai import OpenAI

def test_greeting(dotmock):
    client = OpenAI(base_url=dotmock.openai_base_url(), api_key="dotmock")
    res = client.chat.completions.create(
        model="gpt-4o-mini", messages=[{"role": "user", "content": "hello"}]
    )
    assert res.choices[0].message.content == "Hello!"
    dotmock.assert_fixture_matched("greeting", times=1)

Test against a hosted workspace instead

When the suite should exercise the shared hosted mock, set OPENAI_BASE_URL to $MOCK_URL/v1, send a unique X-Dotmock-Session per test run, and reset that session with dotmock llm reset $API_ID --session <id> or POST /mock-apis/$API_ID/llm-fixtures/reset-sequences. Inject the management credential for those CLI calls from the CI secret store as DOTMOCK_API_KEY; the application under test never needs it.