Testing LLM apps in CI with hosted DotMock

Keep the fixtures in the repository, apply them to a hosted LLM mock from the pipeline, give every run its own session, and assert which fixture answered each request. The mock runs in DotMock's cloud; the job only needs an API key secret.

The GitHub Action, @dotmock/cli/testing, and pytest-dotmock ship with @dotmock/cli 0.3.1 and are in preview. The mock itself is the hosted DotMock runtime.

In this guide

Store a DotMock API key as a CI secret
Apply config-as-code from the repo
Isolate each run with a session
Assert fixtures from Vitest, Jest, or pytest

Prerequisites

A hosted LLM DotMock workspace and its ID in $API_ID
A team-scoped DotMock API key stored as the DOTMOCK_API_KEY CI secret
Node.js 18+ for the CLI and test helpers, or Python for pytest-dotmock

On this page

Store the API key as a CI secret
Keep the fixtures in the repository
Isolate each run with a session
Use the GitHub Action
Assert fixtures from Vitest or Jest
Assert fixtures from pytest

Store the API key as a CI secret

Create a team-scoped API key in the DotMock dashboard and save it in your CI secret store as DOTMOCK_API_KEY. The CLI, the GitHub Action, and the test helpers use it to apply configuration, reset sequences, and read the journal.

The application under test never needs it. Provider SDKs pointed at the mock take any non-empty placeholder key, so never pass the DotMock API key as an OpenAI or Anthropic key.

Keep the fixtures in the repository

dotmock init --llm scaffolds dotmock.yaml, a dotmock/v2 definition with kind: llm, runtime settings under protocol.settings, and starter greeting, tool-call, structured-output, refusal, and rate-limit fixtures. Fixture fields are identical to the hosted workspace.

Review changes to the file like any other code. dotmock config apply syncs fixtures (matched by name) and settings to the hosted API; the first apply creates the API and writes its id and stored subdomain back into the file so later applies target the same API. Apply validates the whole file first, applies settings before fixtures, and rolls back if a fixture is rejected. LLM fixtures are live as soon as they are applied, so the fixtures a test depends on ship in the same pull request as the test.

One-time setup

npx @dotmock/cli init --llm          # scaffolds dotmock.yaml
npx @dotmock/cli login               # browser login for local work
npx @dotmock/cli config apply        # creates or updates the hosted LLM mock
npx @dotmock/cli llm connect assistant # prints the base URLs and SDK snippets

dotmock.yaml (excerpt)

schemaVersion: dotmock/v2
kind: llm
name: Assistant
subdomain: assistant
protocol:
  source: llm
  settings:
    fallback: { type: none }
rules: []
fixtures:
  - name: greeting
    priority: 10
    match:
      userMessage: /\b(hi|hello|hey)\b/
    response:
      content: Hello! I'm a DotMock LLM mock. How can I help you today?
      finishReason: stop
  # ... weather-tool-call, structured-output, refusal, rate-limit
  - name: fallback
    priority: 1000
    match: {}
    response:
      content: This is a DotMock mock response. Add a fixture in dotmock.yaml to customize it.
      finishReason: stop

Isolate each run with a session

Sequence counters and journal entries are keyed by API and X-Dotmock-Session. Give every CI run, and ideally every test, its own session ID so parallel jobs share one hosted mock without advancing each other's sequenceIndex fixtures or reading each other's traffic.

Send the header from the client under test, reset the session before a test that relies on sequences, and filter the journal by the same session when asserting.

Client

const client = new OpenAI({
  baseURL: process.env.OPENAI_BASE_URL,   // $MOCK_URL/v1
  apiKey: "dotmock",                      // placeholder, never the DotMock API key
  defaultHeaders: { "X-Dotmock-Session": process.env.DOTMOCK_SESSION ?? "local" },
});

Reset and inspect from the CLI

dotmock llm reset $API_ID --session "$DOTMOCK_SESSION"
dotmock llm journal $API_ID --session "$DOTMOCK_SESSION" --limit 100

Use the GitHub Action

mockitoHQ/dotmock-cli/action@v1 connects the job to a hosted LLM mock. It takes the api-key secret, optionally runs dotmock config apply on a config file from the repository, and exports OPENAI_BASE_URL, ANTHROPIC_BASE_URL, DOTMOCK_URL, DOTMOCK_SESSION, and DOTMOCK_API to later steps, with the key masked for the CLI and test helpers. The session defaults to the workflow run ID and attempt, so reruns and parallel jobs stay isolated; a post step prints the session's journal summary.

Inputs: api-key (required), api, config, session, cli-version, api-url, working-directory, export-dummy-keys, journal-summary
Outputs: url, openai-base-url, anthropic-base-url, api-id, session

.github/workflows/test.yml

jobs:
  test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v5
      - uses: actions/setup-node@v5
        with: { node-version: 22 }
      - name: Connect DotMock
        uses: mockitoHQ/dotmock-cli/action@v1
        with:
          api-key: ${{ secrets.DOTMOCK_API_KEY }}
          config: dotmock.yaml   # optional: apply fixtures from the repo first
          # api: assistant       # id or subdomain when no config is applied
      - run: npm ci
      - run: npm test
      - name: Show which fixtures answered
        if: always()
        run: npx @dotmock/cli llm journal "$DOTMOCK_API" --session "$DOTMOCK_SESSION"

Assert fixtures from Vitest or Jest

@dotmock/cli/testing is an ESM-only export, so Jest must run in ESM mode. The helpers talk to the hosted API with DOTMOCK_API_KEY (or the key saved by dotmock login) and read DOTMOCK_API and DOTMOCK_SESSION from the environment, so inside the GitHub Action no arguments are needed. connectDotmock resolves the hosted base URLs and the session header and resets that session, resetDotmock clears the session's sequence counters, getJournal returns the session's entries, and expectFixtureMatched throws an AssertionError that lists the journal when the expectation fails.

agent.test.ts

import OpenAI from "openai";
import { beforeAll, beforeEach, expect, it } from "vitest";
import { connectDotmock, expectFixtureMatched, resetDotmock } from "@dotmock/cli/testing";

let client: OpenAI;

beforeAll(async () => {
  const dotmock = await connectDotmock({ api: "assistant" }); // id, subdomain, or name; default $DOTMOCK_API
  client = new OpenAI({
    baseURL: dotmock.openaiBaseUrl,
    apiKey: "dotmock",
    defaultHeaders: dotmock.headers, // X-Dotmock-Session
  });
});
beforeEach(() => resetDotmock());

it("greets the user", async () => {
  const res = await client.chat.completions.create({
    model: "gpt-4o-mini",
    messages: [{ role: "user", content: "hello" }],
  });
  expect(res.choices[0].message.content).toMatch(/^Hello! I'm a DotMock LLM mock/);
  await expectFixtureMatched("greeting", { times: 1 });
});

Assert fixtures from pytest

pytest-dotmock is not on PyPI yet; install it from the dotmock-cli repository (pin a tag for reproducible CI). It works against the hosted mock. It reads DOTMOCK_API_KEY and the API from --dotmock-api, DOTMOCK_API, or the dotmock_api ini key, and skips when no key is set. dotmock_api is session-scoped; the function-scoped dotmock fixture gives each test its own X-Dotmock-Session (exported as DOTMOCK_SESSION), resets it before the test, and exposes the hosted base URLs, headers, the journal, and assert_fixture_matched.

test_assistant.py

# Not on PyPI yet; install from the repository:
# pip install "git+https://github.com/mockitoHQ/dotmock-cli#subdirectory=integrations/python"
from openai import OpenAI

def test_greeting(dotmock):
    client = OpenAI(
        base_url=dotmock.openai_base_url(),
        api_key="dotmock",
        default_headers=dotmock.headers,
    )
    res = client.chat.completions.create(
        model="gpt-4o-mini", messages=[{"role": "user", "content": "hello"}]
    )
    assert res.choices[0].message.content.startswith("Hello! I'm a DotMock LLM mock")
    dotmock.assert_fixture_matched("greeting", times=1)