Caching Generated Mock Fixtures in CI

Every CI run regenerates the same mock fixtures from scratch, adding a minute or more to a job whose inputs have not changed since the last green build. Worse, the naive fixes — committing the fixtures or caching them on a branch key — either invite silent drift or serve stale data after the API contract moves. The goal is a cache that hits when the contract is unchanged and misses exactly when it changes.

Why fixtures get regenerated needlessly

Generated fixtures are a pure function of two things: the API specification and the generator that reads it. If neither has changed, the output is identical, so recomputing it is wasted work. Three habits cause the waste:

  • No cache at all. The pipeline runs generate:fixtures unconditionally as a build step, so a spec-unchanged run pays full generation cost.
  • A cache keyed on the wrong thing. Keying on the branch name or a lockfile hash means the cache neither invalidates on a spec change (serving stale fixtures) nor hits across branches that share the same spec.
  • Non-deterministic generation. If the generator is unseeded, two runs on the same spec produce different bytes, so even a correct cache key restores fixtures that no longer match what a fresh generation would produce — and snapshot tests flap.

The fix is a cache keyed on a hash of the schema-driven generation inputs, combined with a seeded generator so the cached output is reproducible. The generation itself relies on deterministic seed management; without a fixed seed the cache is meaningless.

What the cache key has to include The cache key is derived from every input that can change the output: the generator source, the schema, the seed manifest and the generator library version. A change in any of them produces a new key and a regeneration; a change in nothing produces a hit and no work at all. Every input that can change the output must be in the key, or the cache serves stale fixtures. Generator source the script that emits fixtures hashed Schema and seeds the inputs it reads hashed Library version from the lockfile hashed Cache key hit or regenerate deterministic either way A key that omits the library version is the classic mistake: a minor Faker upgrade silently keeps the old data.

Solution

1. Make generation deterministic

Pin the seed so identical inputs yield byte-identical fixtures. Anything less and the cache cannot be trusted:

// scripts/generate-fixtures.ts
import { faker } from '@faker-js/faker';
import { writeFileSync, mkdirSync } from 'node:fs';

faker.seed(Number(process.env.MOCK_SEED ?? 42));
mkdirSync('mocks/generated', { recursive: true });

const users = Array.from({ length: 50 }, () => ({
  id: faker.string.uuid(),
  name: faker.person.fullName(),
  email: faker.internet.email(),
}));

// Stable key order + trailing newline => reproducible bytes across runs.
writeFileSync(
  'mocks/generated/users.json',
  JSON.stringify(users, null, 2) + '\n',
);
console.log(`Wrote ${users.length} users`);

2. Key the cache on the spec hash

actions/cache restores a directory when the key matches a previously saved entry. Build the key from a hash of the spec and the generator version so it changes precisely when either input does:

# .github/workflows/fixtures.yml
name: fixtures
on: [push, pull_request]

jobs:
  build:
    runs-on: ubuntu-latest
    env:
      MOCK_SEED: "42"
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: "20"
          cache: "npm"
      - run: npm ci

      - name: Restore fixture cache
        id: cache
        uses: actions/cache@v4
        with:
          path: mocks/generated
          key: fixtures-v3-${{ hashFiles('specs/api.yaml', 'scripts/generate-fixtures.ts') }}
          restore-keys: |
            fixtures-v3-

      - name: Generate fixtures (cache miss only)
        if: steps.cache.outputs.cache-hit != 'true'
        run: npx tsx scripts/generate-fixtures.ts

      - name: Validate fixtures against the spec
        run: npx tsx scripts/validate-fixtures.ts

      - name: Run tests
        run: npm run test:integration

Two details make this robust. The v3 prefix is a manual bust — bump it when you change the generation logic in a way hashFiles cannot see. The restore-keys fallback lets a spec change restore the previous fixtures as a warm starting point (useful for large fixture sets built incrementally), while the exact-key miss still forces a regenerate.

3. Validate restored fixtures against the spec

A cache hit means the bytes match a prior run, not that they match the current contract if someone bumped the v3 prefix incorrectly. Guard every run — hit or miss — with a validation step so stale or mis-keyed fixtures fail fast:

// scripts/validate-fixtures.ts
import Ajv from 'ajv';
import { readFileSync } from 'node:fs';

const ajv = new Ajv({ allErrors: true });
const schema = JSON.parse(readFileSync('specs/user.schema.json', 'utf8'));
const validate = ajv.compile(schema);
const users = JSON.parse(readFileSync('mocks/generated/users.json', 'utf8'));

const bad = users.filter((u: unknown) => !validate(u));
if (bad.length > 0) {
  console.error('Fixtures violate the schema:', ajv.errorsText(validate.errors));
  process.exit(1);
}
console.log(`All ${users.length} fixtures valid against the spec`);

This validation gate is the same contract check described in Detecting OpenAPI Contract Drift in CI — here it catches a stale cache rather than a drifting mock.

Reading a cache result Four outcomes. A hit with matching inputs is the intended case. A miss after an input change is correct and costs one regeneration. A hit after an input change means the key is incomplete. A miss with unchanged inputs means the key includes something volatile such as a timestamp. Outcome Meaning Action hit, inputs unchanged working as intended none miss, inputs changed correct — regenerate once none hit, inputs changed the key is missing an input add it to the key miss, inputs unchanged the key includes something volatile remove it from the key Rows three and four are both silent: nothing fails, the pipeline is just wrong in one direction or the other.

Verification

Confirm the cache behaves as intended by checking the step outcome in the run log:

gh run view --log | grep -E "Cache (restored|not found)"
# First run on a new spec:  "Cache not found for input keys: fixtures-v3-<hash>"
# Re-run with unchanged spec: "Cache restored from key: fixtures-v3-<hash>"

A green re-run that prints Cache restored and skips the generate step proves the cache is keyed correctly.

Gotchas and edge cases

  • Stale cache after an out-of-band data change. If you tweak generate-fixtures.ts in a way the hash covers, the key changes automatically — but if you change a data source the hash does not cover (an external CSV, a seed constant in another file), the cache silently serves old fixtures. Add every generation input to hashFiles, or bump the v3 prefix by hand when in doubt.
  • Key collisions across specs. A repo with several specs must namespace the key (fixtures-users-${{ hashFiles('specs/users.yaml') }}) so one service’s fixtures never restore into another’s directory. A single flat key across specs is the classic collision that ships the wrong data.
  • Reproducibility breaks without a stable seed and stable key order. Non-deterministic map iteration or an unseeded faker call makes two “identical” runs produce different bytes, so the cache is never actually reused and snapshot tests flap. Sort object keys on write and pin MOCK_SEED, as in step 1.
Cache the derived, never the source Three categories. Generated fixtures are pure functions of their inputs and are ideal to cache. Dependencies are safe to cache keyed on the lockfile. Anything mutated during a run — a request journal, a database, a written-back JSON file — must never be cached, because the next run would inherit the previous run's state. Generated fixtures a pure function of the inputs cache, keyed on all of them Dependencies reproducible from the lockfile cache, keyed on the lockfile Anything mutated at runtime journals, databases, written-back files never — the next run inherits it The third row is how a cached pipeline starts producing results that depend on which job ran before it.

When not to cache fixtures at all

Caching is not free, and there are cases where generating from scratch every run is the better trade.

When generation is fast. If the whole fixture set is produced in under a couple of seconds, a cache restore costs about as much as the work it replaces, and it adds a class of failure — a stale or incorrectly keyed cache — that did not previously exist. Measure before adding one.

When the inputs change constantly. A cache is only useful when it hits. If the generator, the schema and the seeds are all edited in most pull requests, the hit rate approaches zero and every run pays restore, miss and save for no benefit.

When the fixtures are committed. A generated set that lives in the repository is already available on checkout; caching it as well caches something git already provides. In that arrangement the useful check is the opposite one — regenerate and diff, to prove the committed files are current.

When correctness matters more than minutes. On a pipeline where a stale fixture would be genuinely dangerous — a privacy-sensitive dataset, a contract fixture that gates a release — the two minutes saved are not worth the possibility of testing against yesterday’s data because a key omitted an input.

Where a cache does pay, one discipline keeps it safe: the key must include every input, and the easiest way to guarantee that is to hash the directory of generator sources and the lockfile, rather than enumerating files by hand. An enumerated list is a list somebody will forget to update, and the failure it produces — a hit when the inputs changed — is completely silent.

A final note on scope: cache the generated output, never the process. Caching a container image or a warmed database is a different operation with different risks, and conflating the two is how a pipeline ends up inheriting mutable state it was never meant to keep.

← Back to Running Mock Servers in CI Pipelines