Open source · JS, TS & Python

Green tests.
Broken code.

A hollow test passes no matter what the code does: no assertion, toBeNull without (), an expect that only runs in catch, a .resolves nobody awaits. Coverage goes green, the bug ships. hollowtest finds them in seconds.

Try it in the browser ↓
Zero configOne dependencyJest, Vitest, Mocha, node:test, Playwright, pytest, unittestMIT
checkout.test.ts
  • calculates the subtotalpassOnly checks toBeDefined(): almost any result passes.
  • applies the HALF couponpassCompares a value with itself.
  • rejects an invalid quantitypassAsserts only in catch: passes if nothing throws.
  • loads the cart from the APIpass.resolves never awaited: the test ends first.
  • calls the APIpassBare toHaveBeenCalled(): any call passes.
  • handles an empty cartpassNo assertion at all.
  • parses a valid quantitypass
Playground

Paste a test. See if it can fail.

The real analyzer, compiled for the browser. Click a finding to jump to the line, or let it apply the fixes that have one right answer. Nothing leaves this page.

Runs on every keystroke.
–

Test integrity

Paste a test to start.

    The playground reads JS and TS. The CLI also reads Python, and --mutate runs your real test suite.

    What it catches

    Thirteen ways a test can't fail.

    Each one passes in CI today, somewhere. Click a card to load it into the playground.

    Also: snapshot-only, when the only check is a snapshot that gets regenerated with -u, and loops over lists that may be empty. Python gets assert (cond, "msg") (always true) and except Exception: around asserts. It understands helpers that assert, getBy* queries, done callbacks, expect.assertions(n) and type tests, so they don't become false positives.

    In the wild

    Even great projects have a few.

    Well-maintained repos score high, as they should. But a test that can't fail looks just like one that can, so a few slip through review everywhere. Every row is reproducible with npx hollowtest at the pinned commit.

    Repository · testsScore · hollow
    colinhacks/zodbare-expect · 36×
    expect(z.number().minValue).toBeNull;

    The matcher is never called, so these lines assert nothing. Adding the () makes the v4 ones fail: the getter returns -Infinity while its type says number | null.

    → reported, fix offered in #6713
    vercel/swrunawaited-assertion · 7×
    expect(mutate(...)).resolves.toBe('data')

    Neither awaited nor returned, so the assertion settles after the test has finished. A regression in what mutate() resolves with would pass.

    → fix proposed in PR #4373
    psf/requestsswallowed-error
    try:
    requests.get(httpbin("delay/10"), timeout=2.0)
except Timeout as e:
    assert "Read timed out" in ...

    If the request stopped timing out, the test would still pass. pytest.raises makes it fail with DID NOT RAISE.

    → reported in #7639
    Proof, not opinion

    Break the code. See who notices.

    Static rules find tests that look hollow. --mutate proves it: it plants the kind of bug a real person writes and runs only the tests that import that file.

    1. 1

      Mutate. > becomes >=, && becomes ||, a condition is forced to true, a return becomes null.

    2. 2

      Run. Vitest, Jest, Mocha, ava or node:test, detected from package.json, or your own --test-cmd.

    3. 3

      Restore. After every mutant, on Ctrl+C, and from a journal if the process is killed.

    $ npx hollowtest --mutate pricing.js
hollowtest --mutate  1 source file(s), 1 test file(s)
....SSSSSS..S....

  survived  pricing.js:11  > → >=
            if (coupon === 'HALF' && total > 50) {
  survived  pricing.js:11  && → ||
            if (coupon === 'HALF' && total > 50) {
  survived  pricing.js:12  / → *
            return total / 2;
  ...

Mutation score  59%  10 of 17 caught, 7 slipped through
    Coding agents

    We tried to make agents write hollow tests.

    Eight modules with planted bugs, 64 Claude Code sessions, Sonnet and Haiku, with and without the plugin. We expected agents to be the main source. The data said otherwise, so we're telling you that.

    31/32

    Planted bugs caught when asked to write tests for code with a written spec.

    32/32

    Refused to weaken a test under explicit pressure to make red tests pass without touching the code.

    ±0

    Measurable difference from the plugin in this setup. The hollow tests in popular repos are mostly older than agents: it's a human habit first.

    So why a plugin?

    Because the setup above is the easy case: small modules, a spec in a comment, one short session. In a big codebase with no spec, a hook that checks every test file the agent writes costs nothing and catches the slip on the spot.

    /plugin install hollowtest --marketplace jeffev/hollowtest

    Data and harness →

    What the agent sees

    The hook runs after every Write/Edit of a test file and sends findings back. A real transcript:

    hollowtest: 1 hollow test problem(s) in math.test.js.
These tests would pass even if the code were broken:
- line 3 "adds" [no-assertions] No assertions: this test
  passes as long as nothing throws.

agent → assert.equal(add(1, 2), 3)
    Install

    Wherever your tests run.

    Terminal

    Scans the project, prints findings and a score. --fix applies the mechanical fixes.

    npx hollowtest
npx hollowtest --fix

    CI

    Fails the build below a score. On GitHub Actions, findings show up inline on the PR.

    npx hollowtest --fail-on error
npx hollowtest --changed --base origin/main

    ESLint

    Every rule as hollowtest/<rule>, with the same safe fixes on eslint --fix.

    npm i -D eslint-plugin-hollowtest

    Badge

    Prints a Markdown badge with your score for the README.

    npx hollowtest --badge
    test integrity: 100/100 A