Test Infrastructure

INFO

This article serves as an invitation for sharing ideas to improve our test infrastructure, feel free to contact us on Discord.

In Oxc, correctness and reliability are taken extremely seriously.

We spend a great deal of time strengthening the test infrastructure to prevent problems from propagating to downstream tools.

Parser

Conformance

Parser tests from Test262, Babel, and TypeScript are used to test JavaScript, TypeScript, and JSX syntax.

For Test262, all stage 4 and regular expression tests are included.

All conformance results are stored in a snapshot file for tracking changes:

• test262.snap, reporting 43765/43765 (100.00%).
• babel.snap, reporting 2093/2101 (99.62%).
• typescript.snap, reporting 6470/6479 (99.86%).

All syntax errors are written to these snapshot files for diffing changes.

Fuzzing

To ensure that the parser does not panic when encountering random data, three fuzzers are used:

1. cargo fuzz for sending random bytes to the parser.
2. shift-fuzzer-js by bakkot for producing random but valid ASTs.
3. Automated-Fuzzer by qarmin, which actively reports crashes.

Memory Safety

The oxc parser uses bumpalo as the memory allocator for its AST. None of the AST nodes have a drop implementation. Miri is used to ensure that no heap-allocated data is stored on the AST nodes, which would result in memory leaks when the allocator is dropped.

Linter

Snapshot Diagnostics

All linter diagnostics are written to a snapshot file for testing against regressions.

For example:

  ⚠ typescript-eslint(adjacent-overload-signatures): All "foo" signatures should be adjacent.
   ╭─[adjacent_overload_signatures.tsx:3:18]
 2 │         function foo(s: string);
 3 │         function foo(n: number);
   ·                  ───
 4 │         type bar = number;
 5 │         function foo(sn: string | number) {}
   ·                  ───
 6 │       }
   ╰────

Ecosystem CI

oxlint-ecosystem-ci runs oxlint against large repositories to check for false positives, regressions, and panics. The repositories tested include:

• rolldown/rolldown
• napi-rs/napi-rs
• toeverything/affine
• preactjs/preact
• microsoft/vscode
• bbc/simorgh
• elastic/kibana
• DefinitelyTyped/DefinitelyTyped

Idempotency

Idempotency testing is used for integration tests and end-to-end tests on all tools.

An idempotency test follows this procedure:

let sourceText = "foo";
let printed = tool(sourceText);
let printed2 = tool(printed);
assert(printed == printed2);

For example, idempotently minifying a piece of code should yield the same result.

All tools (parser, transformer, minifer, etc.) are idempotently tested on Test262, Babel and TypeScript test files.

Integration Tests

Integration tests are preferred over unit tests.

codecov currently reports line coverage.

End to End

The repository monitor-oxc performs end-to-end tests against the top 3000 npm packages from npm-high-impact.

Its package.json has 3000 dependencies:

"devDependencies": {
  "@aashutoshrathi/word-wrap": "latest",
  "@actions/http-client": "latest",
  "@adobe/css-tools": "latest",
  "@alloc/quick-lru": "latest",
 ...
  "zip-stream": "latest",
  "zod": "latest",
  "zone.js": "latest",
  "zustand": "latest"
}

And a test file that imports these packages and asserts the import:

src/dynamic.test.mjs

import test from "node:test";
import assert from "node:assert";
test("@aashutoshrathi/word-wrap", () => import("@aashutoshrathi/word-wrap").then(assert.ok));
test("@actions/http-client", () => import("@actions/http-client").then(assert.ok));
test("@adobe/css-tools", () => import("@adobe/css-tools").then(assert.ok));
test("@alloc/quick-lru", () => import("@alloc/quick-lru").then(assert.ok));
...
test("zod", () => import("zod").then(assert.ok));
test("zone.js", () => import("zone.js").then(assert.ok));
test("zustand", () => import("zustand").then(assert.ok));
test("zwitch", () => import("zwitch").then(assert.ok));

This test file is run after each of the tools (codegen, transformer, minifier, etc.) rewrites all the files in node_modules.

The packages are updated to their latest versions daily.

This setup has caught many obscure bugs that the conformance test suites missed.

---

If you any ideas on how to improve our test infrastructure, feel free to contact us on Discord.