Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Reports and evidence

What a run prints and saves: console, jsonl, JUnit, Allure, the HTML report, video, traces, watch and shards.

  • --html report.html writes the run as one page that needs nothing else to open: failed scenarios first and open, each with its steps and their times, the failure (message, where the page was, what came closest, the note) and its screenshot in the page (up to 2 MB; a bigger one is linked). Flaky ones show how their first try failed. Secrets are masked as in the console. A screenshot two scenarios share (a failed before all) is in the page once; a bigger one, and any once the page holds 10 MB of them (failures first), is a link relative to the report, so upload the report and .bitti/runs/ together. With --repeat, a pass count per scenario comes first. A run that finds no scenario (exit 2) writes no report.
  • --allure <dir> writes Allure 2 results for allure generate <dir> or an Allure server: one <uuid>-result.json per scenario run with its steps, times, status (a flaky one passed, marked flaky), labels (the file, the .feature’s name, tags, the worker) and the failure’s screenshot attached. The directory is kept, so several runs or shards can feed one report; the history id is the same from run to run.
  • --video on-failure (or all) records what the page showed while the scenario ran, a JPEG each time its pixels changed (none while it is still), into video/ in the scenario’s evidence; video/index.html plays the frames at their pace. The console and JSON lines name it and --html links it. A passing scenario’s video is dropped under on-failure. Recording cost no measurable time on the examples.
  • --trace on-failure (or all) records each step of a scenario: where the page was, the console messages and the requests it brought, in trace.jsonl beside the scenario’s screenshot, with the page’s ax tree at a failure in ax.txt. bitti trace <dir> prints it (a run’s directory lists its traces); bitti trace <dir> <line> shows one step in full, with the tree when it is the one that failed. Each step also keeps the page as it was after it, beside trace.jsonl: NNN-L<line>.ax.txt (its ax tree) and NNN-L<line>.jpg (the viewport, JPEG quality 60, about 10 to 50 kB), both named in the step’s line (ax, shot); trace <dir> <line> shows them. In --html a traced scenario’s steps are a timeline: a row per step (line, step, time, a bar against the slowest) that opens to its screenshot, ax tree, console messages, requests and error, with no script in the page (the images follow the 2 MB and 10 MB limits above). Secrets are masked in the text files and the console as everywhere; a screenshot is pixels, as the failure’s own is.
  • --watch runs the files, then waits: each scenario file that changes (or appears) runs again, its own scenarios only, until Ctrl-C. The files are looked at every 300 ms (no file-watching dependency) and an edit counts once it has stayed quiet for one more look. A file that no longer parses is reported and the watching goes on. The files a use or a flow brings in are not watched on their own: save the scenario file to run it. --junit and --html hold the latest run.
  • bitti merge <dir|file.jsonl>... joins the results of shards into one console summary, one --junit and one --html, as if one run had made them: save each shard’s --reporter jsonl output (bitti run --shard 1/3 --reporter jsonl > shard-1.jsonl), put the files in one place and name it. A directory is searched for .jsonl files. The steps and traces stay with the shards, so the report has each scenario’s status, time and failure; the time is the longest shard’s. Exit 1 when a scenario failed (--fail-on-flaky counts a flaky one), 2 when there are no results.
  • --retries n runs a failing scenario again. One that passes then is reported FLAKY with how its first try failed. --fail-on-flaky fails the run on one. --shard i/n takes every n-th scenario, for CI matrix jobs.
  • --repeat 20 --shuffle hunts flakes and order dependence: every scenario runs 20 times (name #k), in a random order whose seed is printed (--shuffle <seed> runs that order again), then a pass count per scenario, FLAKY where it isn’t all of them.
  • --reporter jsonl prints one JSON object per scenario as it finishes (the console moves to stderr); github writes workflow annotations on the failing lines, beside the console (on by itself under GitHub Actions). --json prints what the scenarios saved, for a pipeline’s next step.

Made from docs/running.md.