Testing and validation
A forensic tool has to be right, not just work. TRACE is tested the way its results will be questioned: against public evidence with published answers, on every platform it runs on, from a clean install.
- automated tests
- ~600
- platforms in CI
- 4
- Python versions
- 3
- images with a full manifest
- 16
Every platform, from a clean install
Section titled “Every platform, from a clean install”Continuous integration (GitHub Actions) runs the real installers – install_windows.ps1 and install.sh, unattended – on fresh machines, then the test suite, on:
- Windows (x64), macOS on Apple Silicon and on Intel, and Linux (Ubuntu 24.04);
- Python 3.10 and 3.14 for every change to
masterand every pull request, with 3.12 added in the weekly run.
A branch push gets a quick run on one machine; every change that reaches master runs on every platform, and the weekly run catches a dependency release that breaks an untouched part. CI tests what a change touched: it works out which test files reach the changed code and which public images those tests need – and anything it cannot place runs everything.
CI also checks that nothing was compiled during the install, and a separate job refuses any disk image or case database committed to the repository.
Public evidence, published answers
Section titled “Public evidence, published answers”The tests download their images from the publishers, checked against recorded SHA-256s, and never use private evidence. Every image and sample, and what it tests →
- DFTT (Digital Forensics Tool Testing) images – FAT, NTFS, ext2/3, ISO 9660, keyword search and carving, each with its published answer key.
- DFRWS 2006 and 2007 forensic challenges – the standard file-carving benchmarks.
- NIST CFReDS deleted-file recovery images.
- NPS (Naval Postgraduate School) disk images, including an E01 whose stored hashes are verified.
- Artifact samples from plaso, dfvfs, EVTX-ATTACK-SAMPLES and others: virtual disks, BitLocker, FileVault and LUKS volumes, shadow copies, APFS, LVM, XFS, AD1 and L01 images, mailboxes – pinned by commit and SHA-256.
Manifests: a whole file system, checked
Section titled “Manifests: a whole file system, checked”For each of 16 public images, a manifest records what TRACE reads: every partition, and for every entry its inode, size, deleted state, every timestamp and its SHA-256. The tests re-read the image and require the result to equal the committed manifest, byte for byte.
That is what proved a 2026 upgrade of The Sleuth Kit changed nothing – and caught the one thing it did: FAT times shifting with the machine’s time zone, now pinned. A manifest is only regenerated after checking the difference against the raw bytes on disk.
Carving, scored against the answer keys
Section titled “Carving, scored against the answer keys”tools/carve_score.py runs the real carvers over the DFTT and DFRWS images and a corpus of real published files, and compares what they find with the answer keys their authors published:
| Test image | Located | Byte-exact |
|---|---|---|
DFTT #11 11-carve-fat.dd |
15 / 15 | – |
DFTT #12 12-carve-ext2.dd |
10 / 10 | 3 / 3 |
| DFRWS 2006 challenge | 27 / 27 | 12 / 12 |
| DFRWS 2007 challenge | 78 / 114 | 16 / 16 |
| Real-file corpus | 63 / 63 | 63 / 63 |
The run fails if a score drops below its recorded baseline, and reassembly has its own baseline – a rebuilt file that regresses to a contiguous carve still locates, so locating alone would not notice. Carves no answer key accounts for are listed: that number is the false-positive rate, and it is watched. The corpus plants 27 signature decoys – a format’s magic followed by junk – and none may be carved.
Checked against reference implementations
Section titled “Checked against reference implementations”Where another trusted tool reads the same thing, TRACE’s output is compared with it on the same files:
| TRACE’s | Compared with |
|---|---|
| event logs, Prefetch, LNK, shell items, registry artifacts | plaso’s published test values, python-evtx |
| PE, ELF and Mach-O parsing | pefile, pyelftools |
| registry transaction-log recovery | yarp – byte-identical recovered hives |
| AFF4 reader | pyaff4’s disk hashes on the AFF4 reference images |
| AD1 hashing | FTK Imager’s own log |
| zstd decoder | facebook/zstd’s golden files, valid and must-fail |
| keyword search | DFTT #2’s published answer key |
| Sigma | SigmaHQ’s 2,547 Windows rules over EVTX-ATTACK-SAMPLES |
The packages test themselves
Section titled “The packages test themselves”Every Windows zip and macOS DMG is unpacked or mounted – into a path with spaces and a non-ASCII letter, or read-only from the DMG – and started with --self-test against public images. Every engine and native library must load, the full case workflow must run, and each image’s manifest must equal the test suite’s. Only then is the package kept, and a release is published only if all of them pass. You can run the same self-test on your own copy.