Case study — 03
A web archiver in Rust that crawls URLs in parallel, writes WARC archives, and flags leaked secrets and API endpoints.
Archiving a site for audit or preservation takes more than a recursive download. Modern sites render in JavaScript, and the output has to follow a standard other preservation tools can read.
Tokio tasks run under a JoinSet and a Semaphore, so many URLs are processed at once without overwhelming the machine.
Pages run their JavaScript before capture, so single-page apps are archived with the content a visitor actually sees.
Secret and endpoint signatures live in YAML files, so they can change without recompiling the tool.
Archives follow ISO 28500 (WARC) for interoperability with other preservation tools. Crawl state is kept in SQLite, so later runs proceed incrementally.