warc-file-format · shipping · also WARC, Web ARChive

WARC file format

WARC is a container format that stores captured resources and associated information as a sequence of typed records.

Record boundaries: a file contains one or more records, and a segmented logical record can span several records. WARC Content-Length counts block octets, excluding the WARC header and trailing separators; an HTTP response block includes its HTTP headers. [1]

Digest scope: block digests exclude WARC headers. Payload digests can describe content absent from a revisit record, whose profile governs interpretation. Retain referenced content as well as the referencing capture. [1]

Access: record-at-a-time gzip permits individual compressed records to be read by offset. A CDX index can support lookup without being part of the WARC record itself. [2]

Relations

related
WARC block digest, WARC payload digest, WARC revisit record, Data deduplication, Backup verification

Advantages

  • Resources of different formats can share one archival container with associated metadata. [2]

Limitations

  • Compression is optional; WARC defines no internal compression scheme. [1]
  • Format validity does not establish complete website replay or authenticate capture history. [3]

Citations

  1. IIPC WARC 1.1: file and record model, named fields, revisit, segmentation and Annex D (accessed 2026-11-21)
  2. Library of Congress: WARC format description, aggregation, gzip and CDX indexing (accessed 2026-11-21)
  3. Hesela: WARC capture dependencies and bounded recovery controls (accessed 2026-11-21)
Sources & caveats
  • Source synthesis for WARC 1.1; no new parser, capture or replay experiment. [1] [2]
  • Annex D is informative compression guidance, not a requirement for every WARC file. [1]

JSON: /data/terms/warc-file-format.json

Published by Hesela team · Updated

Report a correction · Editorial standards · Reuse guidance

Cite this page
Hesela team. WARC file format. Hesela. 2026-11-21. https://hesela.dev/terms/warc-file-format/