WARC file format
WARC is a container format that stores captured resources and associated information as a sequence of typed records.
Record boundaries: a file contains one or more records, and a segmented logical record can span several records. WARC Content-Length counts block octets, excluding the WARC header and trailing separators; an HTTP response block includes its HTTP headers. [1]
Digest scope: block digests exclude WARC headers. Payload digests can describe content absent from a revisit record, whose profile governs interpretation. Retain referenced content as well as the referencing capture. [1]
Access: record-at-a-time gzip permits individual compressed records to be read by offset. A CDX index can support lookup without being part of the WARC record itself. [2]
Relations
- related
- WARC block digest, WARC payload digest, WARC revisit record, Data deduplication, Backup verification
Advantages
- Resources of different formats can share one archival container with associated metadata. [2]
Limitations
- Compression is optional; WARC defines no internal compression scheme. [1]
- Format validity does not establish complete website replay or authenticate capture history. [3]