Physical Address
304 North Cardinal St.
Dorchester Center, MA 02124
ISO 28500:2017 – Information and Documentation – WARC File Format
The standard for web archival storage: understanding the WARC (Web ARChive) file format specification
1. Introduction to ISO 28500 and the WARC File Format
ISO 28500:2017 specifies the WARC (Web ARChive) file format, the international standard for storing web archival content. Developed as the successor to the ARC format (used by the Internet Archive since 1996), WARC addresses the growing need for a standardized, extensible format capable of capturing the complex structure of modern web resources. The format supports storing multiple digital resources — including HTTP responses, DNS records, and metadata — within a single archive file while preserving the relationships between resources and their contextual metadata.
WARC files are the backbone of major web archiving initiatives worldwide, including the Internet Archive’s Wayback Machine, national library web collections, and corporate record-keeping systems. Understanding the WARC format is essential for anyone working in digital preservation, web crawling, or archival data management.
2. WARC File and Record Model
A WARC file is a sequence of concatenated records, each consisting of a header section and an optional content block. The standard defines several mandatory and optional named fields within each record header:
| Field Name | Mandatory | Description |
|---|---|---|
| WARC-Record-ID | Yes | Globally unique identifier for the record (URI format) |
| Content-Length | Yes | Length of the content block in bytes |
| WARC-Date | Yes | Date/time of resource capture (ISO 8601 format, with seconds precision) |
| WARC-Type | Yes | Record type: response, request, resource, metadata, conversion, revisit, warcinfo, or continuation |
| Content-Type | No | MIME type of the content block (e.g., application/http; msgtype=response) |
| WARC-Block-Digest | No | Hash digest of the content block (e.g., sha1:X4QJ7P…) |
| WARC-Payload-Digest | No | Hash digest of the payload within the content block |
| WARC-IP-Address | No | IP address of the server that served the resource |
| WARC-Refers-To | No | Reference to a related record (e.g., the original response for a revisit record) |
The WARC format’s use of mandatory record identifiers and digests ensures data integrity and traceability. Each record has a globally unique ID, making it possible to reference specific captures across different archive files and collections. The combination of block and payload digests allows verification of both the stored content and the extracted payload.
3. Record Types and Their Applications
WARC defines eight record types, each serving a specific function in the web archiving workflow:
3.1 Core Record Types
warcinfo describes the archive file itself (crawler software, operator, parameters). response stores the full HTTP response including headers and body. request captures the HTTP request that generated a response, enabling reconstruction of the complete client-server interaction. resource stores any digital resource not encapsulated in an HTTP envelope, such as FTP downloads or DNS records.
3.2 Metadata and Derived Records
metadata stores supplementary information about another record (e.g., crawl log entries, extracted text). conversion records represent transformed versions of the original resource, such as OCR output or format migrations. revisit records indicate that a resource’s content is identical to a previously captured version, using the WARC-Refers-To field to establish the relationship — this is critical for efficient storage of repeated crawls.
3.3 Continuation Records
The continuation record type allows splitting large content blocks across multiple WARC records, essential when individual resources exceed practical file size limits. This mechanism ensures that WARC remains usable with both strict file size constraints (e.g., 4 GB for some filesystems) and very large web resources.
The WARC format strictly requires that all records within a single WARC file share the same value for the WARC-Date field, with exceptions for warcinfo and continuation records. This requirement ensures chronological ordering within archive files and simplifies temporal navigation during replay.
4. Engineering Design Insights for WARC Implementation
Implementing WARC-based systems requires careful consideration of several engineering aspects:
Compression Strategy: WARC files compress exceptionally well with gzip (typical ratios of 5:1 to 10:1 for HTML content). However, individual record compression (gzip per record) is preferred over file-level compression for random access. The standard recommends using Content-Encoding: gzip at the record level.
File Size Management: Common practice limits individual WARC files to 1 GB uncompressed or approximately 100 MB compressed. This balances storage efficiency with practical handling requirements. The WARC specification permits any file size but recommends implementors support at least 4 GB files.
Deduplication Strategies: Revisit records combined with payload digests enable significant storage savings during repeated crawls. A typical weekly crawl of a 10-million-page site might store only 10-20% new content after the initial crawl, with the remainder stored as revisit records referencing the original capture.
Error Handling: Robust WARC implementations should handle truncated records gracefully, log validation errors, and support partial file recovery. The standard’s record-based structure means that damage to one record does not affect others in the same file.
WARC files must not be modified after creation. The format is designed as a write-once archival format. Any modification invalidates the content digests and compromises data integrity. For corrections or annotations, use metadata or conversion records rather than modifying existing records.
5. Frequently Asked Questions
Q1: What is the relationship between WARC and the older ARC format?
A: WARC is the successor to the ARC format. Key improvements include the addition of named fields (rather than positional), support for multiple record types, embedded digests for integrity checking, and the continuation mechanism for large resources. The Internet Archive and most national libraries have migrated from ARC to WARC.
A: WARC is the successor to the ARC format. Key improvements include the addition of named fields (rather than positional), support for multiple record types, embedded digests for integrity checking, and the continuation mechanism for large resources. The Internet Archive and most national libraries have migrated from ARC to WARC.
Q2: How does WARC handle JavaScript-rendered content?
A: WARC stores the HTTP response as served by the server, which typically contains JavaScript source code rather than rendered output. To archive rendered content, crawlers may use headless browsers and store the post-render DOM via resource or conversion records. This is an active area of development in the web archiving community.
A: WARC stores the HTTP response as served by the server, which typically contains JavaScript source code rather than rendered output. To archive rendered content, crawlers may use headless browsers and store the post-render DOM via resource or conversion records. This is an active area of development in the web archiving community.
Q3: Can WARC files be read without specialized software?
A: Yes, WARC files use plain text headers following RFC 2822 conventions, making them human-readable with standard tools. The content blocks are typically binary (e.g., HTTP responses), but the headers can be inspected with any text editor. Several open-source libraries (warcit, JWAT, warcio) provide programmatic access.
A: Yes, WARC files use plain text headers following RFC 2822 conventions, making them human-readable with standard tools. The content blocks are typically binary (e.g., HTTP responses), but the headers can be inspected with any text editor. Several open-source libraries (warcit, JWAT, warcio) provide programmatic access.
Q4: What is the maximum recommended WARC file size?
A: While the format itself has no maximum size, practical implementations typically limit files to 1 GB uncompressed for manageability. The Internet Archive commonly uses 1 GB uncompressed / ~100 MB compressed files. Files larger than 4 GB may cause compatibility issues with some filesystems and tools.
A: While the format itself has no maximum size, practical implementations typically limit files to 1 GB uncompressed for manageability. The Internet Archive commonly uses 1 GB uncompressed / ~100 MB compressed files. Files larger than 4 GB may cause compatibility issues with some filesystems and tools.
