Technical and passive OSINT · Hosted free and open-data sources

Common Crawl

Licence or access: free CDXJ/JSON index and WARC

status access: free CDXJ/JSON index and WARC
i o and fit: URL/domain → captures/status/MIME/WARC; strong offline history
current limitations and safe interpretation: incomplete/stale; source copyright/privacy persists; prefer bulk data to hammering index
official source: Index · Terms

Official sources

Index https://commoncrawl.org/cdxj-index
Terms https://commoncrawl.org/terms-of-use

Catalogued at the 2026-09-01 research baseline. Verify the licence, free-tier limits, API schema and source terms again before deployment, and record a version-pinned registry entry for whatever you select.

Source: file 05 · 3-hosted-free-open-data-sources · line 47

More from this library GET /v1/tools/common-crawl