{"number":"05","slug":"technical-osint-tool-library","filename":"05-technical-osint-tool-library.md","title":"Technical and Passive OSINT Tool Library","bytes":16474,"lines":146,"sha256":"5114d06a6c94c04776cb3ac8b91366dcfc13c69b17717e9dec1bd12ab597df22","sections":11,"tables":7,"code_blocks":1,"href":"/v1/docs/technical-osint-tool-library","outline":[{"anchor":"1-mode-legend","level":2,"title":"1. Mode legend","line":5,"tables":["05.1-mode-legend.t1"],"code":[]},{"anchor":"2-deployable-open-source-components","level":2,"title":"2. Deployable open-source components","line":16,"tables":["05.2-deployable-open-source-components.t1"],"code":[]},{"anchor":"conditional-aggregator","level":3,"title":"Conditional aggregator","line":41,"tables":[],"code":[]},{"anchor":"3-hosted-free-open-data-sources","level":2,"title":"3. Hosted free/open-data sources","line":45,"tables":["05.3-hosted-free-open-data-sources.t1"],"code":[]},{"anchor":"4-not-a-commercial-free-core","level":2,"title":"4. Not a commercial free core","line":68,"tables":["05.4-not-a-commercial-free-core.t1"],"code":[]},{"anchor":"5-gaps-a-free-passive-stack-cannot-solve","level":2,"title":"5. Gaps a free passive stack cannot solve","line":82,"tables":[],"code":[]},{"anchor":"6-recommended-passive-core","level":2,"title":"6. Recommended passive core","line":90,"tables":["05.6-recommended-passive-core.t1"],"code":[]},{"anchor":"7-laka-implementation-ladder","level":2,"title":"7. LAKA implementation ladder","line":103,"tables":["05.7-laka-implementation-ladder.t1"],"code":[]},{"anchor":"8-claim-gates-and-freshness","level":2,"title":"8. Claim gates and freshness","line":113,"tables":["05.8-claim-gates-and-freshness.t1"],"code":[]},{"anchor":"9-mandatory-connector-fields","level":2,"title":"9. Mandatory connector fields","line":126,"tables":[],"code":["05.9-mandatory-connector-fields.c1"]}],"blocks":[{"type":"heading","level":1,"text":"Technical and Passive OSINT Tool Library","anchor":"technical-and-passive-osint-tool-library","line":1},{"type":"paragraph","text":"Research snapshot: 2026-09-01. Verify licenses, free-tier limits, API schemas, and source terms again before deployment. An open-source collector does not grant rights to collect from an upstream source.","line":3},{"type":"heading","level":2,"text":"1. Mode legend","anchor":"1-mode-legend","line":5},{"type":"table","headers":["Mode","Meaning"],"align":["left","left"],"rows":[["CM0","query third-party/open datasets or analyze already obtained artifacts; passive core"],["CM1","low-impact direct retrieval such as an ordinary public page/DNS lookup or permitted rendered-page capture; policy and rate review required"],["CM2","active resolution/probing, port/service interaction, or third-party scan submission; excluded and routed to a separately scoped security engagement"],["CM3","authentication, credential validation, exploitation, disruption, or alteration; prohibited in this system"]],"line":7,"id":"05.1-mode-legend.t1"},{"type":"paragraph","text":"Many tools support several modes. The connector profile—not the product name—determines whether use is passive.","line":14},{"type":"heading","level":2,"text":"2. Deployable open-source components","anchor":"2-deployable-open-source-components","line":16},{"type":"table","headers":["Tool","License/status","Purpose and I/O","Integration fit","Passive profile and limitations","Official source"],"align":["left","left","left","left","left","left"],"rows":[["OWASP Amass v5","Apache-2.0; components may vary","domain/org seeds → asset graph, names, IPs, ASNs, relationships, provenance","Go CLI/library, Docker, PostgreSQL","source allow-list; disable active enumeration, brute force, probing and alteration; discovery does not prove ownership","[Repository](https://github.com/owasp-amass/amass) · [Docs](https://owasp-amass.github.io/docs/)"],["SpiderFoot","MIT","multi-source OSINT aggregation and relationship exploration from approved targets","Python CLI/web UI; many independent modules and upstream APIs","create a module-level CM profile; allow only reviewed passive providers, disable active scan/breach/credential modules, and review every upstream key/term/quota separately; aggregator output is a lead, not proof","[Repository and license](https://github.com/smicallef/spiderfoot)"],["ProjectDiscovery Subfinder","MIT","domains → candidate subdomains from passive providers","Go CLI/library, JSON, streams","providers and quotas vary; optional DNS resolution is CM1; keep upstream provenance","[Repository](https://github.com/projectdiscovery/subfinder)"],["Findomain","GPL-3.0","domains → CT/archive/provider candidates, SQLite/monitor outputs","Rust binary, Docker","disable DNS/HTTP/screenshot/port-scan features; review each provider's terms","[Repository](https://github.com/Findomain/Findomain)"],["assetfinder","MIT","domain → related domains/subdomains","small Go streaming CLI","aging provider integrations; lead generator only","[Repository](https://github.com/tomnomnom/assetfinder)"],["Cert Spotter","MPL-2.0","watched domains → new certificate-transparency events","Go CLI/hooks","certificate issuance is historical evidence, not current deployment or ownership","[Repository](https://github.com/SSLMate/certspotter)"],["openrdap/rdap","MIT","domain/IP/ASN/entity → authoritative current registration records","Go CLI/library","redaction and registry rate limits; current, not historical WHOIS","[Repository](https://github.com/openrdap/rdap) · [IANA bootstrap](https://www.iana.org/assignments/rdap-dns/)"],["gau","MIT","domains → URLs known to Wayback, Common Crawl, OTX and urlscan","Go CLI, JSON, filters","URLs may be stale; do not automatically fetch/test them","[Repository](https://github.com/lc/gau)"],["WappalyzerGo","MIT","previously fetched headers/body → possible technology/version map","embedded Go library","heuristic; record matched rule/evidence; never jump to vulnerability claim","[Repository](https://github.com/projectdiscovery/wappalyzergo)"],["webappanalyzer fingerprints","GPL-3.0","headers/HTML/scripts/cookies/DNS → technology matches","versioned fingerprint JSON","license review before embedding; heuristic and version-sensitive","[Repository](https://github.com/enthec/webappanalyzer)"],["WhatWeb","GPL-2.0","public URL → technology fingerprint; JSON/XML/SQL","Ruby CLI, large plugin set","makes direct requests; CM1 with explicit rate/terms profile; aggression level 1 only","[Repository](https://github.com/urbanadventurer/WhatWeb)"],["ExifTool","Artistic License or GPL, same terms as Perl","local document/media → metadata, JSON/CSV","mature CLI/Perl API","isolate untrusted files; metadata can be false; redact author/GPS/PII","[Repository](https://github.com/exiftool/exiftool) · [Site](https://exiftool.org/)"],["Apache Tika","Apache-2.0","local files → MIME, text and metadata","Java library/CLI/REST","containerize without network; file/page/time/memory limits; stay patched","[Project](https://tika.apache.org/)"],["oletools","BSD-style package; bundled parts vary","Office/OLE/RTF/OOXML → macros, objects, DDE/XLM indicators","Python library/CLIs","never execute documents/macros; indicators are not proof of maliciousness","[Repository](https://github.com/decalage2/oletools)"],["FOCA","GPL-3.0","public documents → metadata, paths, names and relationships","Windows GUI/.NET/SQL Server","weaker unattended Linux fit; prefer local analysis and avoid search-engine scraping","[Repository](https://github.com/ElevenPaths/FOCA)"],["Metagoofil maintained fork","GPL-3.0-or-later","domain/query → public document discovery/download","Python/Docker","Google scraping is brittle; no proxies/CAPTCHA/rate evasion; prefer official/archive discovery","[Repository](https://github.com/opsdisk/metagoofil)"],["Gitleaks","MIT","authorized/public repository history → possible secrets, SARIF/JSON/CSV","Go CLI/CI","never validate/use secrets; retain redacted type/path/commit/fingerprint only","[Repository](https://github.com/gitleaks/gitleaks)"],["detect-secrets","Apache-2.0","local files/repos → heuristic secret baseline","Python CLI/library/plugins","use `--no-verify`; detector output is “possible secret,” not proof","[Repository](https://github.com/Yelp/detect-secrets)"],["TruffleHog","AGPL-3.0","repos/files/cloud connectors → secret candidates/provenance","CLI/container","credential verification is CM3 and excluded; use only after validating a no-verification configuration and license fit","[Repository](https://github.com/trufflesecurity/trufflehog)"],["OSV-Scanner","Apache-2.0","authorized manifests/lockfiles/SBOMs/images → package advisory matches","CLI/Go, offline DB, JSON/SARIF/SBOM","public artifact must be confidently attributed; package match does not prove deployed/reachable code","[Repository](https://github.com/google/osv-scanner)"]],"line":18,"id":"05.2-deployable-open-source-components.t1"},{"type":"heading","level":3,"text":"Conditional aggregator","anchor":"conditional-aggregator","line":41},{"type":"paragraph","text":"[theHarvester](https://github.com/laramies/theHarvester) distinguishes passive, DNS, and active HTTP/TLS/scan sources and can emit JSONL/SQLite with provenance. Only passive and ordinary DNS profiles map to CM0/CM1 here; its active modes map to excluded CM2. The repository's licensing should be rechecked before vendoring, and every upstream provider must pass its own commercial-use review.","line":43},{"type":"heading","level":2,"text":"3. Hosted free/open-data sources","anchor":"3-hosted-free-open-data-sources","line":45},{"type":"table","headers":["Service","Status/access","I/O and fit","Current limitations and safe interpretation","Official source"],"align":["left","left","left","left","left"],"rows":[["crt.sh","no-key hosted CT search","domain/pattern → certificate/name records; commonly JSON","no formal API SLA/quota; throttle/cache; issuance ≠ current asset","[crt.sh](https://crt.sh/)"],["Censys Free","hosted account/PAT","host/web/certificate search and lookups; official SDKs/CLI","about 100 credits/month; standard query 5, lookup 1; limited pages/concurrency; no Free vuln/history timeline","[Entitlements](https://docs.censys.com/docs/data-access-tiers-entitlements) · [API](https://docs.censys.com/reference/get-started)"],["BuiltWith Free API","key, hosted free endpoint","root domain → technology group/category counts and first/last indexed dates","1 request/second; not the paid detailed technology inventory; root domains only","[Free API](https://api.builtwith.com/free-api)"],["GreyNoise Community","keyless limited or free key","IP → noise/RIOT/classification/last-seen context","limited; free-key allowance documented around 50 searches/week; does not find target assets or prove compromise","[Docs](https://docs.greynoise.io/docs/using-the-greynoise-community-api)"],["urlscan.io Search","API key","existing scan index → page/network/screenshot metadata","search-only is CM0; submitting scans is CM2 and can leak URLs/tokens; dynamic quotas and visibility classes","[Search API](https://docs.urlscan.io/apis/urlscan-openapi/search) · [MIT client](https://github.com/urlscan/urlscan-python)"],["AlienVault OTX","community API","domain/IP/URL/hash → community IOC/pulse context","community data needs corroboration; IOC association ≠ compromise/ownership","[API](https://otx.alienvault.com/api) · [Apache-2.0 SDK](https://github.com/AlienVault-OTX/OTX-Python-SDK)"],["Common Crawl","free CDXJ/JSON index and WARC","URL/domain → captures/status/MIME/WARC; strong offline history","incomplete/stale; source copyright/privacy persists; prefer bulk data to hammering index","[Index](https://commoncrawl.org/cdxj-index) · [Terms](https://commoncrawl.org/terms-of-use)"],["Wayback Machine","public archive","URL → captures and snapshots","incomplete/excluded/throttled; capture date ≠ publication; Save Page Now is an external write","[Wayback](https://wayback.archive.org/) · [Help](https://help.archive.org/help/using-the-wayback-machine/)"],["Arquivo.pt","free archive APIs, GPL-3.0 stack","full-text/images/history/Memento → captures","coverage weighted toward Portuguese/selected international web; historical evidence only","[Repository](https://github.com/arquivo/pwa-technologies)"],["GitHub REST Code Search","authenticated public-code search","org/repo/query → repo/path/SHA/fragments","max 1,000 results, default branch and query/file limits, search rate limits; never validate secrets","[Docs](https://docs.github.com/en/rest/search/search#search-code)"],["Have I Been Pwned (HIBP) Pwned Passwords","no-key k-anonymity API/downloads","hash prefix → suffix/count","for authorized password hygiene, not prospect discovery; never send/store plaintext","[API](https://haveibeenpwned.com/API/v3#PwnedPasswords)"],["NVD","free feeds/API","CVE/CPE/CVSS/references → vulnerability knowledge","no-key 5 requests/30s; key 50/30s; banner CPE ≠ target vulnerability","[API](https://nvd.nist.gov/developers/vulnerabilities)"],["CISA KEV","free JSON/CSV","CVE → known-exploited status/dates","exploitation observed somewhere, not evidence target is affected/exploited","[Catalog](https://www.cisa.gov/known-exploited-vulnerabilities-catalog)"],["FIRST EPSS","free daily CSV/API","CVE → estimated 30-day exploitation probability","probability ≠ exposure/impact; do not multiply blindly by CVSS; treat KEV separately","[FAQ/API](https://www.first.org/epss/faq)"],["OSV API","free","package/version/commit → advisories","needs reliable package/version; no deployment/reachability proof","[API](https://google.github.io/osv.dev/api/)"],["GitHub Advisory Database","CC-BY-4.0 data","cloneable OSV JSON advisories","same applicability limits; attribution required","[Repository](https://github.com/github/advisory-database)"],["ProjectDiscovery Chaos","account/API","domain → known subdomains; MIT client","commercial/free entitlement is not guaranteed; fail closed until recorded","[Site](https://chaos.projectdiscovery.io/) · [Client](https://github.com/projectdiscovery/chaos-client)"],["LeakIX API","free key advertised for researchers","domain/IP → indexed services/leak records","researcher access may not authorize commercial prospecting; obtain terms clarity; cached records need corroboration","[Docs](https://leakix.net/api-documentation)"]],"line":47,"id":"05.3-hosted-free-open-data-sources.t1"},{"type":"heading","level":2,"text":"4. Not a commercial free core","anchor":"4-not-a-commercial-free-core","line":68},{"type":"table","headers":["Tool/source","Why excluded from the default"],"align":["left","left"],"rows":[["[Shodan InternetDB](https://internetdb.shodan.io/)","free access is non-commercial; commercial audits need licensing; cached `vulns` are heuristic"],["[Netlas Community](https://netlas.io/pricing/)","free Community plan is explicitly personal-use"],["[VirusTotal Public API](https://docs.virustotal.com/reference/public-vs-premium-api)","public API is noncommercial/academic and restricted for business/product use"],["[SecurityTrails API](https://securitytrails.com/corp/api)","API/historical passive DNS/WHOIS access is paid; do not call it free"],["[Google Custom Search JSON API](https://developers.google.com/custom-search/v1/overview)","closed to new customers; legacy users must transition before 2027-01-01"],["HIBP email/domain search","account lookups require subscription; domain enumeration requires proof of control; not a free “dark web scan”"],["Nmap, Masscan, Naabu, Nuclei, sqlmap, testssl.sh, scan submissions","direct active interaction; separate written-authorization workflow"],["credential-verification features","prohibited in prospect research; do not authenticate, redeem, or call detected secrets"],["private dark-web database claims","no legitimate comprehensive free/OSS source; licensed specialist plus explicit client authority required"]],"line":70,"id":"05.4-not-a-commercial-free-core.t1"},{"type":"heading","level":2,"text":"5. Gaps a free passive stack cannot solve","anchor":"5-gaps-a-free-passive-stack-cannot-solve","line":82},{"type":"list","ordered":true,"checklist":false,"items":[{"text":"Comprehensive historical WHOIS and passive DNS are largely commercial.","depth":0,"checked":null},{"text":"Current TLS protocols/ciphers require a recent cached observation or an authorized direct assessment; CT cannot show them.","depth":0,"checked":null},{"text":"Public breach sources cannot prove a password still works, MFA is absent, or the target itself was breached.","depth":0,"checked":null},{"text":"A passive technology fingerprint cannot establish reachable exploitability.","depth":0,"checked":null},{"text":"Shared cloud/CDN IPs cannot establish asset ownership.","depth":0,"checked":null}],"line":84},{"type":"heading","level":2,"text":"6. Recommended passive core","anchor":"6-recommended-passive-core","line":90},{"type":"table","headers":["Need","First choice","Secondary/conditional"],"align":["left","left","left"],"rows":[["entity/domain registration","RDAP + official company/registry records","current DNS under CM1 profile"],["subdomain/certificate history","crt.sh/Cert Spotter + Subfinder passive sources","Amass passive graph"],["historical URLs/pages","Common Crawl + Wayback + Arquivo.pt","gau as adapter"],["technology hints","WappalyzerGo on permitted/cached HTML","BuiltWith free group signal"],["public documents","ordinary permitted discovery + Tika/ExifTool in isolated local worker","FOCA/manual review"],["public code/dependencies","GitHub official search + OSV-Scanner","Gitleaks/detect-secrets with redaction/no verification"],["advisory enrichment","OSV + NVD + CISA KEV + EPSS","GitHub Advisory DB offline mirror"],["cached service context","Censys Free/urlscan search under quota and terms","OTX/GreyNoise context"]],"line":92,"id":"05.6-recommended-passive-core.t1"},{"type":"heading","level":2,"text":"7. LAKA implementation ladder","anchor":"7-laka-implementation-ladder","line":103},{"type":"table","headers":["Change state","Technical intelligence capability"],"align":["left","left"],"rows":[["Baseline","manual official-domain/registry/CT/archive snapshot with source ledger"],["Minor","scheduled passive adapters, normalization, provenance and change alerts"],["Major","asset graph, attribution confidence, SBOM/advisory enrichment and human triage"],["Structural","event-sourced evidence, terms registry, redaction/quarantine, responsible-disclosure route"],["Paradigm","client-authorized continuous exposure-management product where evidence is tied to business dependencies and interventions—not sales scare tactics"]],"line":105,"id":"05.7-laka-implementation-ladder.t1"},{"type":"heading","level":2,"text":"8. Claim gates and freshness","anchor":"8-claim-gates-and-freshness","line":113},{"type":"table","headers":["Claim","Minimum gate"],"align":["left","left"],"rows":[["owned asset","current RDAP/DNS/client inventory or strong target-controlled linkage; shared IP never enough"],["current exposed service","recent provider last-seen plus current linkage or authorized validation"],["vulnerable system","exact product/version/config applicability + authoritative advisory + confirmed reachable authorized asset"],["breach exposure","“identifier appeared in named corpus”; never current password/MFA/company breach claim"],["secret exposure","“possible exposed secret”; redacted fingerprint and provenance; no validation"],["negative result","“not observed in sources checked as of [UTC],” never “secure/none”"]],"line":115,"id":"05.8-claim-gates-and-freshness.t1"},{"type":"paragraph","text":"Refresh DNS/RDAP used in outreach within 24–72 hours, technology hints within 30 days, and NVD/KEV/EPSS/OSV daily. Always show the last-seen date of cached service evidence.","line":124},{"type":"heading","level":2,"text":"9. Mandatory connector fields","anchor":"9-mandatory-connector-fields","line":126},{"type":"code","lang":"yaml","content":"connector_id: \"\"\nmode: CM0|CM1|CM2|CM3\ncommercial_ok: true|false|unknown\nautomation_ok: true|false|unknown\nredistribution_ok: true|false|unknown\nauthentication: none|key|oauth|verified-domain|contract\nquota: \"\"\ncache_rule: \"\"\ndeletion_rule: \"\"\nattribution_rule: \"\"\nlicense: \"\"\nterms_url: \"\"\nterms_reviewed_at: \"\"\nfail_closed: true","line":128,"id":"05.9-mandatory-connector-fields.c1"},{"type":"paragraph","text":"Unknown rights fail closed.","line":145}]}