File 05
Technical and Passive OSINT Tool Library
Research snapshot: 2026-09-01. Verify licenses, free-tier limits, API schemas, and source terms again before deployment. An open-source collector does not grant rights to collect from an upstream source.
1. Mode legend
| Mode | Meaning |
|---|---|
| CM0 | query third-party/open datasets or analyze already obtained artifacts; passive core |
| CM1 | low-impact direct retrieval such as an ordinary public page/DNS lookup or permitted rendered-page capture; policy and rate review required |
| CM2 | active resolution/probing, port/service interaction, or third-party scan submission; excluded and routed to a separately scoped security engagement |
| CM3 | authentication, credential validation, exploitation, disruption, or alteration; prohibited in this system |
Many tools support several modes. The connector profile—not the product name—determines whether use is passive.
2. Deployable open-source components
| Tool | License/status | Purpose and I/O | Integration fit | Passive profile and limitations | Official source |
|---|---|---|---|---|---|
| OWASP Amass v5 | Apache-2.0; components may vary | domain/org seeds → asset graph, names, IPs, ASNs, relationships, provenance | Go CLI/library, Docker, PostgreSQL | source allow-list; disable active enumeration, brute force, probing and alteration; discovery does not prove ownership | Repository · Docs |
| SpiderFoot | MIT | multi-source OSINT aggregation and relationship exploration from approved targets | Python CLI/web UI; many independent modules and upstream APIs | create a module-level CM profile; allow only reviewed passive providers, disable active scan/breach/credential modules, and review every upstream key/term/quota separately; aggregator output is a lead, not proof | Repository and license |
| ProjectDiscovery Subfinder | MIT | domains → candidate subdomains from passive providers | Go CLI/library, JSON, streams | providers and quotas vary; optional DNS resolution is CM1; keep upstream provenance | Repository |
| Findomain | GPL-3.0 | domains → CT/archive/provider candidates, SQLite/monitor outputs | Rust binary, Docker | disable DNS/HTTP/screenshot/port-scan features; review each provider's terms | Repository |
| assetfinder | MIT | domain → related domains/subdomains | small Go streaming CLI | aging provider integrations; lead generator only | Repository |
| Cert Spotter | MPL-2.0 | watched domains → new certificate-transparency events | Go CLI/hooks | certificate issuance is historical evidence, not current deployment or ownership | Repository |
| openrdap/rdap | MIT | domain/IP/ASN/entity → authoritative current registration records | Go CLI/library | redaction and registry rate limits; current, not historical WHOIS | Repository · IANA bootstrap |
| gau | MIT | domains → URLs known to Wayback, Common Crawl, OTX and urlscan | Go CLI, JSON, filters | URLs may be stale; do not automatically fetch/test them | Repository |
| WappalyzerGo | MIT | previously fetched headers/body → possible technology/version map | embedded Go library | heuristic; record matched rule/evidence; never jump to vulnerability claim | Repository |
| webappanalyzer fingerprints | GPL-3.0 | headers/HTML/scripts/cookies/DNS → technology matches | versioned fingerprint JSON | license review before embedding; heuristic and version-sensitive | Repository |
| WhatWeb | GPL-2.0 | public URL → technology fingerprint; JSON/XML/SQL | Ruby CLI, large plugin set | makes direct requests; CM1 with explicit rate/terms profile; aggression level 1 only | Repository |
| ExifTool | Artistic License or GPL, same terms as Perl | local document/media → metadata, JSON/CSV | mature CLI/Perl API | isolate untrusted files; metadata can be false; redact author/GPS/PII | Repository · Site |
| Apache Tika | Apache-2.0 | local files → MIME, text and metadata | Java library/CLI/REST | containerize without network; file/page/time/memory limits; stay patched | Project |
| oletools | BSD-style package; bundled parts vary | Office/OLE/RTF/OOXML → macros, objects, DDE/XLM indicators | Python library/CLIs | never execute documents/macros; indicators are not proof of maliciousness | Repository |
| FOCA | GPL-3.0 | public documents → metadata, paths, names and relationships | Windows GUI/.NET/SQL Server | weaker unattended Linux fit; prefer local analysis and avoid search-engine scraping | Repository |
| Metagoofil maintained fork | GPL-3.0-or-later | domain/query → public document discovery/download | Python/Docker | Google scraping is brittle; no proxies/CAPTCHA/rate evasion; prefer official/archive discovery | Repository |
| Gitleaks | MIT | authorized/public repository history → possible secrets, SARIF/JSON/CSV | Go CLI/CI | never validate/use secrets; retain redacted type/path/commit/fingerprint only | Repository |
| detect-secrets | Apache-2.0 | local files/repos → heuristic secret baseline | Python CLI/library/plugins | use --no-verify; detector output is “possible secret,” not proof | Repository |
| TruffleHog | AGPL-3.0 | repos/files/cloud connectors → secret candidates/provenance | CLI/container | credential verification is CM3 and excluded; use only after validating a no-verification configuration and license fit | Repository |
| OSV-Scanner | Apache-2.0 | authorized manifests/lockfiles/SBOMs/images → package advisory matches | CLI/Go, offline DB, JSON/SARIF/SBOM | public artifact must be confidently attributed; package match does not prove deployed/reachable code | Repository |
Conditional aggregator
theHarvester distinguishes passive, DNS, and active HTTP/TLS/scan sources and can emit JSONL/SQLite with provenance. Only passive and ordinary DNS profiles map to CM0/CM1 here; its active modes map to excluded CM2. The repository's licensing should be rechecked before vendoring, and every upstream provider must pass its own commercial-use review.
3. Hosted free/open-data sources
| Service | Status/access | I/O and fit | Current limitations and safe interpretation | Official source |
|---|---|---|---|---|
| crt.sh | no-key hosted CT search | domain/pattern → certificate/name records; commonly JSON | no formal API SLA/quota; throttle/cache; issuance ≠ current asset | crt.sh |
| Censys Free | hosted account/PAT | host/web/certificate search and lookups; official SDKs/CLI | about 100 credits/month; standard query 5, lookup 1; limited pages/concurrency; no Free vuln/history timeline | Entitlements · API |
| BuiltWith Free API | key, hosted free endpoint | root domain → technology group/category counts and first/last indexed dates | 1 request/second; not the paid detailed technology inventory; root domains only | Free API |
| GreyNoise Community | keyless limited or free key | IP → noise/RIOT/classification/last-seen context | limited; free-key allowance documented around 50 searches/week; does not find target assets or prove compromise | Docs |
| urlscan.io Search | API key | existing scan index → page/network/screenshot metadata | search-only is CM0; submitting scans is CM2 and can leak URLs/tokens; dynamic quotas and visibility classes | Search API · MIT client |
| AlienVault OTX | community API | domain/IP/URL/hash → community IOC/pulse context | community data needs corroboration; IOC association ≠ compromise/ownership | API · Apache-2.0 SDK |
| Common Crawl | free CDXJ/JSON index and WARC | URL/domain → captures/status/MIME/WARC; strong offline history | incomplete/stale; source copyright/privacy persists; prefer bulk data to hammering index | Index · Terms |
| Wayback Machine | public archive | URL → captures and snapshots | incomplete/excluded/throttled; capture date ≠ publication; Save Page Now is an external write | Wayback · Help |
| Arquivo.pt | free archive APIs, GPL-3.0 stack | full-text/images/history/Memento → captures | coverage weighted toward Portuguese/selected international web; historical evidence only | Repository |
| GitHub REST Code Search | authenticated public-code search | org/repo/query → repo/path/SHA/fragments | max 1,000 results, default branch and query/file limits, search rate limits; never validate secrets | Docs |
| Have I Been Pwned (HIBP) Pwned Passwords | no-key k-anonymity API/downloads | hash prefix → suffix/count | for authorized password hygiene, not prospect discovery; never send/store plaintext | API |
| NVD | free feeds/API | CVE/CPE/CVSS/references → vulnerability knowledge | no-key 5 requests/30s; key 50/30s; banner CPE ≠ target vulnerability | API |
| CISA KEV | free JSON/CSV | CVE → known-exploited status/dates | exploitation observed somewhere, not evidence target is affected/exploited | Catalog |
| FIRST EPSS | free daily CSV/API | CVE → estimated 30-day exploitation probability | probability ≠ exposure/impact; do not multiply blindly by CVSS; treat KEV separately | FAQ/API |
| OSV API | free | package/version/commit → advisories | needs reliable package/version; no deployment/reachability proof | API |
| GitHub Advisory Database | CC-BY-4.0 data | cloneable OSV JSON advisories | same applicability limits; attribution required | Repository |
| ProjectDiscovery Chaos | account/API | domain → known subdomains; MIT client | commercial/free entitlement is not guaranteed; fail closed until recorded | Site · Client |
| LeakIX API | free key advertised for researchers | domain/IP → indexed services/leak records | researcher access may not authorize commercial prospecting; obtain terms clarity; cached records need corroboration | Docs |
4. Not a commercial free core
| Tool/source | Why excluded from the default |
|---|---|
| Shodan InternetDB | free access is non-commercial; commercial audits need licensing; cached vulns are heuristic |
| Netlas Community | free Community plan is explicitly personal-use |
| VirusTotal Public API | public API is noncommercial/academic and restricted for business/product use |
| SecurityTrails API | API/historical passive DNS/WHOIS access is paid; do not call it free |
| Google Custom Search JSON API | closed to new customers; legacy users must transition before 2027-01-01 |
| HIBP email/domain search | account lookups require subscription; domain enumeration requires proof of control; not a free “dark web scan” |
| Nmap, Masscan, Naabu, Nuclei, sqlmap, testssl.sh, scan submissions | direct active interaction; separate written-authorization workflow |
| credential-verification features | prohibited in prospect research; do not authenticate, redeem, or call detected secrets |
| private dark-web database claims | no legitimate comprehensive free/OSS source; licensed specialist plus explicit client authority required |
5. Gaps a free passive stack cannot solve
- Comprehensive historical WHOIS and passive DNS are largely commercial.
- Current TLS protocols/ciphers require a recent cached observation or an authorized direct assessment; CT cannot show them.
- Public breach sources cannot prove a password still works, MFA is absent, or the target itself was breached.
- A passive technology fingerprint cannot establish reachable exploitability.
- Shared cloud/CDN IPs cannot establish asset ownership.
6. Recommended passive core
| Need | First choice | Secondary/conditional |
|---|---|---|
| entity/domain registration | RDAP + official company/registry records | current DNS under CM1 profile |
| subdomain/certificate history | crt.sh/Cert Spotter + Subfinder passive sources | Amass passive graph |
| historical URLs/pages | Common Crawl + Wayback + Arquivo.pt | gau as adapter |
| technology hints | WappalyzerGo on permitted/cached HTML | BuiltWith free group signal |
| public documents | ordinary permitted discovery + Tika/ExifTool in isolated local worker | FOCA/manual review |
| public code/dependencies | GitHub official search + OSV-Scanner | Gitleaks/detect-secrets with redaction/no verification |
| advisory enrichment | OSV + NVD + CISA KEV + EPSS | GitHub Advisory DB offline mirror |
| cached service context | Censys Free/urlscan search under quota and terms | OTX/GreyNoise context |
7. LAKA implementation ladder
| Change state | Technical intelligence capability |
|---|---|
| Baseline | manual official-domain/registry/CT/archive snapshot with source ledger |
| Minor | scheduled passive adapters, normalization, provenance and change alerts |
| Major | asset graph, attribution confidence, SBOM/advisory enrichment and human triage |
| Structural | event-sourced evidence, terms registry, redaction/quarantine, responsible-disclosure route |
| Paradigm | client-authorized continuous exposure-management product where evidence is tied to business dependencies and interventions—not sales scare tactics |
8. Claim gates and freshness
| Claim | Minimum gate |
|---|---|
| owned asset | current RDAP/DNS/client inventory or strong target-controlled linkage; shared IP never enough |
| current exposed service | recent provider last-seen plus current linkage or authorized validation |
| vulnerable system | exact product/version/config applicability + authoritative advisory + confirmed reachable authorized asset |
| breach exposure | “identifier appeared in named corpus”; never current password/MFA/company breach claim |
| secret exposure | “possible exposed secret”; redacted fingerprint and provenance; no validation |
| negative result | “not observed in sources checked as of [UTC],” never “secure/none” |
Refresh DNS/RDAP used in outreach within 24–72 hours, technology hints within 30 days, and NVD/KEV/EPSS/OSV daily. Always show the last-seen date of cached service evidence.
9. Mandatory connector fields
connector_id: ""
mode: CM0|CM1|CM2|CM3
commercial_ok: true|false|unknown
automation_ok: true|false|unknown
redistribution_ok: true|false|unknown
authentication: none|key|oauth|verified-domain|contract
quota: ""
cache_rule: ""
deletion_rule: ""
attribution_rule: ""
license: ""
terms_url: ""
terms_reviewed_at: ""
fail_closed: trueUnknown rights fail closed.
05-technical-osint-tool-library.md · 146 lines · 16474 bytes ·
SHA-256 5114d06a6c94c047