What CensysInspectBot does
CensysInspectBot performs continuous internet-wide scanning of the public IPv4 address space and discovered IPv6 addresses, covering all 65,535 ports and detecting over 200 Layer 7 protocols. The data it collects powers the Censys Internet Map, which supports the Censys Platform and Attack Surface Management products. Cloud provider targets on AWS, Azure, and GCP are scanned at least once daily, and known services are rescanned once their last observation is older than 24 hours. Censys states the scanner only gathers information and never attempts to log in to services, access databases, or gain authenticated entry to any system.
How to control CensysInspectBot
Target CensysInspectBot in your robots.txt with the CensysInspect token. Rules apply per bot, so you can allow this one while blocking others, or the reverse.
Allow CensysInspectBot
User-agent: CensysInspect
Allow: /Block CensysInspectBot
User-agent: CensysInspect
Disallow: /How to identify CensysInspectBot
Temso identifies CensysInspectBottraffic by matching the request's User-Agent header against this pattern. User-agent strings can be spoofed, so treat a match as an indication, not proof.
CensysInspectCensysInspectBot FAQ
What is CensysInspectBot?
CensysInspectBot is the scanner Censys uses for continuous internet-wide scanning of publicly accessible services across the public IPv4 space and discovered IPv6 addresses. The data it collects builds the Censys Internet Map behind the Censys Platform and Attack Surface Management products.
How do I block CensysInspectBot?
You can add a robots.txt group with 'User-agent: CensysInspect' and 'Disallow: /'. However, the Censys documentation describes IP and port scanning of services rather than link-following web crawling, and it references a separate opt-out process for data collection.
About Operational Bots
Monitoring, security scanning, link previews, webhooks, and feed fetchers: the utility traffic that keeps integrations and shared links working.
Most of this traffic is service-critical or user-triggered. Webhooks and previews don't follow robots.txt, and blocking them tends to break things you rely on.
