What VelenPublicWebCrawler does
VelenPublicWebCrawler is a crawler written in Go for Hunter that analyzes millions of public internet pages every month to build business datasets and machine learning models. It processes only publicly accessible content and does not crawl pages behind a login or other authentication. The operator states it follows robots.txt and meta instructions, detects new blocks within a few hours, and keeps load low by fetching at most one page at a time and no more than one page every 2 seconds. Bug reports and feedback go to engineering@velen.io.
How to control VelenPublicWebCrawler
Target VelenPublicWebCrawler in your robots.txt with the VelenPublicWebCrawler token. Rules apply per bot, so you can allow this one while blocking others, or the reverse.
Allow VelenPublicWebCrawler
User-agent: VelenPublicWebCrawler
Allow: /Block VelenPublicWebCrawler
User-agent: VelenPublicWebCrawler
Disallow: /The operator states that robots.txt blocks targeting its user agent are picked up within a few hours and that the crawler requests at most one page at a time, no more than one page every 2 seconds.
How to identify VelenPublicWebCrawler
Temso identifies VelenPublicWebCrawlertraffic by matching the request's User-Agent header against this pattern. User-agent strings can be spoofed, so treat a match as an indication, not proof.
VelenPublicWebCrawlerVelenPublicWebCrawler FAQ
What is VelenPublicWebCrawler?
VelenPublicWebCrawler is a Go crawler built for Hunter that analyzes millions of publicly accessible web pages each month to build business datasets and machine learning models. It does not crawl content behind logins or other authentication.
How do I block VelenPublicWebCrawler?
Add a robots.txt group with "User-agent: VelenPublicWebCrawler" followed by "Disallow: /". The operator says the crawler follows robots.txt and meta instructions and detects new blocks within a few hours; feedback can also be sent to engineering@velen.io.
About AI Training Crawlers
Crawlers that collect content to train foundation models. Data gathered today shapes what future model versions know about your brand.
Blocking these doesn't affect live citations, but it limits what future models learn about you from your own site, leaving third-party sources to fill the gap.
