What Omgilibot does
Omgilibot is a web crawler developed by Webz.io that formerly powered the Omgili search engine and now indexes public web content for Webz.io data feeds. It focuses on news sites, blogs, reviews, ecommerce sites, message boards, and online discussions. Requests appear in logs with the user agent "omgili/0.5 +https://omgili.com". Webz.io states the bot is designed to minimize the resources it requires and that it complies with blocking requests made via robots.txt or by contacting the company.
How to control Omgilibot
Target Omgilibot in your robots.txt with the omgilibot token. Rules apply per bot, so you can allow this one while blocking others, or the reverse.
Allow Omgilibot
User-agent: omgilibot
Allow: /Block Omgilibot
User-agent: omgilibot
Disallow: /How to identify Omgilibot
Temso identifies Omgilibottraffic by matching the request's User-Agent header against this pattern. User-agent strings can be spoofed, so treat a match as an indication, not proof.
omgilibotOmgilibot FAQ
What is Omgilibot?
Omgilibot is a Webz.io crawler that indexes public content such as news, blogs, reviews, ecommerce pages, and forum discussions for Webz.io data feeds. In server logs its requests carry the user agent "omgili/0.5 +https://omgili.com".
How do I block Omgilibot?
Add a robots.txt group with "User-agent: omgilibot" followed by "Disallow: /". Webz.io states that it complies with blocking requests and can also be contacted directly to stop crawling.
About AI Training Crawlers
Crawlers that collect content to train foundation models. Data gathered today shapes what future model versions know about your brand.
Blocking these doesn't affect live citations, but it limits what future models learn about you from your own site, leaving third-party sources to fill the gap.
