Information about the "bot": http://en.wikipedia.org/wiki/WgetMarcus Wendel wrote:A bad bot added and banned:
Bot name: Wget [SpamBot]
Agent match: Wget
I disallowed this crawler in robots.txt because it scrapes and then presents the crawled forum content nearly in its entirety. The omgili FAQ offers this explanation:3Di wrote:This one I banned its IP, I recall (lurking at my database and my online guests flags) that creates a lot of sessions, like a Spider, it is not a BOT, it's a crawler or spider IMO.Marcus Wendel wrote:Bot name: Omgili [Bot]
Agent match: omgilibot
Information on the bot: http://www.omgili.com/Crawler.html
omgili FAQ wrote: What is the preview feature?
The previews are a neat feature that gives the users the ability to view most of the discussion rather then the two snippets the search engine returns on the results page. The content may also be available in case the forum is down or the post was deleted due to database restrains. The original post link is marked in bold on the top of the page.
How do I tell Omgili to disable previews for my forum?
If you decide you don't want to enable the preview feature for your board, add the noarchive meta tag to your board's topics:
<meta name="omgilibot" content="noarchive">
<meta name="googlebot" content="noarchive">