Recently, I have seen several IPs crawling my forum all to do with a company called Colocation America. From research I had undertaken, they are a server hosting and colocation company and also did further analysis of why its crawling my forum so often... for market research and SEO/content strategy, which I found strange for a hosting company to do...

 

I should also note that this happened minutes after I promoted one of threads of my forum to its official Mastodon account about a week ago, so maybe it's coincidental, I don't know...

 

Does anybody know about the above-mentioned company? And have they seen it crawling their forum also?

Have you tried reaching out to Colocation America? They might be able to provide some insight into this activity. It's possible they’re training an AI model or gathering data for another purpose. While it's not unusual for companies in tech to collect such information, this kind of activity is more commonly done by marketing firms than hosting providers.

I haven't contacted them, but I will at some point.

While I've not heard from them since contacting them, I now see that they no longer crawl my forum constantly. Good, I suppose...

I've been having the same issue lately. By doing a WHOIS lookup, a company by the name "Hetzner" has been crawling my board for the past two days every X seconds. It's weird all the types of bots there are these days.

Hetzner is a hosting company. It's almost certainly not them themselves doing that but someone running a bot on Hetzner servers.

 

Very likely it's a bot looking for vulnerabilities (this happens more than you'd think)

 

Hetzner themselves have no real reason to run a bot like that.

Very likely it's a bot looking for vulnerabilities

Should I ban the IP address, if that is the case?

Should I ban the IP address, if that is the case?

Well, you probably should - but if they're coming around regularly and visiting, I'd be tempted to look at the requests to see what they're actually doing and whether it's worth reporting it to Hetzner as abuse of their network. They don't like people doing that kind of thing.

I'd be tempted to look at the requests to see what they're actually doing

The most I can do without server-side access is to see what pages they view.

 

I'll ban the IP address and contact the forum software owner to see if he thinks they are looking for vulnerabilities.

 

Thanks for the heads up!

Should I ban the IP address, if that is the case?

Depends on how many different IP addresses the "bots" are coming as... if you see like twenty different IP address ranges from them, then banning them one-by-one would be very difficult to do and also time-consuming as well.

Depends on how many different IP addresses the "bots" are coming as.

It has just been one IP address.

 

Here was the forum software owner's reply:

 

This IP address identifies itself as AwarioBot, which we have blocked in robots.txt but of course it doesn't listen. They claim it is harmless.

 

While I can't provide you the access logs for privacy policy reasons, I see no evidence that it's looking for vulnerabilities. This IP address is simply scanning pages. It does not have any patterns that we see when vulnerabilities are attempted to be exploited.

 

We see a lot of crawlers that are hosted on the Hetzner network and we actually have numerous blocks of Hetzner's datacenter banned for excessive crawling and/or exploitation attempts so this comes as no surprise.

 

I hope this gives you some insight as to what it is.

And coincidentally, I got a reply from Colocation America, although just an email acknowledging that my request has been sent. A bit late, considering that their "bots" have buggered off from my forum... dunno if they were looking for vulnerabilities either 🤷‍♂️

Depends on how many different IP addresses the "bots" are coming as... if you see like twenty different IP address ranges from them, then banning them one-by-one would be very difficult to do and also time-consuming as well.

You can block their entire zone if you’d like.

 

For example, if their IP address is 47.238.13.18, you can block the entire zone by using 47.238.13.*.

 

 

 

 

 

This way, any IP addresses within that range will be blocked, preventing them from accessing your forum.

 

 

Blocking a zone is much easier than blocking individual IP addresses manually.