Is my forum being scraped?

Do not post support requests, bug reports or feature requests. Discuss phpBB here. Non-phpBB related discussion goes in General Discussion!
Scam Warning
User avatar
Tastenplayer
Registered User
Posts: 410
Joined: Thu Jul 03, 2014 9:20 pm
Location: Switzerland
Name: Jutta Koliofotis
Contact:

Re: Is my forum being scraped?

Post by Tastenplayer » Sun Nov 17, 2019 5:04 pm

We must block not only China, but above all Singapore. With my server configuration this is extremely difficult because I can't block countries. Blocking the IP's of China has now worked to some extent. Unfortunately not yet to block them from Singapore (200-800 visitors a day). Now from Huawei Clouds Singapore, what bots are for sure. But there's just too many coming at once.

The problem is that people from China and Singapore sometimes also want a style. That's why I don't like to block whole countries. However, the visitors from China have already shot me the forum.
My phpBB Style Board & More3.3.0-b2 StyleTBChristmas calendar (Changing style background & song in announcement)
Be the best version of yourself rather than a bad copy of someone else!
Excuse me for my English, but I learned the language by speaking to people and not at school.

User avatar
John connor
Registered User
Posts: 2344
Joined: Fri Nov 14, 2014 5:14 pm
Location: U S Of A
Name: Aaron
Contact:

Re: Is my forum being scraped?

Post by John connor » Mon Nov 18, 2019 10:50 am

thecoalman wrote:
Sun Sep 22, 2019 8:16 am
As a side note Cloudflare adds a header with country code with the http request to the server so you could also do stuff with it server side.
Here's one better. https://community.cloudflare.com/t/stop ... ting/91203

I don't really need to, but I use the Workers methoud and the country header method in my htaccess file.

User avatar
John connor
Registered User
Posts: 2344
Joined: Fri Nov 14, 2014 5:14 pm
Location: U S Of A
Name: Aaron
Contact:

Re: Is my forum being scraped?

Post by John connor » Mon Nov 18, 2019 10:54 am

skybound wrote:
Sun Nov 17, 2019 3:29 pm
Resolved my last issue by blocking China outright.

Now getting attacked by 17.58.100.* and a few other in that 17.58. range. Those resolve to Apple Inc. Around about 400 guests presently. Any thoughts on this?
Yes, I have some.

I got lots of Apple hits as well and was told that Apple may be in the process of creating their own search engine, but I also heard they may or may not be reading and adhering to a bots text file. Since my bots file makes sure only the good buts follow it, and I only allow the big three search engines, I wasn't taking the risk of having Apple index my folders that they shouldn't. So I just blocked all of Apple's ASNs in CloudFlare. Problem solved.

Since I just allow Google, Bing and Yahoo, I still get search engine hits from DuckDuckGo, etc because they use data from Bing or what ever. And that's good enough as far as I'm concerned.

User avatar
thecoalman
Community Team Member
Community Team Member
Posts: 3405
Joined: Wed Dec 22, 2004 3:52 am
Location: Pennsylvania, U.S.A.
Contact:

Re: Is my forum being scraped?

Post by thecoalman » Mon Nov 18, 2019 4:52 pm

John connor wrote:
Mon Nov 18, 2019 10:50 am
thecoalman wrote:
Sun Sep 22, 2019 8:16 am
As a side note Cloudflare adds a header with country code with the http request to the server so you could also do stuff with it server side.
Here's one better. https://community.cloudflare.com/t/stop ... ting/91203

I don't really need to, but I use the Workers methoud and the country header method in my htaccess file.
That may be workaround for shared hosting but if you have the ability to modify the firewall you just block all IP's but Cloudflare IP's. No sense messing around when you can use a sledgehammer.

That is besides the point of my post, you have reliable information about the country they are from. A phpBB extension could for example be devised to flag registrants as potential spammers when the country code is common spammer nation. As another example you could list the country code next to the IP for any moderation tools.
“Results! Why, man, I have gotten a lot of results! I have found several thousand things that won’t work.”

Attributed - Thomas Edison

User avatar
John connor
Registered User
Posts: 2344
Joined: Fri Nov 14, 2014 5:14 pm
Location: U S Of A
Name: Aaron
Contact:

Re: Is my forum being scraped?

Post by John connor » Mon Nov 18, 2019 9:11 pm

Yeah, I know that if one were to run a VPS or bare metal server they should use IPtables or what ever and block all IPs except CloudFlare's then stay abreast of any CF IP changes. Which thankfully don't change all that often. I have wrote a topic on how to secure your website if you use CloudFlare on my own forum.

Since I use a shared account and probably a lot of other people, that option with Workers is nice to use. Or use this in the htaccess file:

Code: Select all

RewriteCond %{HTTP:CF-IPCountry} ^$
RewriteRule ^ - [F,L]
What that does is 403 all none CloudFlare access attempts since if the IP didn't have the proper CF country header from CF they would get blocked. Though that solution can be forged by a good crafty hacker. That's why the other solution using Workers setting a custom header is more full proof. Since the headers are only seen server to server and can't be forged.

Post Reply

Return to “phpBB Discussion”