Forum shows high traffic but Google Analytics does not?

Discussion of non-phpBB related topics with other phpBB.com users.
Forum rules
General Discussion is a bonus forum for discussion of non-phpBB related topics with other phpBB.com users. All site rules apply.
Post Reply
thetgc
Registered User
Posts: 38
Joined: Tue Apr 25, 2017 3:51 pm

Forum shows high traffic but Google Analytics does not?

Post by thetgc »

Got a bit of an odd situation which has cropped up recently. My forum has a genuine 30-40k visitors per month but over the last few weeks this figure has ballooned enormously. Where once I would see maybe 50-100 users online at any one time I am now seeing thousands (not registered users by the way)

As a result the site responsiveness has slowed to a crawl.

Google Analytics however shows just a handful of people online in real time. Which is in keeping with the sites normal traffic figures.

Has anyone else come across this before? Could it be some sort of script/bot thing going on?

KYPREO
Jr. Extension Validator
Posts: 392
Joined: Fri Feb 02, 2018 9:56 am
Contact:

Re: Forum shows high traffic but Google Analytics does not?

Post by KYPREO »

Google Analytics only shows results of users who accept the tracking cookie/code setup for your site. It also excludes web crawlers and bot traffic.

The guests shown in forum stats is not an accurate indicator of actual users online because:
(a) by default, it displays results for active users over a 5 minute period, not instaneously.
(b) it includes bots and crawlers. If you actually click on "who is online", then click "display guests" in the following list, you will see the IP address and browser identifier of each guest connection. There you will likely find that most of the guests identify as a bot of some kind. If you set up a proper robots.txt and look at other kinds of filtering (eg Cloudflare or server firewall configuration), unwanted connections will disappear and will save you valuable server resources.

User avatar
WelshPaul
Registered User
Posts: 382
Joined: Tue Aug 19, 2014 2:09 pm

Re: Forum shows high traffic but Google Analytics does not?

Post by WelshPaul »

thetgc wrote:
Mon Sep 30, 2019 3:27 pm
Has anyone else come across this before? Could it be some sort of script/bot thing going on?
Very likely this but it could be a low scale DDOS? Do you have access to any server side logs?

thetgc
Registered User
Posts: 38
Joined: Tue Apr 25, 2017 3:51 pm

Re: Forum shows high traffic but Google Analytics does not?

Post by thetgc »

Thanks for the replies. Checked the logs and found it was Chinese scraper bots.

Used this in the .htaccess as a starter and it has completely killed all that unwanted traffic...for now anyway and it is easy to update.

Code: Select all

Options +FollowSymLinks
RewriteEngine On
RewriteBase /
RewriteCond %{HTTP_USER_AGENT} Mb2345Browser|LieBaoFast|zh-CN|MicroMessenger|zh_CN|Kinza [NC]
RewriteRule ^ - [F,L]
Cloudfare is an option but another $20 a month and as it is not a revenue generating forum and costs me quite a bit each month already I would rather avoid yet more expense if possible.

User avatar
WelshPaul
Registered User
Posts: 382
Joined: Tue Aug 19, 2014 2:09 pm

Re: Forum shows high traffic but Google Analytics does not?

Post by WelshPaul »

thetgc wrote:
Tue Oct 01, 2019 9:02 am
Thanks for the replies. Checked the logs and found it was Chinese scraper bots.

Used this in the .htaccess as a starter and it has completely killed all that unwanted traffic...for now anyway and it is easy to update.

Code: Select all

Options +FollowSymLinks
RewriteEngine On
RewriteBase /
RewriteCond %{HTTP_USER_AGENT} Mb2345Browser|LieBaoFast|zh-CN|MicroMessenger|zh_CN|Kinza [NC]
RewriteRule ^ - [F,L]
Cloudfare is an option but another $20 a month and as it is not a revenue generating forum and costs me quite a bit each month already I would rather avoid yet more expense if possible.
You really want to stop it before it hits your server.

thetgc
Registered User
Posts: 38
Joined: Tue Apr 25, 2017 3:51 pm

Re: Forum shows high traffic but Google Analytics does not?

Post by thetgc »

Yes but the only way to do that is with Cloudfare?

KYPREO
Jr. Extension Validator
Posts: 392
Joined: Fri Feb 02, 2018 9:56 am
Contact:

Re: Forum shows high traffic but Google Analytics does not?

Post by KYPREO »

Basic Cloudflare with DDos protection, CDN and SSL is free.

thetgc
Registered User
Posts: 38
Joined: Tue Apr 25, 2017 3:51 pm

Re: Forum shows high traffic but Google Analytics does not?

Post by thetgc »

I was told by Amazon Web Services that it was an extra $20 per month :(

WWu777
Registered User
Posts: 742
Joined: Tue Aug 14, 2007 12:40 pm
Contact:

Re: Forum shows high traffic but Google Analytics does not?

Post by WWu777 »

KYPREO wrote:
Tue Oct 01, 2019 2:40 am
Google Analytics only shows results of users who accept the tracking cookie/code setup for your site. It also excludes web crawlers and bot traffic.

The guests shown in forum stats is not an accurate indicator of actual users online because:
(a) by default, it displays results for active users over a 5 minute period, not instaneously.
(b) it includes bots and crawlers. If you actually click on "who is online", then click "display guests" in the following list, you will see the IP address and browser identifier of each guest connection. There you will likely find that most of the guests identify as a bot of some kind. If you set up a proper robots.txt and look at other kinds of filtering (eg Cloudflare or server firewall configuration), unwanted connections will disappear and will save you valuable server resources.
How can you tell if a guest is a bot or a real user? Is there a way?

User avatar
warmweer
Jr. Extension Validator
Posts: 4466
Joined: Fri Jul 04, 2003 6:34 am
Location: Van Allen Bel ... gium

Re: Forum shows high traffic but Google Analytics does not?

Post by warmweer »

WWu777 wrote:
Wed May 06, 2020 11:43 pm
How can you tell if a guest is a bot or a real user? Is there a way?
If you're lucky the bot will use a genuine user agent, but malicious bots won't do that (some can be flagged/identified because because the engine version is outdated).
So basically you'll have to depend on behaviour analysis.
If you get many guests simultaneously from the same IP range: the probability is those being bots is high.
How long do they stay, do they try to post, how fast do they switch pages, how regular are the visits, etc...

WWu777
Registered User
Posts: 742
Joined: Tue Aug 14, 2007 12:40 pm
Contact:

Re: Forum shows high traffic but Google Analytics does not?

Post by WWu777 »

warmweer wrote:
Thu May 07, 2020 12:00 pm
WWu777 wrote:
Wed May 06, 2020 11:43 pm
How can you tell if a guest is a bot or a real user? Is there a way?
If you're lucky the bot will use a genuine user agent, but malicious bots won't do that (some can be flagged/identified because because the engine version is outdated).
So basically you'll have to depend on behaviour analysis.
If you get many guests simultaneously from the same IP range: the probability is those being bots is high.
How long do they stay, do they try to post, how fast do they switch pages, how regular are the visits, etc...
Ok I clicked on "Who is Online" link at the bottom of my forum. And then "Display Guests". I do see a lot of bots with similar IP ranges. Am I allowed to post them here in this board? If those are bots, what is their purpose? Did someone target my site for a DDOS attack? If not, are these bots trying to scrape information? I don't understand their purpose. When I click on WhoIs for their IP, it appears to come from a Chinese industrial park with a domain for Huawei from Singapore. Singapore is not known for spam or hackers like China is right? What could their purpose be? Am I allowed to post their IP's here to show you all?

User avatar
KevC
Support Team Member
Support Team Member
Posts: 69901
Joined: Fri Jun 04, 2004 10:44 am
Location: Oxford, UK
Contact:

Re: Forum shows high traffic but Google Analytics does not?

Post by KevC »

It's not worth losing sleep over All sorts of bots and people visit sites all day long. Search bots will be looking for 'key words' so when people do a web search they have a greater chance of finding your site if it's the same subject matter. Hackers are relatively unlikely. They don't gain much if anything by attacking a humble forum and the security of the server is largely down to how good your host is so you can't do that much about it.

Sit back, relax, enjoy your site for whatever it is and let the hosts worry about traffic. I don't think I've looked at visit logs and who is online on my sites for 10 years!
-:|:- Support Request Template -:|:-
Image
Cheap UK Hosting
"In the land of the blind the little green bloke with no pupils is king - init!"

WWu777
Registered User
Posts: 742
Joined: Tue Aug 14, 2007 12:40 pm
Contact:

Re: Forum shows high traffic but Google Analytics does not?

Post by WWu777 »

KevC wrote:
Thu May 07, 2020 12:58 pm
It's not worth losing sleep over All sorts of bots and people visit sites all day long. Search bots will be looking for 'key words' so when people do a web search they have a greater chance of finding your site if it's the same subject matter. Hackers are relatively unlikely. They don't gain much if anything by attacking a humble forum and the security of the server is largely down to how good your host is so you can't do that much about it.

Sit back, relax, enjoy your site for whatever it is and let the hosts worry about traffic. I don't think I've looked at visit logs and who is online on my sites for 10 years!
I've had DDOS attacks on my forum before. There a lot of hateful people out there. One time it even crashed my database for a whole month. I didn't have a backup. It took over a month for my webhost to retrieve a backup or restored copy. That's why I have to be wary of this, because it's happened before.

Right now I have about 400 to 500 bots on my forum at all hours of the day. Mostly from Singapore. Is that indicative of a DDOS attack? Is 400 bots all day and all night normal or unusual? How many bots can the phpbb board handle before it starts to crash or slow down? I've increased the RAM in my VPS server to 8GB so it should be able to hold a lot.

User avatar
KevC
Support Team Member
Support Team Member
Posts: 69901
Joined: Fri Jun 04, 2004 10:44 am
Location: Oxford, UK
Contact:

Re: Forum shows high traffic but Google Analytics does not?

Post by KevC »

WWu777 wrote:
Thu May 07, 2020 1:48 pm
I've had DDOS attacks on my forum before.
It's more likely that was on your host, not just your forum. There isn't a lot you can do about those. They pick a random server IP and flood it.

I have no idea how many bots are on my site every day. The site works, that's fine. If you don't want those bots there and you think they're highly unlikely as legitimate guests, block the IPs (or the range) at the server side.
-:|:- Support Request Template -:|:-
Image
Cheap UK Hosting
"In the land of the blind the little green bloke with no pupils is king - init!"

User avatar
P_I
Registered User
Posts: 1150
Joined: Tue Mar 01, 2011 8:35 pm
Location: Staying home - Calgary
Contact:

Re: Forum shows high traffic but Google Analytics does not?

Post by P_I »

KevC wrote:
Thu May 07, 2020 12:58 pm
It's not worth losing sleep over All sorts of bots and people visit sites all day long. Search bots will be looking for 'key words' so when people do a web search they have a greater chance of finding your site if it's the same subject matter. Hackers are relatively unlikely. They don't gain much if anything by attacking a humble forum and the security of the server is largely down to how good your host is so you can't do that much about it.

Sit back, relax, enjoy your site for whatever it is and let the hosts worry about traffic. I don't think I've looked at visit logs and who is online on my sites for 10 years!
Generally I would concur with this. Up until the point in time when something happens that negatively changes the board's performance and impacts my membership.

On my boards something changed last November. The count of the number of guest on the board exploded from the normal range which is up to low hundreds, into the thousands. Database connection errors like
Error establishing a database connection

SQL ERROR [ mysqli ]
User phpbb already has more than 'max_user_connections' active connections [1203]
That's when it does become time to peek under the covers to determine the root cause and take steps to mitigate the badly behaved bots/crawlers.

Depending on your hosting arrangement, there are various solutions available. In my case my boards run on shared hosting plans and courtesy of another member here, I found and implemented https://www.johnlarge.co.uk/blocking-ag ... pers-bots/ and my boards returned to 'normal'.

Post Reply

Return to “General Discussion”