Intermittent slow loading of forum pages

Get help with installation and running phpBB 3.3.x here. Please do not post bug reports, feature requests, or extension related questions here.
Mrs Doyle
Registered User
Posts: 119
Joined: Mon Nov 11, 2019 1:04 pm

Re: Intermittent slow loading of forum pages

Post by Mrs Doyle »

I've got Google, Bing, Yahoo and Alexa enabled as bots.
Mrs Doyle
Registered User
Posts: 119
Joined: Mon Nov 11, 2019 1:04 pm

Re: Intermittent slow loading of forum pages

Post by Mrs Doyle »

Experienced a slow loading page at 13:08 today, so I guess what I've done so far didnt work. :cry:
User avatar
P_I
Community Team Member
Community Team Member
Posts: 2344
Joined: Tue Mar 01, 2011 8:35 pm
Location: Western Canada 🇨🇦
Contact:

Re: Intermittent slow loading of forum pages

Post by P_I »

Mrs Doyle wrote: Thu May 13, 2021 1:35 pm I've got Google, Bing, Yahoo and Alexa enabled as bots.
There are good (desirable) bots and bad (undesirable) bots.

For the good bots (search robots) you can use the functionality that phpBB provides for Managing Search Robots. These search robots provide information to the search engines that can provide user traffic to your forum.

The linked blog post, Blocking aggressive Chinese crawlers/scrapers/bots - John Large .co.uk, provides a solution for bad (undesirable) bots that don't provide any value to your forum and only consumer resources.
Mrs Doyle wrote: Thu May 13, 2021 1:38 pm Experienced a slow loading page at 13:08 today, so I guess what I've done so far didnt work. :cry:
On a shared hosting plan that will happen occasionally. It is the trade-off between low cost hosting and the potential to be impacted by other accounts on the same server that occasionally overload the server.

If it happens regularly or you can determine a specific pattern (time of day?) then that's worth reporting back to your hosting provider to have them investigate. So far based on what you've reported here, the bad bots mentioned in the John Large article might indeed have been a cause of excessive resource consumption on your board.
Normal people… believe that if it ain’t broke, don’t fix it. Engineers believe that if it ain’t broke, it doesn’t have enough features yet. – Scott Adams
Mrs Doyle
Registered User
Posts: 119
Joined: Mon Nov 11, 2019 1:04 pm

Re: Intermittent slow loading of forum pages

Post by Mrs Doyle »

Thanks, I understand the different types of bots.

Just looking at my robots.txt Does this look right?

Code: Select all

User-agent: *
Allow: /
sitemap: http://lofcforum.com/sitemapindex.xml
sitemap: http://lofcforum.com/?sitemapindex.xml
sitemap: http://lofcforum.com/sitemap_470.xml
sitemap: http://lofcforum.com/?sitemap_8494.xml
sitemap: http://lofcforum.com/?sitemap_1742.xml
sitemap: http://lofcforum.com/?sitemap_9905.xml
sitemap: http://lofcforum.com/sitemap_7320.xml
sitemap: http://lofcforum.com/sitemap_8545.xml
sitemap: http://lofcforum.com/sitemap_739.xml
sitemap: http://lofcforum.com/?sitemap_9050.xml
sitemap: http://lofcforum.com/?sitemap_590.xml
sitemap: http://lofcforum.com/sitemap_9498.xml
sitemap: http://lofcforum.com/sitemap_424.xml
sitemap: http://lofcforum.com/sitemap_7559.xml
sitemap: http://lofcforum.com/sitemap_9703.xml
sitemap: http://lofcforum.com/?sitemap_4186.xml
sitemap: http://lofcforum.com/sitemap_3545.xml
sitemap: http://lofcforum.com/?sitemap_775.xml
sitemap: http://lofcforum.com/?sitemap_5244.xml
sitemap: http://lofcforum.com/?sitemap_5913.xml
sitemap: http://lofcforum.com/sitemap_9002.xml
sitemap: http://lofcforum.com/sitemap_6002.xml
sitemap: http://lofcforum.com/?sitemap_5051.xml
sitemap: http://lofcforum.com/sitemap_8640.xml
sitemap: http://lofcforum.com/sitemap_6420.xml
sitemap: http://lofcforum.com/sitemap_3924.xml
sitemap: http://lofcforum.com/sitemap_9572.xml
sitemap: http://lofcforum.com/?sitemap_1761.xml
sitemap: http://lofcforum.com/sitemap_2058.xml
sitemap: http://lofcforum.com/?sitemap_6331.xml
sitemap: http://lofcforum.com/?sitemap_9231.xml
sitemap: http://lofcforum.com/?sitemap_3200.xml
sitemap: http://lofcforum.com/sitemap_9436.xml
sitemap: http://lofcforum.com/sitemap_6363.xml
sitemap: http://lofcforum.com/?sitemap_4025.xml
sitemap: http://lofcforum.com/sitemap_9376.xml
sitemap: http://lofcforum.com/sitemap_20.xml
sitemap: http://lofcforum.com/?sitemap_8753.xml
sitemap: http://lofcforum.com/?sitemap_9483.xml
sitemap: http://lofcforum.com/sitemap_3693.xml
sitemap: http://lofcforum.com/sitemap_6392.xml
sitemap: http://lofcforum.com/sitemap_3366.xml
sitemap: http://lofcforum.com/?sitemap_4819.xml
sitemap: http://lofcforum.com/?sitemap_5280.xml
sitemap: http://lofcforum.com/?sitemap_2593.xml
sitemap: http://lofcforum.com/?sitemap_425.xml
sitemap: http://lofcforum.com/?sitemap_2562.xml
sitemap: http://lofcforum.com/?sitemap_2155.xml
sitemap: http://lofcforum.com/sitemap_9537.xml
sitemap: http://lofcforum.com/sitemap_1833.xml
sitemap: http://lofcforum.com/sitemap_8821.xml
sitemap: http://lofcforum.com/sitemap_4998.xml
sitemap: http://lofcforum.com/?sitemap_8525.xml
sitemap: http://lofcforum.com/?sitemap_4097.xml
sitemap: http://lofcforum.com/sitemap_7396.xml
sitemap: http://lofcforum.com/?sitemap_4212.xml
sitemap: http://lofcforum.com/?sitemap_3196.xml
sitemap: http://lofcforum.com/?sitemap_390.xml
sitemap: http://lofcforum.com/sitemap_9390.xml
sitemap: http://lofcforum.com/sitemap_4152.xml
sitemap: http://lofcforum.com/sitemap_4719.xml
User avatar
janus_zonstraal
Registered User
Posts: 6414
Joined: Sat Aug 30, 2014 1:30 pm

Re: Intermittent slow loading of forum pages

Post by janus_zonstraal »

Mrs Doyle wrote: Thu May 13, 2021 1:38 pm Experienced a slow loading page at 13:08 today, so I guess what I've done so far didnt work. :cry:
And how many guest/members where online that time?

And about the robots text normally there isn't a robots text in phpbb.
So how did yours come there and why so many sitemaps?
Sorry! My English is bat ;) !!!
Mrs Doyle
Registered User
Posts: 119
Joined: Mon Nov 11, 2019 1:04 pm

Re: Intermittent slow loading of forum pages

Post by Mrs Doyle »

janus_zonstraal wrote: Thu May 13, 2021 3:04 pm
Mrs Doyle wrote: Thu May 13, 2021 1:38 pm Experienced a slow loading page at 13:08 today, so I guess what I've done so far didnt work. :cry:
And how many guest/members where online that time?

And about the robots text normally there isn't a robots text in phpbb.
So how did yours come there and why so many sitemaps?
There often seems to be 30 odd guests, mostly mobile devices or nothing that looks suspicious to me. Its difficult to analyse because the page seems to refresh itself quite frequently.

I've no idea why robots.txt is there or what a sitemap is, perhaps I'll delete it.
User avatar
thecoalman
Community Team Member
Community Team Member
Posts: 5850
Joined: Wed Dec 22, 2004 3:52 am
Location: Pennsylvania, U.S.A.
Contact:

Re: Intermittent slow loading of forum pages

Post by thecoalman »

If you have not created that robots.txt file or have not installed some kind of sitemap extension that created it then it may be indication your server is hacked. robots.txt files don't just magically appear, especially ones with sitemap links.

The way a sitemap normally works is a bot like Google will download robots.txt and those listing are to indicate where the sitemap files can be found. A sitemap is used to give the bots links so it can understand how your site is structured and what links it may want to view.

The exploit is that sitemap was generated by whoever hacked your server (an assumption by me at this point). The links in it go to a crypto mining script or something else they want to execute intermittently, it could just be spam page. Google will download the sitemap and follow the links in it executing the file, effectively the sitemap is cron job to get Google to request the page. This would also explain the intermittent slowness of pages.

You are going to want to look for suspicious files especially ones with recent modified dates. Also look in the root of the domain for one with a random string that is about 20 or 30 characters long ending with .html, if you have this file and have not created Google Webmaster account that was created by hacker to verify a Google Webmaster account which is something you will need to address.

Simply deleting files is not going to fix the issue unless the root cause is found. If you determine you have been hacked it's unlikely they gained access through phpBB but you can file a security report here.

https://tracker.phpbb.com/projects/INCIDENT
“Results! Why, man, I have gotten a lot of results! I have found several thousand things that won’t work.”

Attributed - Thomas Edison
Mrs Doyle
Registered User
Posts: 119
Joined: Mon Nov 11, 2019 1:04 pm

Re: Intermittent slow loading of forum pages

Post by Mrs Doyle »

We were hacked a couple of years ago via something to do with Wordpress.

It was dealt with at the time, but may not have been 100% cleaned up. I'm not seeing any other indications that we're hacked, it was quite obvious before.

I've checked the modified date on the robots.txt (29 sept 2020), this is a lot later than I remember our last hack being. Looking at other files with the same date, its all WP stuff. Its also the date I think I tried an upgrade.

I cant see any of the referenced sitemap files on the server.
Last edited by Mrs Doyle on Fri May 14, 2021 9:44 am, edited 1 time in total.
Mrs Doyle
Registered User
Posts: 119
Joined: Mon Nov 11, 2019 1:04 pm

Re: Intermittent slow loading of forum pages

Post by Mrs Doyle »

Some slow loading last night and this morning. Bingbot always seems to be hanging around at the time, so I may try blocking all bots with a robots file and see if that stops it. What do you think?

There seems to be around 30 guests at all hours which I think is a bit odd. igh proportion of Apple devices. Would people leaving a tab open but not used cause a guest to be listed?
Mrs Doyle
Registered User
Posts: 119
Joined: Mon Nov 11, 2019 1:04 pm

Re: Intermittent slow loading of forum pages

Post by Mrs Doyle »

Robots.txt hasnt deterred Bingbot :x
User avatar
janus_zonstraal
Registered User
Posts: 6414
Joined: Sat Aug 30, 2014 1:30 pm

Re: Intermittent slow loading of forum pages

Post by janus_zonstraal »

Sorry! My English is bat ;) !!!
Mrs Doyle
Registered User
Posts: 119
Joined: Mon Nov 11, 2019 1:04 pm

Re: Intermittent slow loading of forum pages

Post by Mrs Doyle »

Set a crawl delay, see where that gets me.

Still getting 40 odd guests.
User avatar
janus_zonstraal
Registered User
Posts: 6414
Joined: Sat Aug 30, 2014 1:30 pm

Re: Intermittent slow loading of forum pages

Post by janus_zonstraal »

Still getting 40 odd guests.
Can you place a screen print of this guests here?
Sorry! My English is bat ;) !!!
User avatar
P_I
Community Team Member
Community Team Member
Posts: 2344
Joined: Tue Mar 01, 2011 8:35 pm
Location: Western Canada 🇨🇦
Contact:

Re: Intermittent slow loading of forum pages

Post by P_I »

Mrs Doyle wrote: Tue May 18, 2021 8:32 am Set a crawl delay, see where that gets me.

Still getting 40 odd guests.
Guests (lurkers) are normal and expected. The goal should be to gain some understanding of them and determine if any of them are non-human (bots) that negatively impact your forum.

Using the 'Who is online' information, phpBB provides information about their IP address and user-agent string, which often can be used to help understand if they are human or bots. If you see many with the same user-agent string that doesn't match common browser user-agent strings that generally is what I'd investigate further.
Normal people… believe that if it ain’t broke, don’t fix it. Engineers believe that if it ain’t broke, it doesn’t have enough features yet. – Scott Adams
User avatar
thecoalman
Community Team Member
Community Team Member
Posts: 5850
Joined: Wed Dec 22, 2004 3:52 am
Location: Pennsylvania, U.S.A.
Contact:

Re: Intermittent slow loading of forum pages

Post by thecoalman »

You are using

Code: Select all

User-agent: *
Disallow: /
This will stop all legitimate bot traffic(Google,Bing etc) and your site will drop from from the search results.
“Results! Why, man, I have gotten a lot of results! I have found several thousand things that won’t work.”

Attributed - Thomas Edison
Post Reply

Return to “[3.3.x] Support Forum”