Best way to handle long processes to avoid timeouts

Discussion forum for Extension Writers regarding Extension Development.
Post Reply
User avatar
MarkDHamill
Registered User
Posts: 4342
Joined: Fri Aug 02, 2002 12:36 am
Location: Florence, MA USA
Contact:

Best way to handle long processes to avoid timeouts

Post by MarkDHamill »

I am working on a spam remover extension that uses the Akismet service.

Since boards can vary in size from tiny to enormous, when searching for spam I need an interface similar to the one used on phpBB installations or conversions that gives progress dialog boxes, refreshes itself regularly and has logic to reduce the likelihood of PHP timeouts by processing reasonably sized chunks of content. I do this toward the top of the program, but based on my experience it's not likely to be enough:

Code: Select all

		set_time_limit(0);
I'll be passing a lot of content to Akismet using its API, probably one post or private message at a time as I don't see a way to do this in bulk. I will have an interface that allows the administrator to refine the content sent, such as to a narrow range of dates. Often a board is spam free then can get a huge bulk of spam in a short period of time too voluminous to be removed by visual inspection of the content. My extension will have a way to flag and review potentially spammy posts and private messages and individually mark items as ham, once Akismet delivers its assessment.

I assume something AJAX-less is preferred. I've looked through the installer and while I'm sure the logic used is in there somewhere it is deeply abstracted away. Maybe there is a better way of doing this than the way the installer does it.

A best practices/built-in ways of doing this within phpBB would be appreciated.
Need phpBB services or a phpBB consultant? I offer most phpBB services. Getting lost managing phpBB? Buy my book, Mastering phpBB Administration. eBook and paper versions available.
User avatar
david63
Registered User
Posts: 18821
Joined: Thu Dec 19, 2002 8:08 am
Location: Lancashire, UK
Contact:

Re: Best way to handle long processes to avoid timeouts

Post by david63 »

Not sure if it would work but have you considered running it as a cron job? That way it would run in background and you would possibly be able to split it into manageable chunks.
David
Remember: You only know what you know and - you don't know what you don't know!
My CDB Contributions | How to install an extension
I will not be accepting translations for any of my extensions in Github - please post any translations in the appropriate topic.
No support requests via PM or email as they will be ignored
User avatar
MarkDHamill
Registered User
Posts: 4342
Joined: Fri Aug 02, 2002 12:36 am
Location: Florence, MA USA
Contact:

Re: Best way to handle long processes to avoid timeouts

Post by MarkDHamill »

It doesn't provide a great user experience to run as a cron job. But maybe it could be kicked off and the ACP interface would let you know when done, or provide some sort of current statistics on the page of the current status, disabling other modules until it's completed.
Need phpBB services or a phpBB consultant? I offer most phpBB services. Getting lost managing phpBB? Buy my book, Mastering phpBB Administration. eBook and paper versions available.
User avatar
RMcGirr83
Former Team Member
Posts: 21821
Joined: Wed Jun 22, 2005 4:33 pm
Location: Your display
Name: Rich McGirr

Re: Best way to handle long processes to avoid timeouts

Post by RMcGirr83 »

I somewhat remember the old support tool kit used to chunk re-parsing of bbcode. Maybe that will give you some ideas.
Former Modifications/Extensions Team Member | My extensions | github | All requests for support via PM will be ignored
Appreciate the extensions/mods/support then buy me a beerImage
User avatar
MarkDHamill
Registered User
Posts: 4342
Joined: Fri Aug 02, 2002 12:36 am
Location: Florence, MA USA
Contact:

Re: Best way to handle long processes to avoid timeouts

Post by MarkDHamill »

So far the best idea I can think of is set a configuration variable for the number of posts or private messages to check in a "batch" before a screen refresh provides an update on progress. I have it set to 50, which means 50 "comment checks" (as Akismet calls it, as mostly they check Wordpress comments) occur before a screen refresh. The value could be changed in the ACP on a settings page. In the event of a PHP timeout, resource error or whatever, they could change the number and try again.
Need phpBB services or a phpBB consultant? I offer most phpBB services. Getting lost managing phpBB? Buy my book, Mastering phpBB Administration. eBook and paper versions available.
User avatar
Dark❶
Registered User
Posts: 266
Joined: Mon Jan 15, 2018 1:22 pm
Location: D@rK V0id
Name: Dark❶ [dark1]
Contact:

Re: Best way to handle long processes to avoid timeouts

Post by Dark❶ »

Hi !

Well ! you can do something like search index creation in core phpBB,
Code to refer is here : https://github.com/phpbb/phpbb/blob/f98 ... h.php#L367

This will help you for the "PHP timeouts by processing reasonably sized chunks of content" ...

I hope this was useful for you.

Best regards :+1:
Dark❶ [dark1]
User avatar
MarkDHamill
Registered User
Posts: 4342
Joined: Fri Aug 02, 2002 12:36 am
Location: Florence, MA USA
Contact:

Re: Best way to handle long processes to avoid timeouts

Post by MarkDHamill »

Thanks! The still_on_time() function in /includes/functions.php looks useful.
Need phpBB services or a phpBB consultant? I offer most phpBB services. Getting lost managing phpBB? Buy my book, Mastering phpBB Administration. eBook and paper versions available.
Post Reply

Return to “Extension Writers Discussion”