Best way to handle long processes to avoid timeouts

[MarkDHamill]

I am working on a spam remover extension that uses the Akismet service.

Since boards can vary in size from tiny to enormous, when searching for spam I need an interface similar to the one used on phpBB installations or conversions that gives progress dialog boxes, refreshes itself regularly and has logic to reduce the likelihood of PHP timeouts by processing reasonably sized chunks of content. I do this toward the top of the program, but based on my experience it's not likely to be enough:

Code: Select all

		set_time_limit(0);

I'll be passing a lot of content to Akismet using its API, probably one post or private message at a time as I don't see a way to do this in bulk. I will have an interface that allows the administrator to refine the content sent, such as to a narrow range of dates. Often a board is spam free then can get a huge bulk of spam in a short period of time too voluminous to be removed by visual inspection of the content. My extension will have a way to flag and review potentially spammy posts and private messages and individually mark items as ham, once Akismet delivers its assessment.

I assume something AJAX-less is preferred. I've looked through the installer and while I'm sure the logic used is in there somewhere it is deeply abstracted away. Maybe there is a better way of doing this than the way the installer does it.

A best practices/built-in ways of doing this within phpBB would be appreciated.

[david63]

Not sure if it would work but have you considered running it as a cron job? That way it would run in background and you would possibly be able to split it into manageable chunks.

[MarkDHamill]

It doesn't provide a great user experience to run as a cron job. But maybe it could be kicked off and the ACP interface would let you know when done, or provide some sort of current statistics on the page of the current status, disabling other modules until it's completed.

[RMcGirr83]

I somewhat remember the old support tool kit used to chunk re-parsing of bbcode. Maybe that will give you some ideas.

[MarkDHamill]

So far the best idea I can think of is set a configuration variable for the number of posts or private messages to check in a "batch" before a screen refresh provides an update on progress. I have it set to 50, which means 50 "comment checks" (as Akismet calls it, as mostly they check Wordpress comments) occur before a screen refresh. The value could be changed in the ACP on a settings page. In the event of a PHP timeout, resource error or whatever, they could change the number and try again.

[Dark❶]

Hi !

Well ! you can do something like search index creation in core phpBB,
Code to refer is here : https://github.com/phpbb/phpbb/blob/f98 ... h.php#L367

This will help you for the "PHP timeouts by processing reasonably sized chunks of content" ...

I hope this was useful for you.

Best regards

[MarkDHamill]

Thanks! The still_on_time() function in /includes/functions.php looks useful.