How to fix phpBB notification email delay

Get help with installation and running phpBB 3.0.x here. Please do not post bug reports, feature requests, or MOD-related questions here.
Scam Warning
Forum rules
END OF SUPPORT: 1 January 2017 (announcement)
Locked
webventure
Registered User
Posts: 1
Joined: Tue Oct 23, 2012 1:12 pm
Contact:

How to fix phpBB notification email delay

Post by webventure » Tue Oct 23, 2012 1:26 pm

The notification email system in phpBB is notoriously unreliable, mainly because the default configuration is broken.

The default configuration does not send out any notification emails straight away, but instead waits until someone subsequently accesses any page on the site. Below is an explanation of the problem and a complete solution which I hope will help others suffering the same problems. The explanation can get a bit technical so skip down to the summary if you want to go straight to the solution.


The phpBB email queue

The fundamental cause of the problem is that PHP does not support the concept of background or worker threads, and sending out emails is a time consuming process. There are various hacks and workarounds in PHP for simulating the concept but many rely on an external "cron" process which is not necessarily available on the shared webservers that host phpBB installations, so phpBB has implemented a less-than-perfect solution that simulates a task queue but needs a bit of external prodding to work reliably. I am referring to phpBB 3.0.11 here which is the latest version at the time of writing, but there are hints that the problem will be looked at in a future version, possibly 3.2.

phpBB's queue implementation works by creating a file called cache/queue.php which contains data (in this case notification emails) that is to be processed in the background. If the file doesn't exist it means there is nothing to process. Note that there is typically also a file called queue.php.lock in the same directory which never disappears. Its presence does NOT indicate that the queue is locked as it is there permanently. The tasks in the queue are executed by the script cron.php when it is passed the query string cron_type=queue. This script is called by the inclusion of an element similar to the following in the footer of every page when the cache/queue.php file is present:

Code: Select all

<img src="cron.php?cron_type=queue" width="1" height="1" alt="cron" />
This allows the browser to initiate the cron.php script which sends the notification emails without the user having to wait for it to finish, or even see that it is being called. The script itself returns a 1x1 transparent gif image immediately before it even starts processing, so the browser just displays an invisible dot on the page. The cron script contains code to check that it is not being called multiple times. This is achieved by storing a value in the config table of the database. This is all very clever but suffers the problem that it relies on site receiving visitors to kick off the job.


E-mail package size

The first suggestion you may have read if you are searching for solutions to the problem of unreliable email notifications is to set "E-mail package size" to 0. This setting is found in the Administration Control Panel, General tab, E-mail settings. Setting the value to 0 tells phpBB not to use the queue at all, but instead to send all the emails in the script that is processing the new post on the forum. If you have lots of users subscribed to the forum this can result in a very long delay for the posting user before they get the confirmation that the post was successful, as the confirmation waits until all notification emails are sent. In my case it was taking over a minute, at which time a timeout error appears and the processing stops. Obviously not a viable solution. It may have been possible to implement this in a similar way to the cron script, where it returns all the content to the browser which redirects back to the forum index after displaying the success message, while the script continues to send emails in the background. I'm not sure why it hasn't been implemented that way, but it hasn't, so setting the email package size to zero isn't the solution.

The correct setting for the email package size depends on your hosting provider and configuration, but if you use follow all of the recommendations here you should set it to a value at least as high as the number of registered users. Even a huge number like 100000 should be fine, just not zero.


Use SMTP server for e-mail

Another suggestion you will see is to set "Use SMTP server for e-mail" to "yes". This setting is however only indirectly related to the problem of unreliable emails.

If it is set to "no", emails will be sent by the webserver or whatever default server PHP is configured to send mail with. This is not ideal as shared web servers can easily end up on spam blacklists if one of the hosting provider's customers is sending dodgy emails, and the mail server is not likely to be very closely monitored for abuse. Some providers also place restrictions on the number of emails you can send per hour. The email package size parameter can be used to limit the number of emails sent per hour, as by default phpBB won't send more than the specified number of emails per minute. The bottom line however is that relying on the default mail server with unknown configuration and adjusting parameters to stay under your service provider's limit is a risky and unnecessary game to play.

I recommend using an external SMTP server, so the setting should be set to "yes". I use a provider called Elastic Email (http://elasticemail.com) as they provide a reliable commercial-grade SMTP server for a tiny cost per email with no monthly fees, and throw in an online tracking system for every email sent. There are probably others providing a comparable service, but the point is these external providers specialise in email delivery so there are no lame limits on emails per hour and they actively monitor their servers to maximise performance and reliability. Believe me the benefits are huge in relation to the cost.

When using an external SMTP server it is often necessary to ask your web hosting provider to open up the outgoing port used to connect to the SMTP server. Many are reluctant to open up the standard SMTP port 25, so it is helpful if your SMTP provider makes their service available on an alternative port. Elastic Email provides a service on port 2525.


Ensuring notification emails go out immediately

To ensure notification emails go out immediately while still making use of the task queue, set the configuration parameter "queue_interval" to 0. Also ensure email package size is set to a very large number.

The queue_interval parameter is not accessible via the Administration Control Panel or anywhere else in the web interface. You'll have to set it directly in the database's config table.

The default value is 60, meaning that the queue is processed once every 60 seconds at the most, and in previous versions the default was 600. Setting it to zero may not be an option if your hosting provider places limits on the maximum number of emails sent per hour, which is yet another reason to use a commercial grade SMTP provider.

The reason this change solves the problem is because of the bug in the phpBB queue implementation that is causing the problem. The queue implementation uses the modified timestamp of the queue.php file as an indicator of when the queue was last processed. It does this because if the queue contains more emails that the configured package size limit, the processed emails are removed from the queue and the remaining ones are written back to the queue.php file, so in that case the file modified timestamp would reflect the time the queue was last processed. In a low volume forum however the queue is normally empty. When a user posts a message to the forum the required email notifications are saved to the queue, which creates the queue.php file. When the posting user is redirected back to the forum after the confirmation message, the displayed page includes the trigger for the cron script (described above) as expected, as it has detected the presence of the queue.php file. The emails are however not sent, because the faulty implementation sees that the queue.php file was only modified a few seconds ago and assumes the queue was last processed at this time, when in fact it may have been hours ago since it was last run. If the queue_interval is set to the default 60 seconds, this causes the cron script to exit without processing the queue. If it is set to zero, it allows the queue that was created only seconds ago to be processed. It would be very easy to store the time the queue was last processed in the database instead of using the queue.php file's timestamp, which would also fix the problem, but until the bug is fixed setting queue_interval to 0 is just as effective.

There still may be circumstances where this change alone does not solve the problem completely. For example I have not tested what happens if posts are made while the queue is currently processing. Assuming the notification emails are correctly and reliably added to the queue.php file it is likely the case that they are not processed immediately after the ones currently being processed. If no-one visits the forum for hours they may still sit in the queue for hours. This is why the next step is necessary.


Ensuring the email queue is processed regularly

To ensure no emails languish in the queue for extended periods of time the cron script must be called regularly. If the forum isn't visited frequently enough to ensure the timely delivery of queued emails, an external process must be set up to call it periodically. The easiest way to achieve this is to use a site monitoring service to monitor the page http://yourforum.com/cron.php?cron_type=queue every five minutes or so. There are many free services to choose from.


Ensuring notifications are sent regardless of whether the user has visited the forum recently

By default a user receives only one notification email to say there are new posts on the forum, and receives no more until they have visited the forum. This can be problematic because the user may have missed an email, or it didn't get through, or just didn't get around to visiting the forum. Then they will receive no more notifications on that forum or topic.

There is a mod called Prime Notify (https://www.phpbb.com/customise/db/mod/prime_notify/) that allows you to include the actual message text in each notification email, and send it regardless of whether the user has visited the forum since the last one.


Further tweaks

My phpBB installation is used for members of a small community group to communicate with each other. We initially had problems getting a critical mass of participation because posting something there wasn't seen by many people because very few people had subscribed to any forums, and no-one bothered subscribing because there was nothing much to see. An opt-out approach is far more effective with a small user base and a new technology. There are mods that automatically subscribe all users to new forums, but I tweaked my installation so that rows in the forums_watch and topics_watch tables represent unsubscribes instead of subscribes. The code changes required are surprisingly minimal, and it allows something that is impossible with the standard use of these tables. In a default installation if someone is subscribed to a forum they will receive notifications of all new posts and replies to that forum, and there is no way for them to unsubscribe from a single topic. By changing unsubscribes instead of subscribes, a user can choose to unsubscribe either from an individual topic or an entire forum. In the case of a small community group this is far more useful than being able to subscribe to a topic without subscribing to an entire forum. Ideally phpBB would allow both of these options out of the box.


Summary

1. Set the configuration parameter "queue_interval" to 0 (must be set directly in the database)

2. Set the configuration parameter "email_package_size" to a large number like 100000. This can be done through the Administration Control Panel.

3. Set up an external monitor to access the URL http://yourforum.com/cron.php?cron_type=queue every 5 minutes or so.

4. (optional but highly recommended) Use a fully monitored external SMTP server instead of relying on the default PHP mail system.

5. Tweak the notification system to include the message in notification emails and to send a notification to every subscribed user regardless of whether they have visited the forum since their last notification. See https://www.phpbb.com/customise/db/mod/prime_notify/


I hope others find this information useful as it took me several long and frustrating hours to figure out how to make this basic functionality work reliably. I was tempted many times to ditch phpBB and migrate to something else but I eventually ended up with a robust solution.

Martin Jericho
Last edited by HGN on Tue Oct 23, 2012 1:41 pm, edited 1 time in total.
Reason: Removed unnecessary (show off) link from post

User avatar
Brf
Support Team Member
Support Team Member
Posts: 51776
Joined: Tue May 10, 2005 7:47 pm
Location: {postrow.POSTER_FROM}
Contact:

Re: How to fix phpBB notification email delay

Post by Brf » Tue Oct 23, 2012 3:58 pm

If you set the Email Package Size to zero on the Email Settings page, phpBB does not queue up the emails.

User avatar
AmigoJack
Registered User
Posts: 5601
Joined: Tue Jun 15, 2010 11:33 am
Location: グリーン ヒル ゾーン
Contact:

Re: How to fix phpBB notification email delay

Post by AmigoJack » Wed Oct 24, 2012 6:17 am

I do not see the reason why you need a BBS at all - if you want a one-to-all communication via e-mail you'd be better off with a traditional discussion list.
The worst thing about censorship is ███████████
Affin wrote:
Tue Nov 20, 2018 9:51 am
The problem is probably not my English but you do not want to understand correctly.
...
We will not come anybody anyway, nevertheless, it's best to shit this.

rollytan
Registered User
Posts: 5
Joined: Thu Oct 09, 2014 7:08 pm

Re: How to fix phpBB notification email delay

Post by rollytan » Thu Jul 02, 2015 10:19 pm

webventure wrote:I hope others find this information useful as it took me several long and frustrating hours to figure out how to make this basic functionality work reliably. I was tempted many times to ditch phpBB and migrate to something else but I eventually ended up with a robust solution.

Martin Jericho
Yes! I found this perfectly helpful and a much needed solution to my problems with Boardwatch and a growing community that wants all emails to every topic/post.

Thank you very much.

Ronnie

User avatar
stevemaury
Support Team Member
Support Team Member
Posts: 50571
Joined: Thu Nov 02, 2006 12:21 am
Location: The U.P.
Name: Steve
Contact:

Re: How to fix phpBB notification email delay

Post by stevemaury » Thu Jul 02, 2015 11:09 pm

A solution to a non-existent problem.
Brf wrote:If you set the Email Package Size to zero on the Email Settings page, phpBB does not queue up the emails.
For REALLY good and VERY inexpensive hosting CLICK HERE

I can stop all your spam. I can upgrade or update your Board. PM or email me. (Paid support)

rollytan
Registered User
Posts: 5
Joined: Thu Oct 09, 2014 7:08 pm

Re: How to fix phpBB notification email delay

Post by rollytan » Thu Jul 02, 2015 11:13 pm

stevemaury wrote:A solution to a non-existent problem.
Brf wrote:If you set the Email Package Size to zero on the Email Settings page, phpBB does not queue up the emails.
Yes, Email Packet size to 0 sends out all emails at the time of post. As our list grows it was causing the post time to grow and people would go back and resubmit, creating duplicate posts.

By putting Email Packet Size to a large number (10000) and queue_interval to 0 , plus setting up a cron to fire every 5 minutes things seem to be working as they should.

Locked

Return to “[3.0.x] Support Forum”