[CDB] Digests 3.2.11

[whocarez]

Hello Mark,

thank you for your answers. Maybe my suggestion, that this error behavior is linked to the switch to daylight saving time was to hasty.
Digging through php-errors.log I found that:

Code: Select all

[01-Apr-2017 14:30:51 Europe/Berlin] PHP Fatal error:  Uncaught exception 'Twig_Error_Loader' with message 'There are no registered paths for namespace "phpbbservices_digests" in "overall_header.html" at line 67.' in /var/www/forum/vendor/twig/twig/lib/Twig/Loader/Filesystem.php:196
Stack trace:
#0 /var/www/forum/phpbb/template/twig/loader.php(107): Twig_Loader_Filesystem->findTemplate('@phpbbservices_...')
#1 /var/www/forum/vendor/twig/twig/lib/Twig/Loader/Filesystem.php(139): phpbb\template\twig\loader->findTemplate('@phpbbservices_...')
#2 /var/www/forum/vendor/twig/twig/lib/Twig/Environment.php(312): Twig_Loader_Filesystem->getCacheKey('@phpbbservices_...')
#3 /var/www/forum/vendor/twig/twig/lib/Twig/Environment.php(378): Twig_Environment->getTemplateClass('@phpbbservices_...', NULL)
#4 /var/www/forum/phpbb/template/twig/environment.php(166): Twig_Environment->loadTemplate('@phpbbservices_...', NULL)
#5 /run/shm/forum-cache/twig/39/39d6323241eb3ff44d7134ab458c77c4c579fd6133f95aa23a3beac77ab52798.php(325): phpbb\template\twig\environment->loadTemplate('@phpbbs in /var/www/forum/vendor/twig/twig/lib/Twig/Loader/Filesystem.php on line 196

[MarkDHamill]

Purging the cache may make this problem go away.

[whocarez]

ok, that's clear, but what can I do, to get a hint, what caused this error?

[MarkDHamill]

Did that problem go away with purging the cache?

Not sure exactly what caused it but phpBB's templating system is complicated, particularly since the introduction of TWIG. I don't need to understand why it occurred, I have just learned that clearing the cache when I see template errors like these usually solves the problem. It will force phpBB to recreate cache files with a .php suffix in the cache folder. Templates are "compiled" into .php files in the cache folder.

[whocarez]

ok,it run through, but the users got their emails still several times, despite the fact, that the last data was correct saved: Sunday 2. April 2017, 22:48. The same user got the same mail on the cron job before at 22:36. My cron job is running every 12 minutes.

[MarkDHamill]

If you use a phpBB cron as opposed to a system cron then digests are sent when board traffic occurs, so they may be delivered later than the scheduled hour. See the first post for more information.

[whocarez]

it's the system cron job, which is running every 12 minutes and somewhat causing to send the same digest on every run.

[MarkDHamill]

Digests should not go out more than once an hour, as when its cron is called it is checked to make sure an hour has elapsed. If it hasn't, the mailer quits quickly. With a properly configured system cron that executes at least once an hour, users should get their digest in the hour requested, based on the user_digest_send_hour_gmt stored in the database. The mail server PHP hands it off to is responsible for sending them out and if for some reason it doesn't do this quickly, that may be the issue.

[whocarez]

hm, ok, I had run for some reasons ./bin/phpbbcli.php config:set cron_lock 0 right after ./bin/phpbbcli.php cron:run

If I understand it right, than this may cause the situation, that when the time has come for weekly digests (about 1300 users get it at the same time) the cron job is running longer than these 12 minutes and that's why the cron job starts again in the next cycle.

I'll test it now, but some users got the digest since Saturday about 10 times ...

[whocarez]

Ok, after the automatic release of the cron lock after an hour. I get this error:

Code: Select all

PHP Warning:  date() expects parameter 2 to be long, string given in /var/www/forum/ext/phpbbservices/digests/cron/task/digests.php on line 536
PHP Warning:  date() expects parameter 2 to be long, string given in /var/www/forum/ext/phpbbservices/digests/cron/task/digests.php on line 537
PHP Warning:  cal_days_in_month(): invalid date. in /var/www/forum/ext/phpbbservices/digests/cron/task/digests.php on line 543

so this error is caused here:

Code: Select all

				case constants::DIGESTS_MONTHLY_VALUE:
					// Calculate seconds in previous month, which depends on number of days in that month
					$use_year = date($now, 'Y');
					$use_month = date($now, 'n') - 1;
					if ($use_month == 0)
					{
						$use_month = 12;
						$use_year--;
					}
					$use_days_in_month = cal_days_in_month (CAL_GREGORIAN, $use_month, $use_year);
					if ($row['user_digest_last_sent'] + ($use_days_in_month * 60 * 60 * 24) > $now)
					{
						continue;
					}
				break;

[MarkDHamill]

hm, ok, I had run for some reasons ./bin/phpbbcli.php config:set cron_lock 0 right after ./bin/phpbbcli.php cron:run

If I understand it right, than this may cause the situation, that when the time has come for weekly digests (about 1300 users get it at the same time) the cron job is running longer than these 12 minutes and that's why the cron job starts again in the next cycle.

I'll test it now, but some users got the digest since Saturday about 10 times ...

According to the Wiki, a system cron should look something like this:

* * * * * cd "/path/to/board"; ./bin/phpbbcli.php cron:run

I assume you first cd to your phpBB root directory.

You should not need to add items to the cron that affect the cron. While the cron is taking place it locks itself until all crons are completed then it unlocks itself.

Yes, sending 1300 digests should take a long time. I assume you have a dedicated or virtual server and no quotas on outgoing emails. It's going to take a while if resource limitations don't kick in and stop it. I recommending using the digests balance load function in the ACP to spread these across 24 hours. Sending all at the same hour invites issues like you are having.

[MarkDHamill]

Ok, after the automatic release of the cron lock after an hour. I get this error:

Code: Select all

PHP Warning:  date() expects parameter 2 to be long, string given in /var/www/forum/ext/phpbbservices/digests/cron/task/digests.php on line 536
PHP Warning:  date() expects parameter 2 to be long, string given in /var/www/forum/ext/phpbbservices/digests/cron/task/digests.php on line 537
PHP Warning:  cal_days_in_month(): invalid date. in /var/www/forum/ext/phpbbservices/digests/cron/task/digests.php on line 543

so this error is caused here:

Code: Select all

				case constants::DIGESTS_MONTHLY_VALUE:
					// Calculate seconds in previous month, which depends on number of days in that month
					$use_year = date($now, 'Y');
					$use_month = date($now, 'n') - 1;
					if ($use_month == 0)
					{
						$use_month = 12;
						$use_year--;
					}
					$use_days_in_month = cal_days_in_month (CAL_GREGORIAN, $use_month, $use_year);
					if ($row['user_digest_last_sent'] + ($use_days_in_month * 60 * 60 * 24) > $now)
					{
						continue;
					}
				break;

Interesting how long this bug has been undiscovered. The two lines should be:

Code: Select all

					$use_year = date('Y', $now);
					$use_month = date('n', $now) - 1;

[MarkDHamill]

Hi Mark

I've upgraded to PHP 7.1, the forum was on something ancient like 5.4 or 5,3, I have discovered an additional error code (sorry).

24 Mar 2017, 08:38 Error while creating image
» Error in [ROOT]/ext/phpbbservices/digests/cron/task/digests.php on line 1595: array_multisort(): Argument #5 is expected to be an array or a sort flag
24 Mar 2017, 08:38 Error while creating image
» Error in [ROOT]/phpbb/user.php on line 599: DateTime::setTimezone() expects parameter 1 to be DateTimeZone, null given
24 Mar 2017, 08:38 Error while creating image
» Error in [ROOT]/ext/phpbbservices/digests/cron/task/digests.php on line 1595: array_multisort(): Argument #5 is expected to be an array or a sort flag
24 Mar 2017, 08:38 Error while creating image
» Error in [ROOT]/phpbb/user.php on line 599: DateTime::setTimezone() expects parameter 1 to be DateTimeZone, null given

I have again been through the user table, everyone has been set to UTC including the 'anonymous' account and the bots.

Do you have any suggestions on where I should look now? Many thanks, Steve

steveosonic, I found the error. Ironically, it was a result of the PhpStorm editor I was using, which complains if you have instances of uninitialized variables. I initialized these variables and in most cases I did not declare them as null arrays. So please replace these lines with:

Code: Select all

		$topic_first_poster_name = array();	// Keep PhpStorm happy
		$username_clean = array();			// Keep PhpStorm happy
		$post_time = array();				// Keep PhpStorm happy
		$post_subject = array();			// Keep PhpStorm happy
		$topic_last_post_time = array();	// Keep PhpStorm happy
		$topic_replies = array();			// Keep PhpStorm happy
		$topic_title = array();				// Keep PhpStorm happy
		$topic_views = array();				// Keep PhpStorm happy

[whocarez]

Yes, sending 1300 digests should take a long time. I assume you have a dedicated or virtual server and no quotas on outgoing emails. It's going to take a while if resource limitations don't kick in and stop it. I recommending using the digests balance load function in the ACP to spread these across 24 hours. Sending all at the same hour invites issues like you are having.

Would be a nice feature to balance sending time within a defined period. So lets say between 7 and 21, because no one - I hope - reads email at 4 o'clock in the morning.

Interesting how long this bug has been undiscovered. The two lines should be:

Code: Select all

					$use_year = date('Y', $now);
					$use_month = date('n', $now) - 1;

Thanks for the quick solution.
I'm hesitating to reenable digest sendings, because a lot of my users got these weekly digests on Saturday and Sunday several times and started complaining. Is there an easy way to make dry runs triggered by the cron job? I know this manual test, but I need something to test the whole cycle - only looking at the log entries - but without really sending the digest or creating instead of sending files as in the manual test.

What do I need to write to $result in html_messenger.php, so the function is working normally, but not sending anything in reality?

Code: Select all

		// Send message ...
		$mail_to = ($to == '') ? 'undisclosed-recipients:;' : $to;
		$err_msg = '';

		if ($config['smtp_delivery'])
		{
			$result = smtpmail($this->addresses, mail_encode($this->subject), wordwrap(utf8_wordwrap($this->msg), 997, "\n", true), $err_msg, $headers);
		}
		else
		{
			$result = phpbb_mail($mail_to, $this->subject, $this->msg, $headers, $this->eol, $err_msg);
		}

		if (!$result)
		{
			$this->error('EMAIL', $err_msg);
			return false;
		}

[MarkDHamill]

I can keep your suggestion for Balance Load as a future enhancement. It would also be good to allow filtering so only certain types of digests are affected (like monthly) or possibly certain types of digests (HTML, text, etc.)

The "Manually run the mailer" feature allows you to simulate any date and hour. So you could do this 24 times for each hour, which would be a bit tedious. You can also write them to cache instead of mailing them with this feature so you can see what they look like in a browser.

Weekly digests are sent on a day of week that you specify. However, it's a day of the week based on Universal Time, so it's possible that some subscribers in the western hemisphere may receive them on Saturday in their local time zone. It may be able to change this with a future enhancement, but the logic is likely to be complicated.