[CDB] Digests 3.2.11

A place for Extension Authors to post and receive feedback on Extensions still in development. No Extensions within this forum should be used within a live environment!
Get Involved
Forum rules
READ: phpBB.com Board-Wide Rules and Regulations

IMPORTANT: Extensions Development rules

IMPORTANT FOR NEEDED EVENTS!!!
If you need an event for your extension please read this for the steps to follow to request the event(s)
Locked
whocarez
Registered User
Posts: 144
Joined: Sat Apr 18, 2009 9:19 pm

Re: [RC] Digests 3.2.0-RC7

Post by whocarez » Mon Apr 03, 2017 7:47 pm

The manual test is not really what I need. I need somehow real circumstances without real consequences. So in the manual test the time oft last sending the digest isn't changed and though can't be recognized in the next cron cycle. My problem was, that the digests were sent in every cron cycle despite the fact, that the same user got his digest already in the cron cycle before. So the extension ignored the last time sent date.
Probably there is no "easy" way, to test it like that.

User avatar
MarkDHamill
Registered User
Posts: 3954
Joined: Fri Aug 02, 2002 12:36 am
Location: Florence, MA USA
Contact:

Re: [RC] Digests 3.2.0-RC7

Post by MarkDHamill » Mon Apr 03, 2017 7:55 pm

The admin log will show all digests mailed and when.

The only way I know to test something like this is risky: change subscriber email addresses to an address that you can access, and restore it later.
Get the latest versions of my Digests and Smartfeed extensions.
Need phpBB services or a phpBB consultant? I offer most phpBB services.

whocarez
Registered User
Posts: 144
Joined: Sat Apr 18, 2009 9:19 pm

Re: [RC] Digests 3.2.0-RC7

Post by whocarez » Mon Apr 03, 2017 8:07 pm

ok, then I'll have to test it in real live.
Anyway, thank you for your patience!

Anuj Dhawan
Registered User
Posts: 375
Joined: Sat Aug 10, 2013 6:44 pm

Re: [RC] Digests 3.2.0-RC7

Post by Anuj Dhawan » Tue Apr 04, 2017 5:00 pm

I think it's been reported already however, just for the record, please have a look here - viewtopic.php?f=466&t=2416806&p=14705996#p14705636
Thanks,

stevenospam
Registered User
Posts: 44
Joined: Thu Dec 15, 2011 2:02 am

Re: [RC] Digests 3.2.0-RC7 migrate from phpbb 3.0.14

Post by stevenospam » Wed Apr 05, 2017 11:32 pm

What is the best way to migrate to 3.2 and keep the digests config info for each user. Do I uninstall digests on the 3.0 board using STK? Leave digests installed on the 3.0 board? Other? On the 3.0 board digests is at the 2.2.23 level.

User avatar
MarkDHamill
Registered User
Posts: 3954
Joined: Fri Aug 02, 2002 12:36 am
Location: Florence, MA USA
Contact:

Re: [RC] Digests 3.2.0-RC7 migrate from phpbb 3.0.14

Post by MarkDHamill » Thu Apr 06, 2017 12:20 am

stevenospam wrote:
Wed Apr 05, 2017 11:32 pm
What is the best way to migrate to 3.2 and keep the digests config info for each user. Do I uninstall digests on the 3.0 board using STK? Leave digests installed on the 3.0 board? Other? On the 3.0 board digests is at the 2.2.23 level.
See the first post of the topic. Migration from 3.0 digest mod is supported. There is a code change on page 30 of this topic related to the tools class that should be applied before installing the extension on 3.2.
Get the latest versions of my Digests and Smartfeed extensions.
Need phpBB services or a phpBB consultant? I offer most phpBB services.

User avatar
MarkDHamill
Registered User
Posts: 3954
Joined: Fri Aug 02, 2002 12:36 am
Location: Florence, MA USA
Contact:

Re: [RC] Digests 3.2.0-RC7

Post by MarkDHamill » Thu Apr 06, 2017 12:45 am

Since the 3.2.0 development release, a lot of testing has been done by posters on this topic, as well as myself. I am working on a new release but of course it takes time. It is recommended that you make the following changes to the following files if you have downloaded and/or installed digest extension version 3.2.0 for phpBB 3.2. Some are serious and are bolded below.

In cron/task/digests.php, around line 410 find:

Code: Select all

		// Get users requesting digests for the current hour. Also, grab the user's style, so the digest will have a familiar look.
After add:

Code: Select all

		$allowed_user_types = 'AND ' . $this->db->sql_in_set('user_type', array(USER_FOUNDER, USER_NORMAL));

Around line 427 find:

Code: Select all

								AND user_inactive_reason = 0
Replace with:

Code: Select all

								AND user_inactive_reason = 0 " . $allowed_user_types . "
Around line 450 find:

Code: Select all

								AND user_inactive_reason = 0
Replace with:

Code: Select all

								AND user_inactive_reason = 0 " . $allowed_user_types . "
Around line 510 find:

Code: Select all

		// Execute the SQL to retrieve the relevant posts. Note, if $this->config['phpbbservices_digests_max_items'] == 0 then there is no limit on the rows returned
		$result_posts = $this->db->sql_query_limit($sql_posts, $this->config['phpbbservices_digests_max_items']); 
Replace with:

Code: Select all

		$result_posts = $this->db->sql_query($sql_posts);
This is important to change as monthly digests logic will be incorrect if not fixed. Around line 452 find:

Code: Select all

					$use_year = date($now, 'Y');
					$use_month = date($now, 'n') - 1;
Replace with:

Code: Select all

					$use_year = date('Y', $now);
					$use_month = date('n', $now) - 1;
This is an important one to fix as post and topic sorting in the digest may be incorrect if not fixed. Around line 1205 find:

Code: Select all

		$topic_first_poster_name = '';	// Keep PhpStorm happy
		$username_clean = '';			// Keep PhpStorm happy
		$post_time = array();			// Keep PhpStorm happy
		$post_subject = '';				// Keep PhpStorm happy
		$topic_last_post_time = '';		// Keep PhpStorm happy
		$topic_replies = 0;				// Keep PhpStorm happy
		$topic_title = '';				// Keep PhpStorm happy
		$topic_views = 0;				// Keep PhpStorm happy
Replace with:

Code: Select all

		$topic_first_poster_name = array();	// Keep PhpStorm happy
		$username_clean = array();			// Keep PhpStorm happy
		$post_time = array();				// Keep PhpStorm happy
		$post_subject = array();			// Keep PhpStorm happy
		$topic_last_post_time = array();	// Keep PhpStorm happy
		$topic_replies = array();			// Keep PhpStorm happy
		$topic_title = array();				// Keep PhpStorm happy
		$topic_views = array();				// Keep PhpStorm happy
In event/main_listener.php

Around line 78 find:

Code: Select all

		if ($this->config['phpbbservices_digests_enable_auto_subscriptions'] == 1 || $subscribe_on_registration)
Replace with:

Code: Select all

		$is_human = ($event['sql_ary']['user_type'] == USER_IGNORE) ? false : true;
		if ($is_human && ($this->config['phpbbservices_digests_enable_auto_subscriptions'] == 1 || $subscribe_on_registration))
In /migrations/convert_mod_schema.php

Fixing this is critical if upgrading from the 3.0 mod. Around line 67 find:

Code: Select all

		$tools = new \phpbb\db\tools($this->db);
Replace with:

Code: Select all

		$tools_factory = new \phpbb\db\tools\factory();
		$tools = $tools_factory->get($this->db);
Get the latest versions of my Digests and Smartfeed extensions.
Need phpBB services or a phpBB consultant? I offer most phpBB services.

whocarez
Registered User
Posts: 144
Joined: Sat Apr 18, 2009 9:19 pm

Re: [RC] Digests 3.2.0-RC7

Post by whocarez » Sat Apr 08, 2017 12:31 pm

Hello Mark,

once again returning to my problem, which appeared after the switch to the daylight saving time two weeks ago.

Today I run the cron job manually and it ended up in an endless loop. I stopped it after the third round.
The logs showed me for the first user sending on the following times:
  • Saturday 8. April 2017, 14:52
  • Saturday 8. April 2017, 14:59
  • Saturday 8. April 2017, 15:06
  • Saturday 8. April 2017, 15:13

For the same user the Digest last sent (Based on 2 UTC) was correctly set to Saturday 8. April 2017, 14:13. You see the difference of one hour. I don't know whether it matters or not.

Any suggestions? Maybe I can send you a diff between the used files by me (there are now already some changes) and the original from github? The problem sits as usually between the chair and the keyboard :-).

User avatar
MarkDHamill
Registered User
Posts: 3954
Joined: Fri Aug 02, 2002 12:36 am
Location: Florence, MA USA
Contact:

Re: [RC] Digests 3.2.0-RC7

Post by MarkDHamill » Sat Apr 08, 2017 1:48 pm

Did the server time get changed as a move to DST? Or was this simply that your timezone changed in the UCP?

It's possible that if server time gets rolled back or forward an hour something like this might occur. It's a test case I never considered.
Get the latest versions of my Digests and Smartfeed extensions.
Need phpBB services or a phpBB consultant? I offer most phpBB services.

whocarez
Registered User
Posts: 144
Joined: Sat Apr 18, 2009 9:19 pm

Re: [RC] Digests 3.2.0-RC7

Post by whocarez » Sat Apr 08, 2017 2:30 pm

The server time switched also. For today date on commandline: Sa 8. Apr 16:29:39 CEST 2017
date("r"); returns Sat, 08 Apr 2017 16:29:36 +0200.

Interesting is, that one user with these weekly digests get the three mails with this $recipient_time
  • Saturday 1. April 2017, 11:52
  • Sunday 2. April 2017, 11:52
  • Monday 3. April 2017, 11:52

User avatar
MarkDHamill
Registered User
Posts: 3954
Joined: Fri Aug 02, 2002 12:36 am
Location: Florence, MA USA
Contact:

Re: [RC] Digests 3.2.0-RC7

Post by MarkDHamill » Sat Apr 08, 2017 3:06 pm

In ACP > System settings. Is "Run periodic tasks from system cron" Yes or No?
Get the latest versions of my Digests and Smartfeed extensions.
Need phpBB services or a phpBB consultant? I offer most phpBB services.

whocarez
Registered User
Posts: 144
Joined: Sat Apr 18, 2009 9:19 pm

Re: [RC] Digests 3.2.0-RC7

Post by whocarez » Sat Apr 08, 2017 3:41 pm

MarkDHamill wrote:
Sat Apr 08, 2017 3:06 pm
In ACP > System settings. Is "Run periodic tasks from system cron" Yes or No?
yes, of course ... that's why I could start it controlled on command line and stop it after the third round.

User avatar
MarkDHamill
Registered User
Posts: 3954
Joined: Fri Aug 02, 2002 12:36 am
Location: Florence, MA USA
Contact:

Re: [RC] Digests 3.2.0-RC7

Post by MarkDHamill » Sat Apr 08, 2017 8:15 pm

ACP > Extensions > Digests > Reset mailer. If you do this and then do it again, does it still cause a problem?
Get the latest versions of my Digests and Smartfeed extensions.
Need phpBB services or a phpBB consultant? I offer most phpBB services.

User avatar
MarkDHamill
Registered User
Posts: 3954
Joined: Fri Aug 02, 2002 12:36 am
Location: Florence, MA USA
Contact:

Re: [RC] Digests 3.2.0-RC7

Post by MarkDHamill » Sun Apr 09, 2017 1:15 am

There is a loop in /cron/task/digests.php (the mailer) that sends out digests for how ever many hours are necessary. Starting around line 281:

Code: Select all

		for ($i=(1 - $hours_to_do); $i <= 0; $i++)
		{
			$success = $this->mail_digests($now, $i);
			if (!$success)
			{
				// Need logic to notify or email admins when mailing digests fails. There are probably emailing issues needing fixing.
				return false;
			}
			else if (!$this->manual_mode)
			{
				// Note that the hour was processed successfully.
				$this->config->set('phpbbservices_digests_cron_task_last_gc', $now + ($i * 60 * 60));
			}
		}
This loop would continue indefinitely if $i starts out as $i > 0. $hours_to_do is supposed to be a positive integer and should never be less than 1, otherwise the loop will continue forever.

Around line 150 I see where $hours_to_do gets set:

Code: Select all

			// How many hours of digests are wanted? We want to do it for the number of hours between now and when digests were last ran successfully.
			$hours_to_do = floor(($now - $this->config['phpbbservices_digests_cron_task_last_gc']) / (60 * 60));
$hours_to_do will be a whole number if $this->config['phpbbservices_digests_cron_task_last_gc'] is less than $now. $now just reads the current time in UTC. On line 115:

Code: Select all

		$now = time();
So for this to occur either $this->config['phpbbservices_digests_cron_task_last_gc'] (the time when digests were last run successfully) is after the current time or your server is incorrectly reporting the current time as a UTC timestamp. I doubt that latter case is possible.

A relatively simple way to see what's going on is to use this query (assuming MySQL and phpbb_ is your table prefix):

SELECT from_unixtime(config_value), from_unixtime(unix_timestamp()) from phpbb_config where config_name = 'phpbbservices_digests_cron_task_last_gc'

In my case I see:

Code: Select all

from_unixtime(config_value) 	from_unixtime(unix_timestamp()) 	
2017-04-06 20:24:19 	2017-04-08 20:45:53
Note that should_run() should keep the run function from running if an hour has not elapsed since digests were last sent out:

Code: Select all

	public function should_run()
	{
		return (bool) ($this->config['phpbbservices_digests_cron_task_last_gc'] + $this->config['phpbbservices_digests_cron_task_gc'] < time());
	}
BTW, $this->config['phpbbservices_digests_cron_task_gc'] is set to 3600 (one hour) when digests are installed. You should check this value too.

I will add code to prevent the mail_digests function from running if $i < 1. That will prohibit the error from happening. But it should never occur in the first place. The first block of code would be:

Code: Select all

		// Process digests for each hour. For example, to do three hours, start with -2 hours from now and end after 0 hours from now (current hour).
		if ($hours_to_do >= 1)
		{
			for ($i=(1 - $hours_to_do); ($i <= 0); $i++)
			{
				$success = $this->mail_digests($now, $i);
				if (!$success)
				{
					// Need logic to notify or email admins when mailing digests fails. There are probably emailing issues needing fixing.
					return false;
				}
				else if (!$this->manual_mode)
				{
					// Note that the hour was processed successfully.
					$this->config->set('phpbbservices_digests_cron_task_last_gc', $now + ($i * 60 * 60));
				}
			}
		}
Get the latest versions of my Digests and Smartfeed extensions.
Need phpBB services or a phpBB consultant? I offer most phpBB services.

whocarez
Registered User
Posts: 144
Joined: Sat Apr 18, 2009 9:19 pm

Re: [RC] Digests 3.2.0-RC7

Post by whocarez » Sun Apr 09, 2017 8:00 am

Hello,

ok, SELECT from_unixtime(config_value), from_unixtime(unix_timestamp()) from phpbb_config where config_name = 'phpbbservices_digests_cron_task_last_gc' shows

Code: Select all

from_unixtime(config_value) 	from_unixtime(unix_timestamp()) 
2017-04-04 10:52:56 	2017-04-09 09:55:10
phpbbservices_digests_cron_task_gc is set to 3600

I reseted it now and as expected the cron is running through without problems. Waiting for the next Saturday :-).

Locked

Return to “Extensions in Development”