[CDB] Digests 3.2.11

[MarkDHamill]

Chris, to get approved the extension must get approved by the phpBB extension review team. The first review took four months so I would not hold your breath. I haven't resubmitted it yet but I have worked through all the things they wanted fixed. Then phpBB 3.2 came along. When I tested it, the mailer interface failed. This is due to updating the TWIG library, so I have to work through all of that. It is clear to me that I will need a branch, one for the 3.1 line and one for the 3.2 line.

While not approved yet, version 3.0.7 addresses all the problems note. However, it will not work on 3.2. The first post of the topic has a link to it. Or you can wait for 3.0.8 which should be quite stable too. Hopefully I can puzzle through all the things that don't work in a week or two and release it, but if TWIG changes turn out to be major it may involve a substantial rewrite to the mailer. If it seems stable, it will go to the extension review team for approval.

In general there tends to be a long backlog of extensions for review. I'm hoping for a "second pass" the team will give it a priority.

[whocarez]

Using latest Digests from github with phpbb 3.1.10 and PHP 5.4.45-0+deb7u6 starting the cron job ended in
PHP Fatal error: Cannot redeclare gen_sort_selects() ...

It turned out, that Digests interfered with another extension.

After changing digest/cron/task/digests.php from

Code: Select all

include($this->path_prefix . 'includes/functions_content.' . $this->phpEx);	

to

Code: Select all

require_once($this->path_prefix . 'includes/functions_content.' . $this->phpEx);	

it ran through. but I got this warning:

Code: Select all

  
  [Exception]                                               
  DateTimeZone::__construct(): Unknown or bad timezone (0) 

Figured out that it was because of via Tapatalk registered users, who had no time zone associated.
user_timezone was set to 0.
After changing their time zones, everything went through without errors and mails were sent out.

Hope this helps to make this extension better.

[MarkDHamill]

Glad it works. I don't think Tapatalk is an approved extension. If it had, the error probably would not have happened. require_once is generally not allowed in phpBB code.

[whocarez]

ah, ok I have a lot of require_once functions in some of these converted mods. So instead it should be include_once.
For this error with over Tapatalk registered users, maybe there should be made a test, whether the user_timezone is set to 0 and in these cases to use the board timezone.

[MarkDHamill]

There is such a test in digests. For example if you install a new forum, then create a digest for the admin in the UCP, a message will come up that you have no timezone assigned. The same will happen to any user if their timezone string is empty. I'm guessing the Tapatalk "user" that gets created never logs in, so there is no UCP to go to, so it never gets assigned.

I noted this glitch in a bug report:

https://tracker.phpbb.com/browse/PHPBB3-14404?filter=-2

I had my Smartfeed mod rejected once for a require_once statement which was in a third party library I used. I actually had to code around it to get it approved. I assume it would still raise a flag with a review team, so I've learned to avoid it.

[whocarez]

ok, thanks for this hint

[MarkDHamill]

Version 3.2.0 (RC7) is released. It works for phpBB 3.2 Rhea only. See the first post for details.

My basic task was to get 3.0.7 working for phpBB 3.2 Rhea, so there is little in the way of bug fixes or additional functionality. Of course there are possibly bugs. There are so many paths and permutations in the extension that it's impossible to test every path.

I chose to call this version 3.2.0 rather than 3.0.8 because it works on 3.2 only. The version number sets the expectation. If you try to run it on 3.1 it should fail to install.

Some notes:

• If you manually run the mailer in the Administration Control Panel, after finishing you will notice the menu on the left mostly disappears. Click on the continue link and it reappears. I did try to find a solution to this but with the template engine changes it was too complex to figure out right now. Most likely you would not notice this anyhow.
• A bug which allowed an entry of a decimal in the field "Maximum number of posts in the digest" in the UCP, Digest post setting, has been fixed.
• Allows a translation author to take credit for their translation. Their (optional) name and contact information, if provided, will show up at the bottom of the User Control Panel for digests.

  Code: Select all

  	'DIGESTS_TRANSLATED_BY'				=> 'translated by',
  	'DIGESTS_TRANSLATOR_NAME'			=> '',	// Leave null string to suppress translator name
  	'DIGESTS_TRANSLATOR_CONTACT'		=> '',	// Leave null string to suppress contact info, if used use: mailto:[email protected] or a URL if you have a website.
  

• Fixed what amounts to an interesting bug in phpBB, at least as it pertains to digests, that affects the display of the Edit Subscribers interface, specifically the "Last Visit" and "Digest Last Sent" column headers. The display of the time zone information was not locale specific because the formats used in the PHP Date syntax is not locale specific. This caused date headers like UTC+2 to appear in French versions when +2UTC was desired. Integrated a function I found to translate the time zone display to strftime format instead.
• Found and fixed a PHP notice that occurred if there were no private messages to show in the digest but their display was requested.

The following Digest translations exist, but as of this posting may not be fully compatible with phpBB 3.2:

French: https://github.com/bonnaphil/fr
German: https://github.com/Praggle/digests

[whocarez]

Hi Mark,
I'm sorry if someone else mentioned that already. I also didn't look into the code. I use the last version for 3.1.x.
Following scenario:
If I set for all users Maximum posts allowed in any digest for example to 50 and they get only the first post of a thread and than I set also up Always exclude these forums: than the algorithm takes first the last 50 posts and then he excludes posts from excluded forums. For example if I have about 300 new posts a week because of RSS integration and this excluded forum contains the first 48 picked up posts, than the user gets a digest with only 2 posts. The 250 posts after 50 aren't checked whether they are fitting the criteria.
Maybe it's not working like that, but in practice from about 40 posts of not excluded forums, the user gets only 2 posts, if I set up this limit for maximum posts allowed.

[MarkDHamill]

No one has reported this before but yes, you describe how it currently works. It is easily fixed. Change lines 510-511 in /cron/task/digests.php from:

Code: Select all

		// Execute the SQL to retrieve the relevant posts. Note, if $this->config['phpbbservices_digests_max_items'] == 0 then there is no limit on the rows returned
		$result_posts = $this->db->sql_query_limit($sql_posts, $this->config['phpbbservices_digests_max_items']);

to:

Code: Select all

		$result_posts = $this->db->sql_query($sql_posts);

It turns out that in the create_content() function I manually count the posts in the digests anyhow, so the code is not necessary. It will be fixed in the next version. Thanks for reporting it.

[MarkDHamill]

One bug was detected by a user that I did not find in my testing, but my upgrade was from 3.0.7. ext/phpbbservices/digests/migrations/convert_mod_schema.php, line 67 is:

Code: Select all

		$tools = new \phpbb\db\tools($this->db);

and should be:

Code: Select all

		$tools_factory = new \phpbb\db\tools\factory();
		$tools = $tools_factory->get($this->db);

I don't recall these changes to the database tools being announced with 3.2 Rhea.

[whocarez]

@MarkDHamill thank you for your quick support!

Today I detected another issue that is linked to the s9e/mediaembeded extension.
If a post contains links embedded by this extension, for example a youtube video, than it renders it within an iframe. Iframes are in emails often considered as dangerous or spammy, so the digest mail which contains it, will not be delivered.
So I suggest a remove of any <iframe>…</iframe> in html mails.

[whocarez]

Another small issue is, that bots are also added to the digest, when a new one is added and Enable automatic subscriptions: is switched on. It is under normal circumstances no problem, but I used a script which added about 200 common bots and some of them had email addresses . So I got a lot of bounced emails for the standard weekly digest for which new users are automatically subscribed.

UPDATE `phpbb_users` SET `user_digest_type` = 'NONE' WHERE `group_id` =6 AND `user_digest_type` LIKE 'WEEK' solved the problem, but maybe the extension should exclude bots from subscribing to the digest .

[MarkDHamill]

Another small issue is, that bots are also added to the digest, when a new one is added and Enable automatic subscriptions: is switched on. It is under normal circumstances no problem, but I used a script which added about 200 common bots and some of them had email addresses . So I got a lot of bounced emails for the standard weekly digest for which new users are automatically subscribed.

UPDATE `phpbb_users` SET `user_digest_type` = 'NONE' WHERE `group_id` =6 AND `user_digest_type` LIKE 'WEEK' solved the problem, but maybe the extension should exclude bots from subscribing to the digest .

Excellent catch! Adding new robots is so unusual I'm not surprised no one has noticed until now.

Lines 78-101 of /ext/phpbbservices/digests/event/main_listener.php are:

Code: Select all

		if ($this->config['phpbbservices_digests_enable_auto_subscriptions'] == 1 || $subscribe_on_registration)
		{
			$sql_ary = $event['sql_ary'];
			$sql_ary['user_digest_attachments'] 		= $this->config['phpbbservices_digests_user_digest_attachments'];
			$sql_ary['user_digest_block_images'] 		= $this->config['phpbbservices_digests_user_digest_block_images'];
			$sql_ary['user_digest_filter_type'] 		= $this->config['phpbbservices_digests_user_digest_filter_type'];
			$sql_ary['user_digest_format'] 				= $this->config['phpbbservices_digests_user_digest_format'];
			$sql_ary['user_digest_max_display_words'] 	= $this->config['phpbbservices_digests_user_digest_max_display_words'];
			$sql_ary['user_digest_max_posts'] 			= $this->config['phpbbservices_digests_user_digest_max_posts'];
			$sql_ary['user_digest_min_words'] 			= $this->config['phpbbservices_digests_user_digest_min_words'];
			$sql_ary['user_digest_new_posts_only'] 		= $this->config['phpbbservices_digests_user_digest_new_posts_only'];
			$sql_ary['user_digest_no_post_text'] 		= $this->config['phpbbservices_digests_user_digest_no_post_text'];
			$sql_ary['user_digest_pm_mark_read'] 		= $this->config['phpbbservices_digests_user_digest_pm_mark_read'];
			$sql_ary['user_digest_remove_foes'] 		= $this->config['phpbbservices_digests_user_digest_remove_foes'];
			$sql_ary['user_digest_reset_lastvisit'] 	= $this->config['phpbbservices_digests_user_digest_reset_lastvisit'];
			$sql_ary['user_digest_send_hour_gmt'] 		= ($this->config['phpbbservices_digests_user_digest_send_hour_gmt'] == -1) ? rand(0,23) : $this->config['phpbbservices_digests_user_digest_send_hour_gmt'];
			$sql_ary['user_digest_send_on_no_posts'] 	= $this->config['phpbbservices_digests_user_digest_send_on_no_posts'];
			$sql_ary['user_digest_show_mine'] 			= ($this->config['phpbbservices_digests_user_digest_show_mine'] == 1) ? 0 : 1;
			$sql_ary['user_digest_show_pms'] 			= $this->config['phpbbservices_digests_user_digest_show_pms'];
			$sql_ary['user_digest_sortby'] 				= $this->config['phpbbservices_digests_user_digest_sortby'];
			$sql_ary['user_digest_toc'] 				= $this->config['phpbbservices_digests_user_digest_toc'];
			$sql_ary['user_digest_type'] 				= $this->config['phpbbservices_digests_user_digest_type'];
			$event['sql_ary'] = $sql_ary;
		}

and should be:

Code: Select all

		$is_human = ($event['sql_ary']['user_type'] == USER_IGNORE) ? false : true;
		if ($is_human && ($this->config['phpbbservices_digests_enable_auto_subscriptions'] == 1 || $subscribe_on_registration))
		{
			$sql_ary['user_digest_attachments'] 		= $this->config['phpbbservices_digests_user_digest_attachments'];
			$sql_ary['user_digest_block_images'] 		= $this->config['phpbbservices_digests_user_digest_block_images'];
			$sql_ary['user_digest_filter_type'] 		= $this->config['phpbbservices_digests_user_digest_filter_type'];
			$sql_ary['user_digest_format'] 				= $this->config['phpbbservices_digests_user_digest_format'];
			$sql_ary['user_digest_max_display_words'] 	= $this->config['phpbbservices_digests_user_digest_max_display_words'];
			$sql_ary['user_digest_max_posts'] 			= $this->config['phpbbservices_digests_user_digest_max_posts'];
			$sql_ary['user_digest_min_words'] 			= $this->config['phpbbservices_digests_user_digest_min_words'];
			$sql_ary['user_digest_new_posts_only'] 		= $this->config['phpbbservices_digests_user_digest_new_posts_only'];
			$sql_ary['user_digest_no_post_text'] 		= $this->config['phpbbservices_digests_user_digest_no_post_text'];
			$sql_ary['user_digest_pm_mark_read'] 		= $this->config['phpbbservices_digests_user_digest_pm_mark_read'];
			$sql_ary['user_digest_remove_foes'] 		= $this->config['phpbbservices_digests_user_digest_remove_foes'];
			$sql_ary['user_digest_reset_lastvisit'] 	= $this->config['phpbbservices_digests_user_digest_reset_lastvisit'];
			$sql_ary['user_digest_send_hour_gmt'] 		= ($this->config['phpbbservices_digests_user_digest_send_hour_gmt'] == -1) ? rand(0,23) : $this->config['phpbbservices_digests_user_digest_send_hour_gmt'];
			$sql_ary['user_digest_send_on_no_posts'] 	= $this->config['phpbbservices_digests_user_digest_send_on_no_posts'];
			$sql_ary['user_digest_show_mine'] 			= ($this->config['phpbbservices_digests_user_digest_show_mine'] == 1) ? 0 : 1;
			$sql_ary['user_digest_show_pms'] 			= $this->config['phpbbservices_digests_user_digest_show_pms'];
			$sql_ary['user_digest_sortby'] 				= $this->config['phpbbservices_digests_user_digest_sortby'];
			$sql_ary['user_digest_toc'] 				= $this->config['phpbbservices_digests_user_digest_toc'];
			$sql_ary['user_digest_type'] 				= $this->config['phpbbservices_digests_user_digest_type'];
			$event['sql_ary'] = array_merge($event['sql_ary'], $sql_ary);
		}

This will be fixed in the next version.

[MarkDHamill]

@MarkDHamill thank you for your quick support!

Today I detected another issue that is linked to the s9e/mediaembeded extension.
If a post contains links embedded by this extension, for example a youtube video, than it renders it within an iframe. Iframes are in emails often considered as dangerous or spammy, so the digest mail which contains it, will not be delivered.
So I suggest a remove of any <iframe>…</iframe> in html mails.

While I appreciate the suggestion, it is not practical for me to tailor the extension to optimally accommodate other extensions. Digests sometimes do end up in a spam folder regardless. The best solution to this (assuming GMail) is probably to add the sending email address as a contact or create a filter not to send these to spam.

[whocarez]

@MarkDHamill thank you for your quick support!

Today I detected another issue that is linked to the s9e/mediaembeded extension.
If a post contains links embedded by this extension, for example a youtube video, than it renders it within an iframe. Iframes are in emails often considered as dangerous or spammy, so the digest mail which contains it, will not be delivered.
So I suggest a remove of any <iframe>…</iframe> in html mails.

While I appreciate the suggestion, it is not practical for me to tailor the extension to optimally accommodate other extensions. Digests sometimes do end up in a spam folder regardless. The best solution to this (assuming GMail) is probably to add the sending email address as a contact or create a filter not to send these to spam.

The problem here is, that these messages are already rejected before delivering with error code 550. For example one of the biggest providers in Germany t-online does body checks like that. And when a requested a reason, then they told me, that it was because of this iframe. The same for one university here.

Code: Select all

 host mx03.t-online.de[194.25.134.73] said: 550-5.7.0
    Message considered as spam or virus, rejected 550-5.7.0

Code: Select all

host mail.rz.uni-passau.de[132.231.51.4] said:
    550 dangerous.iframe id=151291::1487413554-00002A17-7CE8291A/7/0 (in reply
    to end of DATA command)

So unfortunately I see no other way than to remove <iframe>…</iframe> in html mails.