[RC] Digests 3.2.10

A place for Extension Authors to post and receive feedback on Extensions still in development. No Extensions within this forum should be used within a live environment!
Scam Warning
Forum rules
READ: phpBB.com Board-Wide Rules and Regulations

IMPORTANT: Extensions Development rules

IMPORTANT FOR NEEDED EVENTS!!!
If you need an event for your extension please read this for the steps to follow to request the event(s)
User avatar
MarkDHamill
Registered User
Posts: 3507
Joined: Fri Aug 02, 2002 12:36 am
Location: Florence, MA USA
Contact:

Re: [RC] Digests 3.2.7

Post by MarkDHamill » Sat Jul 28, 2018 7:28 pm

When you update the digests extension, you overwrite all the files.

Not sure what the solution here is, but one thing you can check is to look at your board language and if it is English change it to German. This is because the language variable is written in the HTML version of the digest:

Code: Select all

<html lang="{{ S_USER_LANG }}">
Let me know if that solves the problem.
Get the latest versions of my Digests and Smartfeed mods and extensions.
Need phpBB services or a phpBB consultant? I offer most phpBB services.

whocarez
Registered User
Posts: 139
Joined: Sat Apr 18, 2009 9:19 pm

Re: [RC] Digests 3.2.7

Post by whocarez » Sun Jul 29, 2018 10:39 am

Hi, this wasn't the solution. But I figured out, that post_text gets somehow a doubled encoding.
So in any case this is the header:

Code: Select all

<!DOCTYPE html>
<html lang="de_x_sie">
<head>
	<meta charset="utf-8">
this is not correct, because de_x_sie (German (Formal Honorifics)) means only something for PHPBB, but this is an ignorable mistake. More important is the charset and this is by default utf-8. Switching the board language shouldn't be the solution, because we have also Russian and Ukrainian as board languages and sometimes texts with Cyrillic characters.

'ü' is somehow encoded in 'post_text' like that: &Atilde;&frac14;. So I suggest, that somewhere 'post_text' lacks a html_entity_decode or there is a double encoding via htmlentities, but I couldn't find it.
And to be sure: all the titles are correct shown.

This digest mail is a standard html mail. Images are not embedded and "iframe,video" are excluded tags. Maybe the last function is causing this error?

If you want, I can provide you with two example mails. One with correct displaying of all these Umlauts in titles and text and one with displaying Umlauts correctly only in titles.

User avatar
MarkDHamill
Registered User
Posts: 3507
Joined: Fri Aug 02, 2002 12:36 am
Location: Florence, MA USA
Contact:

Re: [RC] Digests 3.2.7

Post by MarkDHamill » Sun Jul 29, 2018 12:21 pm

Well, that was worth a shot.

It's most likely something to do with html_entity_decode(). There is an encoding argument that can be used. You can try ISO-8859-1, ISO-8859-5, ISO-8859-15, cp1251 and cp1252 for this parameter. Probably best to try this approach first.
Get the latest versions of my Digests and Smartfeed mods and extensions.
Need phpBB services or a phpBB consultant? I offer most phpBB services.

whocarez
Registered User
Posts: 139
Joined: Sat Apr 18, 2009 9:19 pm

Re: [RC] Digests 3.2.7

Post by whocarez » Sun Jul 29, 2018 12:41 pm

Hm, maybe I don't understand it correctly, but post_text and post_subject are saved in the database in Unicode and we want the digest to be send in utf-8, what means this is Unicode. So, as far as I understand it, there is no transformation necessary. Cyrillic and Umlauts are saved as they are and can be put in this HTML mail without problems. If necessary the php mailer makes a mime message of it and that's it, isn't it?

If I would define one charset for html_entity_decode - for example iso8859-15, that would destroy other texts and subjects with Cyrillic letters, so that can't be a solution for the whole situation or did I get something wrong?

User avatar
MarkDHamill
Registered User
Posts: 3507
Joined: Fri Aug 02, 2002 12:36 am
Location: Florence, MA USA
Contact:

Re: [RC] Digests 3.2.7

Post by MarkDHamill » Sun Jul 29, 2018 2:38 pm

At this point I was just suggesting trying a few things. Yes, UTF-8 is stored in the database. post_subject though has a utf8_unicode_ci collation but that should make no difference. Something is tripping it up but since you are the first user to report this problem it's probably something kind of odd. I assume this is not a text digest because it uses strip_tags().

Outside of digests, I have noticed issues like this working with clients with forums that were originally in phpBB2 using ISO-8859-1 encoding. Forcing Windows characters with meta tags often solved the problem, although the real issue was that many columns were never really converted to UTF-8.

It may be that some sort of additional HTTP header is needed.
Get the latest versions of my Digests and Smartfeed mods and extensions.
Need phpBB services or a phpBB consultant? I offer most phpBB services.

whocarez
Registered User
Posts: 139
Joined: Sat Apr 18, 2009 9:19 pm

Re: [RC] Digests 3.2.7

Post by whocarez » Sun Jul 29, 2018 3:21 pm

Yes it is a html digest. The database tables are utf8_bin and they were never something in a ISO or Windows charset, because this board used from the beginning both Cyrillic letters and German Umlauts, so utf8 was the only option.
In phpbb_posts post_text and post_subject are saved in utf8_unicode_ci.

I tried to change the html language to something like "de", but this makes no difference, because in the same mail the none ASCII characters are encoded in two different ways for titles and the post_text.

User avatar
MarkDHamill
Registered User
Posts: 3507
Joined: Fri Aug 02, 2002 12:36 am
Location: Florence, MA USA
Contact:

Re: [RC] Digests 3.2.7

Post by MarkDHamill » Sun Jul 29, 2018 5:04 pm

On my development forum, a vanilla installation of phpBB, the collation for the post_text column is utf8_bin.
Get the latest versions of my Digests and Smartfeed mods and extensions.
Need phpBB services or a phpBB consultant? I offer most phpBB services.

whocarez
Registered User
Posts: 139
Joined: Sat Apr 18, 2009 9:19 pm

Re: [RC] Digests 3.2.7

Post by whocarez » Tue Jul 31, 2018 10:25 am

Sorry for my late response, but I had only today time to make some test mails. So, I could determine, that excluding html tags is causing this error and it doesn't matter which tags are defined in options.

With stripping some html tags I get something like this:

Code: Select all

<div class="content"><!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN" "http://www.w3.org/TR/REC-html40/loose.dtd">
<html><body><blockquote class="uncited"><div>Heute l&Atilde;&curren;ngst aus der Mode, geh&Atilde;&para;rte in der Sowjetunion auf jedes Bett eine Tagesdecke. In den M&Atilde;&frac14;ll m&Atilde;&frac14;ssen die Decken trotzdem nicht: Sie eigenen sich hervorragend f&Atilde;&frac14;r den Strand. </div></blockquote></body></html>
</div>
without it looks like this:

Code: Select all

<div class="content"><blockquote class="uncited"><div>Heute längst aus der Mode, gehörte in der Sowjetunion auf jedes Bett eine Tagesdecke. In den Müll müssen die Decken trotzdem nicht: Sie eigenen sich hervorragend für den Strand. </div></blockquote></div>
											</div>
So not the update to 3.2.7 caused this bug, but the use of this "strip html tags"-option, which you by chance integrated more or less on my request :-).

So, I made two changes in cron/task/digests.php. First to get rid of DOCTYPE and the other html wrapper I followed this:
https://stackoverflow.com/questions/487 ... ml-wrapper

and made these changes:

Code: Select all

//							$dom->loadHTML($post_text);
							$dom->loadHTML($post_text, LIBXML_HTML_NOIMPLIED | LIBXML_HTML_NODEFDTD);
Following this suggestions to get utf-8 correctly formated:
https://stackoverflow.com/questions/821 ... 0_20675396

I made the following:

Code: Select all

//							$post_text = $dom->saveHTML();
							$post_text = utf8_decode($dom->saveHTML($dom->documentElement));


After these changes everything seems to work!

User avatar
MarkDHamill
Registered User
Posts: 3507
Joined: Fri Aug 02, 2002 12:36 am
Location: Florence, MA USA
Contact:

Re: [RC] Digests 3.2.7

Post by MarkDHamill » Tue Jul 31, 2018 12:50 pm

Excellent troubleshooting! Some of these options are very obscure. I've made the patches to my code and they will show up in the next version. As is my practice, I have given you credit in the inline comments. Good work!
Get the latest versions of my Digests and Smartfeed mods and extensions.
Need phpBB services or a phpBB consultant? I offer most phpBB services.

User avatar
MarkDHamill
Registered User
Posts: 3507
Joined: Fri Aug 02, 2002 12:36 am
Location: Florence, MA USA
Contact:

Digests 3.2.8 released

Post by MarkDHamill » Thu Aug 09, 2018 8:44 pm

The extension is still not approved yet, but there is a new version 3.2.8. It fixes one bug I caught about the same time the extension review team found it that caused digests to fail when a digest was requested to include a table of contents.

Also fixed the Unicode bug found by whocarez. Great analysis! When examining this code I realized it could be made more efficient, so now it loads the post text into a document tree only once.

I removed fancy language file changes that used sprintf() to embed certain dynamic information, mostly out of laziness. I'm sure it can be done the way the extension review team wants but it was more of a challenge than I wanted to take on considering I was using the $display_vars approach to showing fields and help information in the ACP. Laziness and some exasperation with bugs in the migrator in phpBB 3.2.2 also prompted me to remove code that used to work that removed modules for digests left over in the database if the digests mod was installed previously. These modules should be manually removed before installing the extensions using instructions in the README.md to avoid a likely installation error.

You can see a full list of changes since version 3.2.7 here.

See the first post for a download link.
Last edited by MarkDHamill on Thu Aug 09, 2018 10:07 pm, edited 1 time in total.
Get the latest versions of my Digests and Smartfeed mods and extensions.
Need phpBB services or a phpBB consultant? I offer most phpBB services.

User avatar
3Di
Registered User
Posts: 12992
Joined: Mon Apr 04, 2005 11:09 pm
Location: Milan (IT) Frankfurt (DE)
Name: Marco
Contact:

Re: [RC] Digests 3.2.8

Post by 3Di » Thu Aug 09, 2018 10:05 pm

MarkDHamill wrote:
Thu Aug 09, 2018 8:44 pm
These modules should be manually removed before installing the extensions using instructions in the README.md to avoid a likely installation error
At the end of all, good decision IMHO. Usually who upgrades gets rid of anything via STK as per the KB article.
MarkDHamill wrote:
Thu Aug 09, 2018 8:44 pm
You can see a full list of changes since version 3.2.7 here.
May I suggest the right link?
https://github.com/MarkDHamill/digests/ ... .7...3.2.8
Want to compensate me for my interest? Donate
Please PM me only to request paid works. Thx.
Extensions, Scripts, MOD porting, Update/Upgrades
My development's activity º PhpStorm's proud user

User avatar
MarkDHamill
Registered User
Posts: 3507
Joined: Fri Aug 02, 2002 12:36 am
Location: Florence, MA USA
Contact:

Re: [RC] Digests 3.2.8

Post by MarkDHamill » Thu Aug 09, 2018 10:07 pm

Thanks. Corrected the link.
Get the latest versions of my Digests and Smartfeed mods and extensions.
Need phpBB services or a phpBB consultant? I offer most phpBB services.

kamaleon
Registered User
Posts: 378
Joined: Tue Sep 09, 2014 9:47 am

Re: [RC] Digests 3.2.8

Post by kamaleon » Sun Sep 16, 2018 7:28 pm

Nice extension, but how does it do the mailing? With the server mail?

User avatar
MarkDHamill
Registered User
Posts: 3507
Joined: Fri Aug 02, 2002 12:36 am
Location: Florence, MA USA
Contact:

Re: [RC] Digests 3.2.8

Post by MarkDHamill » Mon Sep 17, 2018 1:31 am

It hands off the mailing to phpBB's mail library, documented in /includes/functions_messenger.php.
Get the latest versions of my Digests and Smartfeed mods and extensions.
Need phpBB services or a phpBB consultant? I offer most phpBB services.

User avatar
jan_2017
Registered User
Posts: 143
Joined: Sun May 07, 2017 3:39 pm

Re: [RC] Digests 3.2.8

Post by jan_2017 » Mon Sep 17, 2018 5:36 pm

Can I change the appearance of the email content?

Bye Jan

Post Reply

Return to “Extensions in Development”

Who is online

Users browsing this forum: Exabot [Bot] and 22 guests