Update the notification email system to be RFC2369 and RFC8058 compliant

https://www.phpbb.com/ideas/
Post Reply
Author:
v12mike
Posted:
Fri Jul 12, 2019 5:53 pm
Rating:
Status:
New
Ticket:
PHPBB3-16099
v12mike
Registered User
Posts: 341
Joined: Thu Jul 09, 2015 5:03 pm

Update the notification email system to be RFC2369 and RFC8058 compliant

Post by v12mike » Fri Jul 12, 2019 5:53 pm

I have recently had a significant problem with Microsoft mail services (hotmail.com, outlook.com, live.com, btinternet.com and others) not accepting emails from our phpBB server. Microsoft have told me it was due to too many users marking our emails as junk. It happens that the particular host that was blocked sends ONLY phpBB notifications, and no other email.

As part of the process of getting the block removed, I had to put hand on heart and say that our server complies with all of the Microsoft email sender policies, which as of phpBB 3.2.7, it does not.

I am thinking of making some improvements to the phpBB notification email system, broadly to achieve the goals:
1. Implement improvements to the notification email system to meet the requirements imposed by major email providers such as Google and Microsoft
2. Full compliance with RFC2369 and RFC8058
3. Implement (optional) handling of bounced emails, with automatic suspension of email notifications to affected users.

There are some other improvements that would be nice to have, but not essential in this context:
4. Allow easier customisation of notification emails by storing the templates (corresponding to installed language packs) in the database and providing a UI for editing them.
5. Allow board admins to view a user's subscriptions and unsubscribe them where necessary.
6. Improve the ability to configure sender and reply-to email addresses for a phpBB board.

RFC2369 is the standard for URLs used for the control of mailing lists. phpBB is currently partially compliant.
RFC8058 is the standard for 1-click unsubscription from mailing lists, phpBB is currently not compliant at all.

In order to meet the applicable parts of RFC2369 and RFC88058, the main new requirements are:
  • A new RFC8058 email header.
  • Email headers protected by a DKIM signature (done in the smtp host not phpBB , but relevant).
  • The phpBB board to provide an unsubscribe URL that for http:// GET requests takes the user to a page where they can easily confirm the they wish to unsubscribe, but an http:// PUSH to the same url results in a 1-click unsubscription without user confirmation.
  • The phpBB board needs to honour unsubscribe requests without requiring a user to log-in (or be logged-in) to the forum, this precludes using the existing UCP pages which require the user to be logged-in.
  • The unsubscribe links must be sufficiently protected to prevent malicious use of the unsubscribe system.
Also...
  • Microsoft have a requirement that a server should stop sending emails to an email address after multiple delivery failures. This requires phpBB to receive email bounce notifications, but it is not difficult to configure an external service to convert unsubscribe emails to http:// requests for this purpose.
  • In addition, some Microsoft email services do not support 1-click unsubscribe via http://, they only support it via //, but as above that is not too difficult to have an external service convert the transport mechanism
I have prototyped a phpBB extension to implement some of the above (mainly RFC2369 and RFC 8058 compliance), but I am not happy with it as an extension, as I don't think that RFC compliance should be an optional feature, and the current notification email template system does not really have any mechanism that allow extensions to interact with email templates, which makes the extension really ugly. So I am proposing that some or all of the above are implemented as enhancements to core phpBB.

I have raised PHPBB3-16099 for this

User avatar
AmigoJack
Registered User
Posts: 5588
Joined: Tue Jun 15, 2010 11:33 am
Location: グリーン ヒル ゾーン
Contact:

Re: Update the notification email system to be RFC2369 and RFC8058 compliant

Post by AmigoJack » Mon Jul 15, 2019 10:09 am

v12mike wrote:
Fri Jul 12, 2019 5:53 pm
meet the requirements imposed by major email providers such as Google and Microsoft
Why not listing and linking them? Since these requirements change over time relative information about them is a bad choice.

v12mike wrote:
Fri Jul 12, 2019 5:53 pm
3. Implement (optional) handling of bounced emails
That's easily said but difficult to achieve: thru the years I've seen many bounces and they can differ a lot, and also carry multiple recipients at once. Extracting the bounce reason isn't trivial.

v12mike wrote:
Fri Jul 12, 2019 5:53 pm
with automatic suspension of email notifications to affected users
Too strict: there are cases where the user himself has done nothing wrong, i.e. his e-mail provider rejecting board e-mails because of (wrongly managed) blacklist(s), or when an MX is temporarily not available. Yet he should be punished by messing with his subscriptions?

And I also learnt users who can be so stupid to mark e-mails as spam when in fact they only want no more notifications, they're also so stupid to not notice there are not coming any more e-mails. A better action would be to put users in a group that isolates them so they're forced to contact the board staff (preferably not via e-mail) to then discuss further steps. Or to provide a global "mute" mode to disable all notifications (instead of removing them). There are still people out there not knowing that an e-mail inbox can be full.

v12mike wrote:
Fri Jul 12, 2019 5:53 pm
4. Allow easier customisation of notification emails by storing the templates (corresponding to installed language packs) in the database and providing a UI for editing them
Speak for yourself: to me editing files is more easily and less error prone than doing it thru the board software itself. Plus: how often are you in the need to change them at all?

v12mike wrote:
Fri Jul 12, 2019 5:53 pm
5. Allow board admins to view a user's subscriptions and unsubscribe them where necessary.
I understand the intention, but this is also disconcerning for privacy reasons.

v12mike wrote:
Fri Jul 12, 2019 5:53 pm
  • Email headers protected by a DKIM signature
What about SPF? And DMARC? That question is rhetorical. If it's one of the e-mail provider requirements: list it. But it's also irrelevant for phpBB.

v12mike wrote:
Fri Jul 12, 2019 5:53 pm
  • The phpBB board needs to honour unsubscribe requests without requiring a user to log-in
  • The unsubscribe links must be sufficiently protected to prevent malicious use of the unsubscribe system.
This is mutually exclusive: without authentication anyone could request the given URI and the board has no chance to find out who did it. If I get my hands on your notification e-mail from 2 months ago then I can either unsubscribe you easily or the URI invalidates for you on a given criterium (i.e. time passed), too.

v12mike wrote:
Fri Jul 12, 2019 5:53 pm
  • Microsoft have a requirement that a server should stop sending emails to an email address after multiple delivery failures.
Without differentiating between the failure reasons this is idiotic, as then nobody knows when temporary problems are gone.

v12mike wrote:
Fri Jul 12, 2019 5:53 pm
  • In addition, some Microsoft email services do not support 1-click unsubscribe via http://, they only support it via //
Can you elaborate? While you surely mean HTTP with "http://" I have no idea what you mean with "via //".


On 3.0 I did
  • include the List-Unsubscribe header with the appropriate URI (unwatching a topic, going to notification setting for PMs and closed reports...).
  • include the Precedence header (value bulk), to indicate that the e-mail is not that important (wouldn't do that today anymore, since the usage is discouraged as per RFC 2076)
  • include the Auto-Submitted header (value auto-generated), to indicate that no human wrote the e-mail (although RFC 3834 indicates this might undermine getting bounce reports)
  • did not include the headers X-Priority, X-MSMail-Priority and X-MimeOLE when the priority was "normal", as in that case no header to indicate that is needed anyway.
I had the impression it helped a bit, but can't prove it in any way.

Cross-linking Automated bounced email handler
The worst thing about censorship is ███████████
Affin wrote:
Tue Nov 20, 2018 9:51 am
The problem is probably not my English but you do not want to understand correctly.
...
We will not come anybody anyway, nevertheless, it's best to shit this.

v12mike
Registered User
Posts: 341
Joined: Thu Jul 09, 2015 5:03 pm

Re: Update the notification email system to be RFC2369 and RFC8058 compliant

Post by v12mike » Mon Jul 15, 2019 11:38 am

AmigoJack wrote:
Mon Jul 15, 2019 10:09 am
v12mike wrote:
Fri Jul 12, 2019 5:53 pm
meet the requirements imposed by major email providers such as Google and Microsoft
Why not listing and linking them? Since these requirements change over time relative information about them is a bad choice.
I could list them, but as there are several pages of such requirements, this did not seem to be the appropriate place. Does it affect the overall desirability of this idea?
AmigoJack wrote:
Mon Jul 15, 2019 10:09 am
v12mike wrote:
Fri Jul 12, 2019 5:53 pm
3. Implement (optional) handling of bounced emails
That's easily said but difficult to achieve: thru the years I've seen many bounces and they can differ a lot, and also carry multiple recipients at once. Extracting the bounce reason isn't trivial.
I know that it is not trivial, but it is possible. RFC3464 defines a standard format which is followed by all of the major email providers
AmigoJack wrote:
Mon Jul 15, 2019 10:09 am
v12mike wrote:
Fri Jul 12, 2019 5:53 pm
with automatic suspension of email notifications to affected users
Too strict: there are cases where the user himself has done nothing wrong, i.e. his e-mail provider rejecting board e-mails because of (wrongly managed) blacklist(s), or when an MX is temporarily not available. Yet he should be punished by messing with his subscriptions?

And I also learnt users who can be so stupid to mark e-mails as spam when in fact they only want no more notifications, they're also so stupid to not notice there are not coming any more e-mails. A better action would be to put users in a group that isolates them so they're forced to contact the board staff (preferably not via e-mail) to then discuss further steps. Or to provide a global "mute" mode to disable all notifications (instead of removing them). There are still people out there not knowing that an e-mail inbox can be full.
In the event of a user's email notifications being suspended, I would probably send an automatic board notification to the user advising them of that, and the reason. Suspension of email delivery following rejection is a mandatory part of the Microsoft rules, but the reception of rejection emails by phpBB would be inherently optional, as it requires the configuration of an external mail forwarding agent.
AmigoJack wrote:
Mon Jul 15, 2019 10:09 am
v12mike wrote:
Fri Jul 12, 2019 5:53 pm
4. Allow easier customisation of notification emails by storing the templates (corresponding to installed language packs) in the database and providing a UI for editing them
Speak for yourself: to me editing files is more easily and less error prone than doing it thru the board software itself. Plus: how often are you in the need to change them at all?
I now think that it would be better to handle this as a separate idea, so I won't comment more here.
AmigoJack wrote:
Mon Jul 15, 2019 10:09 am
v12mike wrote:
Fri Jul 12, 2019 5:53 pm
5. Allow board admins to view a user's subscriptions and unsubscribe them where necessary.
I understand the intention, but this is also disconcerning for privacy reasons.
We recently had a new user (I think the trigger for my recent issues with Microsoft) who managed to subscribe to our busiest forums but could not find the buttons to unsubscribe, so started moving notification emails repeated to junk, then after several days he complained to the board admins, but could not follow instructions, so in the end I had to manually edit the database with SQL to remove his subscriptions. I would rather have done it via the ACP. I don't view subscription information as being confidential from a board admin.
AmigoJack wrote:
Mon Jul 15, 2019 10:09 am
v12mike wrote:
Fri Jul 12, 2019 5:53 pm
  • Email headers protected by a DKIM signature
What about SPF? And DMARC? That question is rhetorical. If it's one of the e-mail provider requirements: list it. But it's also irrelevant for phpBB.
It is relevant here because it is a mandatory but easily overlooked requirement of RFC8058.
AmigoJack wrote:
Mon Jul 15, 2019 10:09 am
v12mike wrote:
Fri Jul 12, 2019 5:53 pm
  • The phpBB board needs to honour unsubscribe requests without requiring a user to log-in
  • The unsubscribe links must be sufficiently protected to prevent malicious use of the unsubscribe system.
This is mutually exclusive: without authentication anyone could request the given URI and the board has no chance to find out who did it. If I get my hands on your notification e-mail from 2 months ago then I can either unsubscribe you easily or the URI invalidates for you on a given criterium (i.e. time passed), too.
That is true, but it is a requirement of the RFC and a Google and Microsoft requirement I think. The feature, as I have prototyped it, has a timeout of a few days on the unsubscribe urls with hash protection so that they cannot be easily edited to be applicable to another user.
AmigoJack wrote:
Mon Jul 15, 2019 10:09 am
v12mike wrote:
Fri Jul 12, 2019 5:53 pm
  • Microsoft have a requirement that a server should stop sending emails to an email address after multiple delivery failures.
Without differentiating between the failure reasons this is idiotic, as then nobody knows when temporary problems are gone.
There are conventions for that... In the case of deliveryt rate exceeded, it is usual to recommence email deliveries after 24 hours. I have not decided whether it is better to queue or discard phpBB notifications during that time.
AmigoJack wrote:
Mon Jul 15, 2019 10:09 am
v12mike wrote:
Fri Jul 12, 2019 5:53 pm
  • In addition, some Microsoft email services do not support 1-click unsubscribe via http://, they only support it via //
Can you elaborate? While you surely mean HTTP with "http://" I have no idea what you mean with "via //".
Sorry, that was a typo. It should have said that some Microsoft delivery services currently only support mailto:// unsubscriptions (i.e. via smtp, not http protocol).

On 3.0 I did
  • include the List-Unsubscribe header with the appropriate URI (unwatching a topic, going to notification setting for PMs and closed reports...).
  • include the Precedence header (value bulk), to indicate that the e-mail is not that important (wouldn't do that today anymore, since the usage is discouraged as per RFC 2076)
  • include the Auto-Submitted header (value auto-generated), to indicate that no human wrote the e-mail (although RFC 3834 indicates this might undermine getting bounce reports)
  • did not include the headers X-Priority, X-MSMail-Priority and X-MimeOLE when the priority was "normal", as in that case no header to indicate that is needed anyway.
I had the impression it helped a bit, but can't prove it in any way.

Cross-linking Automated bounced email handler
[/quote]

Post Reply

Return to “phpBB Ideas”