Text only | Text with Images

ElkArte Community

Project Support => Support => Topic started by: Frenzie on March 22, 2022, 07:42:47 am

Title: Gmail: multiple addresses in From:n550 5.7.1 header are not accepted
Post by: Frenzie on March 22, 2022, 07:42:47 am
I'm not really sure what to make of this, but Gmail seems to be refusing emails. Does anybody know what's up with that? I've got all of the DKIM/SPF/DMARC stuff set up.

Code: [Select]
Messages with multiple addresses in From:n550 5.7.1 header are not accepted.

https://serverfault.com/questions/841465/rejected-emails-messages-with-multiple-addresses-in-from-header-are-not-accept
Title: Re: Gmail: multiple addresses in From:n550 5.7.1 header are not accepted
Post by: tino on March 22, 2022, 08:49:17 am
The easy fix is to only set one email address in the from header. That should be the standard anyway looking at the RFC.
Title: Re: Gmail: multiple addresses in From:n550 5.7.1 header are not accepted
Post by: Frenzie on March 22, 2022, 12:25:31 pm
Let me try to partially rephrase: Elkarte emails are being bounced and Wordpress emails are not. Both are using PHP to send emails. How can I tell Elkarte to set only one email address in the from header? There is no such setting in Maintenance → Mail → Settings.

The settings search does indeed hint at the existence of a "Reply-To and From email address [Setting] " but ?action=admin;area=maillist;sa=emailsettings just opens the main admin page. I suspect it's related to Post by Email Management.

Edit: there's also $webmaster_email in Settings.php as well as Configuration → General → Webmaster email address. This points to an email address on the domain.
Title: Re: Gmail: multiple addresses in From:n550 5.7.1 header are not accepted
Post by: Spuds on March 22, 2022, 04:08:30 pm
Could you post a raw .eml file of what is being sent so I can take a look ?
Title: Re: Gmail: multiple addresses in From:n550 5.7.1 header are not accepted
Post by: Spuds on March 22, 2022, 04:11:20 pm
Also any chance you can try SMTP as well and see if the error repeats with that transport?
Title: Re: Gmail: multiple addresses in From:n550 5.7.1 header are not accepted
Post by: Frenzie on March 23, 2022, 03:29:46 am
I don't know what it looks like to Google of course, but:
Code: [Select]
Return-Path: <thefromemail@dndsanctuary.eu>
Received: from www354.your-server.de
	by www354.your-server.de with LMTP
	id sPIUL87KOmKkIAAA/Nz4eQ
	(envelope-from <thefromemail@dndsanctuary.eu>); Wed, 23 Mar 2022 08:22:54 +0100
Envelope-to: justforsometesting@dndsanctuary.eu
Delivery-date: Wed, 23 Mar 2022 08:22:54 +0100
Received: from localhost ([127.0.0.1] helo=www354.your-server.de)
	by www354.your-server.de with esmtps (TLSv1.3:TLS_AES_256_GCM_SHA384:256)
	(Exim 4.92.3)
	(envelope-from <thefromemail@dndsanctuary.eu>)
	id 1nWvKY-0002VL-Oy
	for justforsometesting@dndsanctuary.eu; Wed, 23 Mar 2022 08:22:54 +0100
To: justforsometesting@dndsanctuary.eu
Subject: Welcome to The DnD Sanctuary
From: "The DnD Sanctuary" <thefromemail@dndsanctuary.eu>
 Return-Path: thefromemail@dndsanctuary.eu
 Date: Wed, 23 Mar 2022 07:22:54 -0000
 X-Mailer: ELK
 Mime-Version: 1.0
 Content-Type: multipart/alternative; boundary="ELK-83f0fe058f0cf03cf18bffd39d24f144"
 Content-Transfer-Encoding: 7bit
 Message-ID: <b9d5348f83cc8893f56575710c4291cb-@dndsanctuary.eu>
Message-Id: <E1nWvKY-0002VG-N5@www354.your-server.de>
Date: Wed, 23 Mar 2022 08:22:54 +0100

I'll try SMTP later.
Title: Re: Gmail: multiple addresses in From:n550 5.7.1 header are not accepted
Post by: Frenzie on March 23, 2022, 03:47:19 am
Hm, so in Vivaldi it just shows

QuoteFrom: The DnD Sanctuary <etc>

But in Thunderbird it says:

QuoteFrom: thefromemail@dndsanctuary.eu et al

Clicking on "2 more" shows the other supposed from addresses are:
Code: [Select]
multipart/alternative <>

And:
Code: [Select]
the-message-id-hash@dndsanctuary.eu

It's likely Gmail parses it the same way, but then decides to reject the message.


Edit: the one from Wordpress looks like this:

Code: [Select]
…@gmail.com; Mon, 21 Mar 2022 20:11:30 +0100
To: …@gmail.com
Subject: Site nonsense
Date: Mon, 21 Mar 2022 19:11:30 +0000
From: WordPress <…@…>
Message-ID: <3gsjHp42Em8qSFqoTD8HHjC7ltZYffugefPCV0f8tE@fransdejonge.com>
X-Mailer: PHPMailer 6.5.3 (https://github.com/PHPMailer/PHPMailer)
MIME-Version: 1.0
Content-Type: text/plain; charset=UTF-8

I don't know the email header spec but that indentation in the email from Elkarte looks suspicious?

Edit 2:

In any case, running it through a header parser shows this is indeed the entire from field:

Code: [Select]
"The DnD Sanctuary" <thefromemail@dndsanctuary.eu> Return-Path: thefromemail@dndsanctuary.eu Date: Wed, 23 Mar 2022 07:22:54 -0000 X-Mailer: ELK Mime-Version: 1.0 Content-Type: multipart/alternative; boundary="ELK-83f0fe058f0cf03cf18bffd39d24f144" Content-Transfer-Encoding: 7bit Message-ID: <b9d5348f83cc8893f56575710c4291cb-@dndsanctuary.eu>

Removing the single-space indentation seems to result in correct parsing.
Title: Re: Gmail: multiple addresses in From:n550 5.7.1 header are not accepted
Post by: Spuds on March 23, 2022, 02:08:33 pm
If you would be so kind as to try something quick, this is a bug I found when I redid the entire mail sending function in 2.0.  I have this change slated for 1.1.9 as well.

In your Mail.subs.php file
Code: (find) [Select]
	// Return path, date, mailer
	$headers .= 'Return-Path: ' . $return_path . $line_break;

Code: (replace) [Select]
	// Return path, date, mailer
	if ($use_sendmail)
		$headers .= 'Return-Path: <' . $return_path . '>' . $line_break;

Lets see if that fixes the issue for you.
Title: Re: Gmail: multiple addresses in From:n550 5.7.1 header are not accepted
Post by: Frenzie on March 23, 2022, 04:06:30 pm
Same bounce error from Gmail I'm afraid. Glancing at the code there it's a complete mystery to me where that single-space indentation could be coming from.
Title: Re: Gmail: multiple addresses in From:n550 5.7.1 header are not accepted
Post by: Frenzie on March 23, 2022, 04:12:05 pm
@Spuds It works when you always use \r\n

Code: (find) [Select]
$line_break = detectServer()->is('windows') || !$use_sendmail ? "\r\n" : "\n";

Code: (replace) [Select]
$line_break = "\r\n";

In other words, this comment right above appears to be false:
Code: [Select]
// Line breaks need to be \r\n only in windows or for SMTP.
Title: Re: Gmail: multiple addresses in From:n550 5.7.1 header are not accepted
Post by: Frenzie on March 23, 2022, 04:30:58 pm
It looks like the current behavior is plainly incorrect:
Quote from: https://www.php.net/manual/en/function.mail.phpMultiple extra headers should be separated with a CRLF (\r\n).

But there is this note:
QuoteNote:

If messages are not received, try using a LF (\n) only. Some Unix mail transfer agents (most notably » qmail) replace LF by CRLF automatically (which leads to doubling CR if CRLF is used). This should be a last resort, as it does not comply with » RFC 2822.
Title: Re: Gmail: multiple addresses in From:n550 5.7.1 header are not accepted
Post by: Spuds on March 23, 2022, 05:14:49 pm
Good detective work :D

I do remember that now ... I came across that when I was redoing things for 2.0 and did the same lookup on php.net and thought that was strange.

Then as I did some more digging, it seems that PHP mail() talks to the native sendmail command of the OS, which should expect line endings to OS native line endings  ... LF (\n) on Unix/Linux and CRLF (\r\n) on Windows. 
SMTP always wants CRLF but that is a lot different than native sendmail. 
So what I'm saying is I feel the PHP net documentation may be misleading, other than maybe the qmail stuff?

Its a bit of a mystery, are you running on *nix or *doz ?
Title: Re: Gmail: multiple addresses in From:n550 5.7.1 header are not accepted
Post by: Frenzie on March 23, 2022, 05:32:29 pm
Windows is a logical impossibility, 'cause then the issue wouldn't have occurred on account of that line. ;)

Quote from: Spuds – So what I'm saying is I feel the PHP net documentation may be misleading, other than maybe the qmail stuff?

I feel that it's not misleading because if it were the issue wouldn't have occurred. :P

Also the PHP documentation is definitely not written Windows-first.
Title: Re: Gmail: multiple addresses in From:n550 5.7.1 header are not accepted
Post by: Frenzie on March 23, 2022, 05:38:10 pm
There are some Windows-only notes too btw. The \r\n thing definitely isn't one of those.

Incidentally, this comment seems to explain the behavior, although it simultaneously contradicts it:

QuoteSecurity advice: Although it is not documented, for the parameters $to and $subject the mail() function changes at least \r and \n to space. So these parameters are safe against injection of additional headers. But you might want to check $to for commas as these separate multiple addresses and you might not want to send to more than one recipient.

The crucial part is the $additional_headers parameter. This parameter can't be cleaned by the mail() function. So it is up to you to prevent unwanted \r or \n to be inserted into the values you put in there. Otherwise you just created a potential spam distributor.

That might indicate it's new, for some value of new for a PHP release after December of 2016, anyway. But it might explain the odd single-space indentation.

Additionally, it's only been a few years that Gmail has started showcasing this bouncing behavior, so it would've been mostly irrelevant in the past.
Title: Re: Gmail: multiple addresses in From:n550 5.7.1 header are not accepted
Post by: Spuds on March 23, 2022, 06:22:45 pm
I was just curious if you were running something like Xampp, or other,  stack on windows, these can do some odd things.

I looked at the "ancestry" and the CRLF vs LF has been that way for ~15 years, it is not something new in Elk  Going back even further it looks like it was CRLF back when php4 was a thing.   If I change it to CRLF the mail tests fail in our github actions testbed which is Ubuntu 20.04, so at least Ubuntu/Debian native sendmail expect only LF. 

So what OS are you  running ? Have you specified anything different in your php.ini mail settings where it may be using SMTP instead of sendmail ?

Title: Re: Gmail: multiple addresses in From:n550 5.7.1 header are not accepted
Post by: Frenzie on March 23, 2022, 06:33:51 pm
Is that something that would be mentioned in phpinfo()? I don't run it from my VPS but from shared hosting (Hetzner). But it's Linux 4.19 with PHP 8.1.4.

Quote from: Spuds – If I change it to CRLF the mail tests fail in our github actions testbed which is Ubuntu 20.04, so at least Ubuntu/Debian native sendmail expect only LF. 
Fails in which way? Sending a message locally won't care if it's all \n.
Title: Re: Gmail: multiple addresses in From:n550 5.7.1 header are not accepted
Post by: tino on March 24, 2022, 02:33:17 pm
CRLF is the rfc standard and Debian follows that for its default mail packages. Only one I know which doesn’t conform well is qmail, but that’s a configuration option from memory.

You can however set the headers as an array from PHP 7.2 so it’s not decided by you but PHP itself, If that’s a benefit or not I don’t know…
Title: Re: Gmail: multiple addresses in From:n550 5.7.1 header are not accepted
Post by: Spuds on March 28, 2022, 12:24:39 pm
In 2.0 the headers are in an [] (but then imploded on the break style) but I can change that !

You might be able to see what OS is running from the ACP Support and Credits section.  Sometimes the OS infor will be in the Version Information area, all depends.  You can also get to your PHP info from there which will show some mail settings such as where the sendmail program is located etc.

Where I was going with the LF was that if PHP is using sendmail I would have thought that the input would be the same as if I was using sendmail from the nix command line, at which point I would not be using CRLF.

Looks like there was some discussion about this over the years as well
This on the docs changing https://bugs.php.net/bug.php?id=15841

And this one on the function changing in php8 https://bugs.php.net/bug.php?id=81158 specifically https://github.com/php/php-src/commit/6983ae751cd301886c966b84367fc7aaa1273b2d#diff-c6922cd89f6f75912eb377833ca1eddb7dd41de088be821024b8a0e340fed3df

So looks like php8+ should get CRLF and perhaps older versions LF?
Title: Re: Gmail: multiple addresses in From:n550 5.7.1 header are not accepted
Post by: Frenzie on March 31, 2022, 04:08:07 pm
I already had issues with 7.1 and 1.0.10, in retrospect most likely on account of Elk supplying \n instead of \r\n like it should. I just never quite got around to investigating it properly sine I was hoping 1.1.x would magically fix the problem.

But it may well be that older PHP is just a broken unusable mess period. Anyway, if your CI thinks you should do that it's probably at least half right. ;)
Title: Re: Gmail: multiple addresses in From:n550 5.7.1 header are not accepted
Post by: Spuds on March 31, 2022, 09:05:24 pm
That comes across a tad bit mocking, no matter, I am trying to figure out why things were that way for many years.
Title: Re: Gmail: multiple addresses in From:n550 5.7.1 header are not accepted
Post by: Steeley on April 01, 2022, 01:17:36 am
Quote from: Spuds – That comes across a tad bit mocking, no matter, I am trying to figure out why things were that way for many years.

"Grandfather" finally died?  :P

[Imagine my surprise when I discovered DOS still supported the old (undocumented in DOS) CP/m "ERA" command... until DOS 6.1, at which point it just no longer did... ]
Title: Re: Gmail: multiple addresses in From:n550 5.7.1 header are not accepted
Post by: Frenzie on April 01, 2022, 06:01:00 am
Quote from: Spuds – That comes across a tad bit mocking, no matter, I am trying to figure out why things were that way for many years.
If you mean of PHP, reading it now I see it was too mean-spirited, but definitely not intended of Elk if that's what you mean! ;)

But anyway, I think PHP is pretty good too.
Title: Re: Gmail: multiple addresses in From:n550 5.7.1 header are not accepted
Post by: Spuds on April 01, 2022, 10:32:11 am
Were you able to see what your PHP sendmail_path setting is? I'd really like to know what that is and from which MTA its from.

On my Ubuntu server its /usr/sbin/sendmail -t -i and the version of sendmail at that location is the version put in place by Postfix which is the MTA I have installed.   

Depending on what MTA is in use, the sendmail command would be the CLI interface to Exim, Postfix, Qmail I guess it could be actual Sendmail (I've only used Postfix or Exim).  All the MTA's are required to provide a sendmail interface, even if they do not support all of the original sendmail options.  The man page for sendmail on my system simply lists several of the options as ignored.

Where I get torn on this is that if I were to use sendmail from the *nix CLI I would not be entering CRLF, not even if I were using predefined header files.  I did find this in the Exim man page which explains what they do, which indicates they do lots of manipulations to get things consistent. 

Quote2. Line endings
RFC 2821 specifies that CRLF (two characters: carriage-return, followed by linefeed) is the line ending for messages transmitted over the Internet using SMTP over TCP/IP. However, within individual operating systems, different conventions are used. For example, Unix-like systems use just LF, but others use CRLF or just CR.

Exim was designed for Unix-like systems, and internally, it stores messages using the system’s convention of a single LF as a line terminator. When receiving a message, all line endings are translated to this standard format. Originally, it was thought that programs that passed messages directly to an MTA within an operating system would use that system’s convention. Experience has shown that this is not the case; for example, there are Unix applications that use CRLF in this circumstance. For this reason, and for compatibility with other MTAs, the way Exim handles line endings for all messages is now as follows:

LF not preceded by CR is treated as a line ending.

CR is treated as a line ending; if it is immediately followed by LF, the LF is ignored.

The sequence “CR, dot, CR” does not terminate an incoming SMTP message, nor a local message in the state where a line containing only a dot is a terminator.

If a bare CR is encountered within a header line, an extra space is added after the line terminator so as not to end the header line. The reasoning behind this is that bare CRs in header lines are most likely either to be mistakes, or people trying to play silly games.

If the first header line received in a message ends with CRLF, a subsequent bare LF in a header line is treated in the same way as a bare CR in a header line.

I think for *nix it seems fine to use LF or CRLF to either Exif or Postfix "sendmail" interfaces but it needs to be consistent. Qmail seems to only accept LF. 

PHP 8 looks to only use CRLF, if it adds headers or if you pass it an array (which seems to indicate that you can't use the array syntax if Qmail is the MTA)

Confused yet, good, me too :smile: It feels like LF, even although not rfc2822 proper, provided, at least at some point,  the widest *nix support to sendmail which in turn sends proper rfc2822 syntax to the MTA.
Title: Re: Gmail: multiple addresses in From:n550 5.7.1 header are not accepted
Post by: Frenzie on April 01, 2022, 01:35:49 pm
Code: [Select]
mail.add_x_header	Off	Off
mail.force_extra_parameters	no value	no value
mail.log	no value	no value

Code: [Select]
sendmail_from	no value	no value
sendmail_path	/usr/sbin/sendmail -t -i	/usr/sbin/sendmail -t -i

Code: [Select]
Path to sendmail 	/usr/sbin/sendmail -t -i

Title: Re: Gmail: multiple addresses in From:n550 5.7.1 header are not accepted
Post by: Spuds on April 01, 2022, 03:27:01 pm
Thank you for that, interesting as well since its almost mirrors what I have. 

Are you able to see what MTA your server has installed?   Just trying to narrow things down.  On my quick test setup ubuntu 20.04 w/postfix either /n or /r/n worked with PHP 7.4 and PHP 8.0 (I'm delinquent on 8.1 testing)  We can assume you are not using Qmail since that is the only one that, according to the php site, requires /n ... that should leave (for the most part) postfix, exim or sendmail.  If you don't have shell access it might show your MTA in outbound email headers.

I don't mind using just CRLF in 2.0 and see how it goes, but I'm hesitant to make a 1.1.9 change.

ETA: Answered my own question, looked back at the one you posted and it says (Exim 4.92.3) Now to set aside some time to setup a test server with that MTA
Text only | Text with Images