Skip to main content ↑ ↓
"Missing Key" error on reply-email Started by rjm · · Read 16980 times 0 Members and 1 Guest are viewing this topic. previous topic - next topic

"Missing Key" error on reply-email

I have just updated to 1.1.8 (from 1.1.6). Now I'm willing to resolve an email-problem. Reply emails (post by email) can be viewd in the moderator view, but tagged with "Missing Key". Auto repair is not possible.

I'd suggest that the message-ID is shaped like this, visible in the email body when view "all headers and message" is activated.
Message-ID: <744f8fa1bfcf919a924dc845228ac5c5-20@forum.xxxxxx.yy>

I can see that the reply to the forum contains the same ID as the received notification.
I also experimented - without success - by copying the key into the message body directly, since a short code review tells me that the key is searched for in the header section as well as in the mail body.

Hint: Elkarte is not setup to operate with piping option since the mail server does not support this (I assume that piping is recommended simply because of no mailbox polling delay).
I think that the settings for this (in my case: POP3, TLS, port 110) is not specifically interesting, since the emails arrive and can be viewed in de moderate queue.

Is there any log that I can access in order to localize the cause of the problem?

 

Re: "Missing Key" error on reply-email

Reply #1 –

Could you attach a raw example message that fails so I can take a look at what may be happening? 

If it can't find an ID in the message no auto repair is possible, so that behavior is correct.

Re: "Missing Key" error on reply-email

Reply #2 –

Thanks for your reply. Elkarte could help me to achieve acceptance  in using a forum (at all) in a small group that communicates mainly via email. Just because Elkarte supports reply and post by email.

Here you are (ending .mbs changed  in .txt, further some details replaced by xxxxx).

EDIT: I have just activated "notity" on this thread - I'd be able to test reply by email using the same mail client.

Re: "Missing Key" error on reply-email

Reply #3 –

That message has no ElkArte post by email 'key' in it :(

The key is, in regex speak
Code: [Select]
~(([a-z0-9]{32})\-(p|t|m)(\d+))~i';

This translates to a 32 alpha numeric string followed by a - followed by the letter p, t or m and then a sequence of numbers that correspond to the the specific P T or M ... the system will then check if that key was sent to the email that responded.  It will look
like

7738c27ae6c431495ad26587f30e2121-m29557

That email has the 32 characters, then the - but it NOT followed by a p, t, or m ??? I'm not sure how that is possible, but what it has is 79aaad41a2afc55bd3c1c17717fad46a-22 which will not process.  I'm not sure where to check TBH



Re: "Missing Key" error on reply-email

Reply #4 –

Hi, thanks for clarifying,
 
the message ID is in the header part. So as for test, I now reply without the copy of the message ID in the bode. Should work. This reply is by email. But it appears not to arrive. I just saw that the sender is a "noreply"-address. Hence, reply by email is not activated.

I'll try to analyze what's going wrong in my Elkarte forum setup. Apparently the key is not reflected to the header section as intended in notify emails.
Could you help point to the relevant code?

Re: "Missing Key" error on reply-email

Reply #5 –

Sorry I missed this post ... the key is added as part of the sendmail function which will be found in the mail.subs.php file

One the ElkArte side follow the information in this guide: https://github.com/elkarte/Elkarte/wiki/Posting-by-Email-Feature

If the sent email has all the right headers (and there is no reason it should not as long as you have followed the above) then the problem would be with the email client that is responding.  Not all email clients are well behaved.

Re: "Missing Key" error on reply-email

Reply #6 –

I tried two different email-domain addresses, and also answered using not my desktop email-client but also webmail client.
Mail arrive still in moderation queue...

BTW: I see a function function mail_insert_key($message, $unq_head, $line_break) (line 785), which is declared as a "safety net for clients that strip out the message-id and in-reply-to headers".
Indeed, I see such a key - e.g.: ELK-d6aa927ccde3acc5826c9f88a7c6a874, but this is not part of the reply. But when I add this to the reply, it still remains in the moderation queue.

I think I'd add a log file entry where generated keys in mail_insert_key are stored - as an aid to find the final cause of the problem.

Re: \

Reply #7 –

Finally, I found out the problem in key generation (just in the last days I reentered this area).

Line 46 on Mail.subs.php seems to contain the problem (v. 1.1.8 ):
Code: [Select]
$maillist = !empty($modSettings['maillist_enabled']) && $from_wrapper !== null && $message_id !== null && $priority < 4 && empty($modSettings['mail_no_message_id']);
When I remove the from_wrapper test, the key is generated and the mail arrives in the forum.

Of course, the problem may also originate from the call of sendmail() without the wrapper argument, resp. a configuration setting that is related to wrapper.

It can clearly be seen that the security key is on the bottom of every email, enclosed by square brackets (would be helpful information into the appropriate wiki page.

Re: "Missing Key" error on reply-email

Reply #8 –

What settings do you have in the maillist area?  Could you take a screen shot and post that? 

Re: "Missing Key" error on reply-email

Reply #9 –

Here you are...

Re: "Missing Key" error on reply-email

Reply #10 –

Are you only wanting to support PM reply by email or all posts in general? 

Currently Allow posting to the forum by Email is not enabled.   I'll have to try that setting to see if how the code handles that condition, looks like improvements could be made.

I'll change my test site to use your settings and see what I get

Re: "Missing Key" error on reply-email

Reply #11 –

All posts in general. Just as Steeley wrote: a replacement for a classic distribution list.

BTW: I experienced - since it's operating now - that the post content in the notification email appears quite towards the bottom of the email (v.1.1.8 ).

QuoteSection with two links and then

The text of the reply is shown below:
test
 
 
Regards,
The xxxx-Forum Team.
 
[df84f9b51e556e770d9be6f3d2afe98e-m120]
 
 
 
It would make more sense if the standard stuff would be below the posting content.
I see that in dev. 2.0 the standard stuff is better structured. As such predestined to appear to the bottom.

On top of the notification including a post, only the following text should appear before the posting content:
Quote=== automated email containing the following new posting from xxxx-Forum. You can reply to this posting or post a reply direclty in the forum - hints & links see bottom section ===

I doubt whether it is good or not to send notifications in html-formatted mails (as is the case with dev 2.0.

Re: "Missing Key" error on reply-email

Reply #12 –

You should be able to make outbound adjustments with edits to the Maillist.Templates  to make it what you need.

For the inbound email, that is where the Filters and Parsers come in to play.  The idea is to make what gets posted look like a post and not an email.

Filters run first, and you can do simple find and replace.

Parsers run after Filters and there you can define regex code with the goal of finding either the start of a signature or the start of a quoted message and cut the message at that point.  Of course that only works if they post above the original in the reply.

I have been working on about 6 generic parser regex codes that seem to handle a majority of cases.  I'll post those if interested.

Re: \

Reply #13 –

Quote from: Spuds – You should be able to make outbound adjustments with edits to the Maillist.Templates  to make it what you need.
Well, the templates section mentions the "template selection list". Hence, I'd believe that the template name is entered elsewhere, where it would be in effect for a specific outbound email class (in this case a "reply-by-email").
According to the header title of the templates section ("Custom bounce email templates"), templates seem only to be related to e.g. error bounce messages.

When I compare to filters resp. parser, a template name is merely optional. They are just simply active on all inbound emails.

Further to templates:
It's the recipient of a reply-by-email (trough notification active) who determines whether or not the posting appears in the email body - via a setting in the account preferences. Despite that, the template (on the admin settings) can indeed determine whether the posting appears e.g. on top of the email? When I inspect the list of template content shortcodes, I would expect a shortcode for "posting content" (to insert on a desired place in the text box "Notification Subject"

Re: "Missing Key" error on reply-email

Reply #14 –

Quote from: rjm –
Further to templates:
It's the recipient of a reply-by-email (trough notification active) who determines whether or not the posting appears in the email body - via a setting in the account preferences. Despite that, the template (on the admin settings) can indeed determine whether the posting appears e.g. on top of the email? When I inspect the list of template content shortcodes, I would expect a shortcode for "posting content" (to insert on a desired place in the text box "Notification Subject"

As I mentioned earlier....

Quote from: Steeley –
{snip} .. hopefully Google doesn't decide in the meantime to implement some new "security/spam/pfishing" Gmail format protocol and try to drive it to the rest of the known universe.


Alas, I've discovered  subsequently that is exactly what they (Google, et. al.)  are doing -->  OAUTH2  [ https://www.pmail.com/devnews.htm ]

It looks like emails are being herded into providing all sorts of new format requirements, and the 'majors' (or at least google) are headed toward gatekeeping/annual -subscription of email clients that need to demonstrate "compliance" with these new (custom) standards for "acceptance" on their platforms.

I imagine much of the necessary formatting and compliance demonstration (none of which is actually RFC - yet anyway) will need to be done at the email server level, and subsystem forum email handlers (for one example) merely need to construct payload for those email servers, at least outbound.  Inbound, however, is a whole 'nother kettle of snakes - I'm not sure what "transport" formatting will be passed through to the subsystems (of which email clients are but one class, forums another) that have to be filtered and parsed for proper payload presentation to the user.

I can see a "transitory" implementation period of chaos if your email server's not yet compliant and paid the mordita to google (in this case) that your email just gets rejected going to gmail clients.  And I can see a longer 'transitory' period of chaos for inbound email as we all have to adapt to the new OAUTH2 formatting (as Dave Harris is also tearing his hair out over with Pegasus), as well as forum developers and admins.  All of which is made longer because of the poor documentation of what exactly OAUTH2 should require.

Now, because of the nature of my forum and user discussions, I've implemented controls to assure as much user 'privacy' as I can given the nature of the internet - a large part involving 'security through obscurity' - our traffic is an infinitesimally small component of the traffic out there. - just a tiny bit of straw in a mountainous haystack.  However, I know for a fact that google scans content of everything it comes in contact with and what it 'stores away' for later use is 'proprietary information' they don't reveal. So, my forum refuses to allow anyone to register with a gmail address, and at least I can keep that privacy vulnerability from crawling into the forum discussions.  Consequently I should not be greatly affected (at least initially) if Google unilaterally implements their interpretation of this new 'pseudo-standard' thinking their 'size and weight' will force everyone else to comply with it, at least until the others in the "consortium' go along.
Spuds has already stumbled over one such gmail "change" that interrupted email functionality in EA.

Yes, most email clients allow the user to decide whether their content appears above or below the quote of the replied-to message,  and for security purposes in my forum I tell those that use email to make sure their content appears at the top, since I strip everything that includes forum links that appears in the reply. But that's just one formatting option that has to be handled by the email parser. Spuds and I just did a long hokey-pokey dance with handling various client email options from just one client (and that will likely change again with their next release).  There's literally hundreds of others out there, and if OAUTH2 takes hold it will likely increase exponentially.

As the saying goes, "Incompatibility is just an upgrade away".  And years ago, someone (I forget who) noted: "Americans love standards, which is why we have so many of them."

It appears to me that there's a major industry thrust to create "barriers to entry" by creating "new standards" that under the banner of "security" are so complex and onerous that it's not economically feasible to maintain "open source" software that will function reliably (if at all). Zuckerberg argued a couple years ago that regulations are "necessary to preserve free speech and privacy rights from authoritarian rule-makers." And he made that argument to...  the "authoritarian rule-makers." He kinda upset them by avoiding answering their questions and making specific recommendations, though. But I don't believe he cares, actually, because he has the resources to comply with whatever they come up with, and if they don't, the "consortium' has enough control to craft those standards and impose it on everyone else themselves (e.g. OAUTH2), without even going through the formal RFC adoption process. "Wag the dog", so to speak.

I guess what I'm suggesting is that I foresee bigger issues going forward than just whether new content is above or below the quoted content  (not to say that the question isn't valid or pertinent), just that I'm not confident we (I'm thinking Spuds at the moment, but us Admins too, and especially those like me who are "marginal programmers" at best and depend on guys like spuds to provide the configuration options we want) are going to be able to keep up with what's coming.

I wouldn't be surprised if native "forum-supported email" becomes a thing of the past at some point, and if you want it you'll have to buy or rent the package from Google (et. al.) and the forum just has to provide the hooks, or "compliant APIs". 

I'm not sure we all appreciate how much beer we owe Spuds...  ;D

(Edited for typos)
Last Edit: August 19, 2022, 02:18:56 pm by Steeley
// Deep inside every dilemma lies a solution that involves explosives //