ElkArte Community

Project Support => Support => Topic started by: rjm on November 25, 2021, 12:18:02 pm

Title: "Missing Key" error on reply-email
Post by: rjm on November 25, 2021, 12:18:02 pm
I have just updated to 1.1.8 (from 1.1.6). Now I'm willing to resolve an email-problem. Reply emails (post by email) can be viewd in the moderator view, but tagged with "Missing Key". Auto repair is not possible.

I'd suggest that the message-ID is shaped like this, visible in the email body when view "all headers and message" is activated.
Message-ID: <744f8fa1bfcf919a924dc845228ac5c5-20@forum.xxxxxx.yy>

I can see that the reply to the forum contains the same ID as the received notification.
I also experimented - without success - by copying the key into the message body directly, since a short code review tells me that the key is searched for in the header section as well as in the mail body.

Hint: Elkarte is not setup to operate with piping option since the mail server does not support this (I assume that piping is recommended simply because of no mailbox polling delay).
I think that the settings for this (in my case: POP3, TLS, port 110) is not specifically interesting, since the emails arrive and can be viewed in de moderate queue.

Is there any log that I can access in order to localize the cause of the problem?

Title: Re: "Missing Key" error on reply-email
Post by: Spuds on November 27, 2021, 09:45:38 pm
Could you attach a raw example message that fails so I can take a look at what may be happening? 

If it can't find an ID in the message no auto repair is possible, so that behavior is correct.
Title: Re: "Missing Key" error on reply-email
Post by: rjm on November 28, 2021, 05:16:52 am
Thanks for your reply. Elkarte could help me to achieve acceptance  in using a forum (at all) in a small group that communicates mainly via email. Just because Elkarte supports reply and post by email.

Here you are (ending .mbs changed  in .txt, further some details replaced by xxxxx).

EDIT: I have just activated "notity" on this thread - I'd be able to test reply by email using the same mail client.
Title: Re: "Missing Key" error on reply-email
Post by: Spuds on November 30, 2021, 10:03:10 pm
That message has no ElkArte post by email 'key' in it :(

The key is, in regex speak
Code: [Select]
~(([a-z0-9]{32})\-(p|t|m)(\d+))~i';

This translates to a 32 alpha numeric string followed by a - followed by the letter p, t or m and then a sequence of numbers that correspond to the the specific P T or M ... the system will then check if that key was sent to the email that responded.  It will look
like

7738c27ae6c431495ad26587f30e2121-m29557

That email has the 32 characters, then the - but it NOT followed by a p, t, or m ??? I'm not sure how that is possible, but what it has is 79aaad41a2afc55bd3c1c17717fad46a-22 which will not process.  I'm not sure where to check TBH



Title: Re: "Missing Key" error on reply-email
Post by: rjm on December 01, 2021, 04:08:21 am
Hi, thanks for clarifying,
 
the message ID is in the header part. So as for test, I now reply without the copy of the message ID in the bode. Should work. This reply is by email. But it appears not to arrive. I just saw that the sender is a "noreply"-address. Hence, reply by email is not activated.

I'll try to analyze what's going wrong in my Elkarte forum setup. Apparently the key is not reflected to the header section as intended in notify emails.
Could you help point to the relevant code?
Title: Re: "Missing Key" error on reply-email
Post by: Spuds on December 05, 2021, 10:42:12 am
Sorry I missed this post ... the key is added as part of the sendmail function which will be found in the mail.subs.php file

One the ElkArte side follow the information in this guide: https://github.com/elkarte/Elkarte/wiki/Posting-by-Email-Feature

If the sent email has all the right headers (and there is no reason it should not as long as you have followed the above) then the problem would be with the email client that is responding.  Not all email clients are well behaved.
Title: Re: "Missing Key" error on reply-email
Post by: rjm on December 05, 2021, 06:24:28 pm
I tried two different email-domain addresses, and also answered using not my desktop email-client but also webmail client.
Mail arrive still in moderation queue...

BTW: I see a function function mail_insert_key($message, $unq_head, $line_break) (line 785), which is declared as a "safety net for clients that strip out the message-id and in-reply-to headers".
Indeed, I see such a key - e.g.: ELK-d6aa927ccde3acc5826c9f88a7c6a874, but this is not part of the reply. But when I add this to the reply, it still remains in the moderation queue.

I think I'd add a log file entry where generated keys in mail_insert_key are stored - as an aid to find the final cause of the problem.
Title: Re: \
Post by: rjm on August 04, 2022, 06:20:50 am
Finally, I found out the problem in key generation (just in the last days I reentered this area).

Line 46 on Mail.subs.php seems to contain the problem (v. 1.1.8 ):
Code: [Select]
$maillist = !empty($modSettings['maillist_enabled']) && $from_wrapper !== null && $message_id !== null && $priority < 4 && empty($modSettings['mail_no_message_id']);
When I remove the from_wrapper test, the key is generated and the mail arrives in the forum.

Of course, the problem may also originate from the call of sendmail() without the wrapper argument, resp. a configuration setting that is related to wrapper.

It can clearly be seen that the security key is on the bottom of every email, enclosed by square brackets (would be helpful information into the appropriate wiki page.
Title: Re: "Missing Key" error on reply-email
Post by: Spuds on August 04, 2022, 07:54:08 am
What settings do you have in the maillist area?  Could you take a screen shot and post that? 
Title: Re: "Missing Key" error on reply-email
Post by: rjm on August 04, 2022, 08:42:24 am
Here you are...
Title: Re: "Missing Key" error on reply-email
Post by: Spuds on August 04, 2022, 06:04:54 pm
Are you only wanting to support PM reply by email or all posts in general? 

Currently Allow posting to the forum by Email is not enabled.   I'll have to try that setting to see if how the code handles that condition, looks like improvements could be made.

I'll change my test site to use your settings and see what I get
Title: Re: "Missing Key" error on reply-email
Post by: rjm on August 06, 2022, 02:23:55 am
All posts in general. Just as Steeley wrote: a replacement for a classic distribution list.

BTW: I experienced - since it's operating now - that the post content in the notification email appears quite towards the bottom of the email (v.1.1.8 ).

QuoteSection with two links and then

The text of the reply is shown below:
test
 
 
Regards,
The xxxx-Forum Team.
 
[df84f9b51e556e770d9be6f3d2afe98e-m120]
 
 
 
It would make more sense if the standard stuff would be below the posting content.
I see that in dev. 2.0 the standard stuff is better structured. As such predestined to appear to the bottom.

On top of the notification including a post, only the following text should appear before the posting content:
Quote=== automated email containing the following new posting from xxxx-Forum. You can reply to this posting or post a reply direclty in the forum - hints & links see bottom section ===

I doubt whether it is good or not to send notifications in html-formatted mails (as is the case with dev 2.0.
Title: Re: "Missing Key" error on reply-email
Post by: Spuds on August 06, 2022, 04:13:52 pm
You should be able to make outbound adjustments with edits to the Maillist.Templates  to make it what you need.

For the inbound email, that is where the Filters and Parsers come in to play.  The idea is to make what gets posted look like a post and not an email.

Filters run first, and you can do simple find and replace.

Parsers run after Filters and there you can define regex code with the goal of finding either the start of a signature or the start of a quoted message and cut the message at that point.  Of course that only works if they post above the original in the reply.

I have been working on about 6 generic parser regex codes that seem to handle a majority of cases.  I'll post those if interested.
Title: Re: \
Post by: rjm on August 07, 2022, 01:33:46 am
Quote from: Spuds – You should be able to make outbound adjustments with edits to the Maillist.Templates  to make it what you need.
Well, the templates section mentions the "template selection list". Hence, I'd believe that the template name is entered elsewhere, where it would be in effect for a specific outbound email class (in this case a "reply-by-email").
According to the header title of the templates section ("Custom bounce email templates"), templates seem only to be related to e.g. error bounce messages.

When I compare to filters resp. parser, a template name is merely optional. They are just simply active on all inbound emails.

Further to templates:
It's the recipient of a reply-by-email (trough notification active) who determines whether or not the posting appears in the email body - via a setting in the account preferences. Despite that, the template (on the admin settings) can indeed determine whether the posting appears e.g. on top of the email? When I inspect the list of template content shortcodes, I would expect a shortcode for "posting content" (to insert on a desired place in the text box "Notification Subject"
Title: Re: "Missing Key" error on reply-email
Post by: Steeley on August 19, 2022, 01:47:45 pm
Quote from: rjm –
Further to templates:
It's the recipient of a reply-by-email (trough notification active) who determines whether or not the posting appears in the email body - via a setting in the account preferences. Despite that, the template (on the admin settings) can indeed determine whether the posting appears e.g. on top of the email? When I inspect the list of template content shortcodes, I would expect a shortcode for "posting content" (to insert on a desired place in the text box "Notification Subject"

As I mentioned earlier....

Quote from: Steeley –
{snip} .. hopefully Google doesn't decide in the meantime to implement some new "security/spam/pfishing" Gmail format protocol and try to drive it to the rest of the known universe.


Alas, I've discovered  subsequently that is exactly what they (Google, et. al.)  are doing -->  OAUTH2  [ https://www.pmail.com/devnews.htm ]

It looks like emails are being herded into providing all sorts of new format requirements, and the 'majors' (or at least google) are headed toward gatekeeping/annual -subscription of email clients that need to demonstrate "compliance" with these new (custom) standards for "acceptance" on their platforms.

I imagine much of the necessary formatting and compliance demonstration (none of which is actually RFC - yet anyway) will need to be done at the email server level, and subsystem forum email handlers (for one example) merely need to construct payload for those email servers, at least outbound.  Inbound, however, is a whole 'nother kettle of snakes - I'm not sure what "transport" formatting will be passed through to the subsystems (of which email clients are but one class, forums another) that have to be filtered and parsed for proper payload presentation to the user.

I can see a "transitory" implementation period of chaos if your email server's not yet compliant and paid the mordita to google (in this case) that your email just gets rejected going to gmail clients.  And I can see a longer 'transitory' period of chaos for inbound email as we all have to adapt to the new OAUTH2 formatting (as Dave Harris is also tearing his hair out over with Pegasus), as well as forum developers and admins.  All of which is made longer because of the poor documentation of what exactly OAUTH2 should require.

Now, because of the nature of my forum and user discussions, I've implemented controls to assure as much user 'privacy' as I can given the nature of the internet - a large part involving 'security through obscurity' - our traffic is an infinitesimally small component of the traffic out there. - just a tiny bit of straw in a mountainous haystack.  However, I know for a fact that google scans content of everything it comes in contact with and what it 'stores away' for later use is 'proprietary information' they don't reveal. So, my forum refuses to allow anyone to register with a gmail address, and at least I can keep that privacy vulnerability from crawling into the forum discussions.  Consequently I should not be greatly affected (at least initially) if Google unilaterally implements their interpretation of this new 'pseudo-standard' thinking their 'size and weight' will force everyone else to comply with it, at least until the others in the "consortium' go along.
Spuds has already stumbled over one such gmail "change" that interrupted email functionality in EA.

Yes, most email clients allow the user to decide whether their content appears above or below the quote of the replied-to message,  and for security purposes in my forum I tell those that use email to make sure their content appears at the top, since I strip everything that includes forum links that appears in the reply. But that's just one formatting option that has to be handled by the email parser. Spuds and I just did a long hokey-pokey dance with handling various client email options from just one client (and that will likely change again with their next release).  There's literally hundreds of others out there, and if OAUTH2 takes hold it will likely increase exponentially.

As the saying goes, "Incompatibility is just an upgrade away".  And years ago, someone (I forget who) noted: "Americans love standards, which is why we have so many of them."

It appears to me that there's a major industry thrust to create "barriers to entry" by creating "new standards" that under the banner of "security" are so complex and onerous that it's not economically feasible to maintain "open source" software that will function reliably (if at all). Zuckerberg argued a couple years ago that regulations are "necessary to preserve free speech and privacy rights from authoritarian rule-makers." And he made that argument to...  the "authoritarian rule-makers." He kinda upset them by avoiding answering their questions and making specific recommendations, though. But I don't believe he cares, actually, because he has the resources to comply with whatever they come up with, and if they don't, the "consortium' has enough control to craft those standards and impose it on everyone else themselves (e.g. OAUTH2), without even going through the formal RFC adoption process. "Wag the dog", so to speak.

I guess what I'm suggesting is that I foresee bigger issues going forward than just whether new content is above or below the quoted content  (not to say that the question isn't valid or pertinent), just that I'm not confident we (I'm thinking Spuds at the moment, but us Admins too, and especially those like me who are "marginal programmers" at best and depend on guys like spuds to provide the configuration options we want) are going to be able to keep up with what's coming.

I wouldn't be surprised if native "forum-supported email" becomes a thing of the past at some point, and if you want it you'll have to buy or rent the package from Google (et. al.) and the forum just has to provide the hooks, or "compliant APIs". 

I'm not sure we all appreciate how much beer we owe Spuds...  ;D

(Edited for typos)
Title: Re: \
Post by: tino on August 19, 2022, 03:32:44 pm
Microsoft of all companies actually have decent documentation on how to integrate with their OAuth client and send a mail. It’s also up to date and works as we’ve just finished integrating with it for a work project as the powers that be decided we should send our mail through outlook rather than use our own postfix server which was locked down to certain IP’s, also had a secure username and password, and finally was monitored and checked daily…. as outlook is more secure. 🤷🏻‍♂️

Their docs are here https://docs.microsoft.com/en-us/exchange/client-developer/legacy-protocols/how-to-authenticate-an-imap-pop-smtp-application-by-using-oauth#register-your-application

edit:
I do like how the man shrugging emoji is interpreted on here.
Title: Re: "Missing Key" error on reply-email
Post by: Steeley on August 19, 2022, 05:15:57 pm
Quote from: tino – Microsoft of all companies actually have decent documentation on how to integrate with their OAuth client and send a mail. It’s also up to date and works as we’ve just finished integrating with it for a work project as the powers that be decided we should send our mail through outlook rather than use our own postfix server which was locked down to certain IP’s, also had a secure username and password, and finally was monitored and checked daily…. as outlook is more secure. 🤷🏻‍♂️

Their docs are here https://docs.microsoft.com/en-us/exchange/client-developer/legacy-protocols/how-to-authenticate-an-imap-pop-smtp-application-by-using-oauth#register-your-application

edit:
I do like how the man shrugging emoji is interpreted on here.

If everyone adopted that approach it would solve just one of the many problems. The issue is that the protocol is so squishy that there's going to be dozens (or more) implementation versions of it, each likely to require identifying which is being used and writing custom filters and parsers for that particular version.
Microsoft has lots of experience with OS integration so I would expect they would be well thought out and documented with their implementation. Google's is different, and apparently not well documented to boot. The others . . will be different still,  Building a "framework" to handle them all could become a nightmare, which more than a few suspect is the intention (including Eran Hammer and apparently Dave Recordon, who were lead authors on the IETF group OAUTH 2 project). 
Title: Re: "Missing Key" error on reply-email
Post by: Spuds on August 19, 2022, 05:23:15 pm
Quote from: tino – I do like how the man shrugging emoji is interpreted on here.
I'm going to assume you did not use the emoji pulldown and instead did a keyboard emoji ?   Do you happen to have the utf8 code used.  The site should be "converting" the ones it knows to use the site icon and not the browser default.  Looks like it did a partial on some multi byte emoji, so you know perhaps a bug.
Title: Re: \
Post by: Spuds on August 19, 2022, 05:24:18 pm
Quote from: Steeley – I'm not sure we all appreciate how much beer we owe Spuds
I like the way this guy thinks :beers: !
Title: Re: "Missing Key" error on reply-email
Post by: tino on August 20, 2022, 05:45:05 am
Quote from: Spuds –
Quote from: tino – I do like how the man shrugging emoji is interpreted on here.
I'm going to assume you did not use the emoji pulldown and instead did a keyboard emoji ?   Do you happen to have the utf8 code used.  The site should be "converting" the ones it knows to use the site icon and not the browser default.  Looks like it did a partial on some multi byte emoji, so you know perhaps a bug.

I used the one on the Apple iPhone keyboard, who uses a PC these days to browse the net? 😝

When I get a chance I’ll try and work out what it is.
Title: Re: \
Post by: tino on August 20, 2022, 04:37:01 pm
🤷‍♂️

https://emojipedia.org/man-shrugging/

If you copy the image on there it seems to also break, not sure if that helps you at all @Spuds
Title: Re: "Missing Key" error on reply-email
Post by: Spuds on August 20, 2022, 07:01:35 pm
Anything repeatable is :moneybag:  I can see its not processing those multi byte emoji properly, to fix!
Title: Re: "Missing Key" error on reply-email
Post by: Spuds on August 22, 2022, 11:27:08 am
I think I have a fix for the emoji issue, its at least something to test. 

I will not drag everyone through emoji / utf-8 / mySql's 3 byte / differences in codes used in libraries / bla.  Lets just say things got needless messy! 

The system attempts to find "keyboard" emoji and display them in the sites emoji set so things look consistent.  When it can't it should just show the image as the browser decides.  It was getting lost due to the juggling that is done to support 4byte in the mysql utf-8 space.  I have not tested the update extensively.

🤷‍♂️<-- That should be man shrugging, which is  1F937-‍200D-2642-️FE0F or generic person shrugging+zero space+man symbol+variant selector in the sites image set.
Title: Re: "Missing Key" error on reply-email
Post by: Steeley on August 22, 2022, 11:14:22 pm
I'm glad I don't have to worry about this stuff on my boards, all my users are mature adults.

(..that use PCs on the internet..  8)  )
Title: Re: "Missing Key" error on reply-email
Post by: tino on August 24, 2022, 09:46:37 am

By mature you mean archaic? 😉
Title: Re: "Missing Key" error on reply-email
Post by: Steeley on August 24, 2022, 06:21:39 pm

The opposite of puerile  ;)

I almost hate to suggest it, but I was thinking the next step is a "Magic 8-ball" that you click on and it randomly pops up "Ask Again Later", "Without a Doubt", "Very Doubtful".. etc.. 
Spawn that into "Angry Child" that spews random curses , tantrums, whatever..

Maybe create an emote board, no text allowed, only emoticons..  you could have a lot of fun with that too.
(Supposedly the shortest recorded "telegram conversation" in history was a "?" query, that elicited a "!" response.)
 
Or how about a Klingon language module..  the possibilities for frivolity are literally endless. 

/Linux is free if your time is worthless/

(Link-10086)
Title: Re: "Missing Key" error on reply-email
Post by: Spuds on August 24, 2022, 08:12:22 pm
I did create a talk like pirate addon that is fun :) ... It did basic substitution for common phrases and changed them into pirate speak!

Could be changed to do Klingon as well :smiley:
Title: Re: \
Post by: Spuds on August 25, 2022, 12:28:53 pm
This are emoji: 🤗✨🤷‍♂️❤️✅ <- all cut and pasted in

The next are all ZWJ dual 4byte, so will go into the database as  &#x1f636;‍[zwj]&#x1f32b;️ as an example
😶‍🌫️ Face in Clouds
😮‍💨 Face Exhaling
😵‍💫 Face with Spiral Eyes

This is a 4byte followed by a 3byte ... this will be stored as &#x1f937[zwj];‍♂️
🤷‍♂️Man Shrugging

A 3 byte followed by a 4 byte
❤️‍🔥 Heart on fire ... this should be ❤️‍[zwj]&#x1f525;
Title: Re: "Missing Key" error on reply-email
Post by: tino on August 25, 2022, 02:51:45 pm
🤷🏻‍♂️😋🤦🏻‍♂️🥵😁

edit: when I go to edit they are shown correctly.
Title: Re: "Missing Key" error on reply-email
Post by: Spuds on August 25, 2022, 03:22:56 pm
Nods ... I had noticed that in your original post (that some were not substituting correctly). 

I know why that is happening in the code.  It does not help that apple is rolling their own ZWJ sequence (remember when MS was the standards problem child, well they have been surpassed!)

I just have to decide on the last part of the fix, I have an idea to try.
Title: Re: "Missing Key" error on reply-email
Post by: Spuds on August 25, 2022, 04:20:53 pm
I put the "fix" that I was thinking may work in place ... seems like those emoji are now displaying more normally.  I'm sure there are some other pages/categories that may cause a problem, just need to look for them. 

The basic problem is there are far more emoji then there is support for them in across the various emoji image libraries and even some browsers.  The browsers catch up pretty quickly, however the the library sets are slow to update, if they do at all.

When it comes time to do the replacements, if a multi-point emoji is not supported but is composed of supported single point emoji, you get all the single point images.  That is supposed to be the correct fallback as the individual emoji string is supposed to convey the same feeling/idea as the compound image.  That said its those modifiers that end up looking really odd.  The apple one was person shrugging - w/pale skin tone -w/male with a ZWJ stuffed in there as well.

So emoji :man_facepalming: :zany_face:
Title: Re: "Missing Key" error on reply-email
Post by: Steeley on August 25, 2022, 05:13:44 pm
Now you have to fix it so they don't look like garbage in emails...  :'(
 
Title: Re: "Missing Key" error on reply-email
Post by: Spuds on August 25, 2022, 09:38:06 pm
Quote from: Steeley – Now you have to fix it so they don't look like garbage in emails...  :'(
 
They look fine in my email ... both in html and plain (base64) ... have you thought about using gmail? :tongue:
Title: Re: "Missing Key" error on reply-email
Post by: Steeley on August 25, 2022, 10:52:47 pm
So you've been assimilated into the borg... for smilies? Your soul is cheap.. :-b

-Steeley 


Did you exchange, a walk-on part in the war for a lead role in a cage?

[~ Pink Floyd: "Wish You Were Here"]


Title: Re: "Missing Key" error on reply-email
Post by: Spuds on August 26, 2022, 09:52:51 am
LOL ... Thats not the only cheap thing !
Title: Re: "Missing Key" error on reply-email
Post by: Steeley on August 26, 2022, 11:23:31 am
If it's laughs you refer to, that's not a bad thing.. ✅