Skip to main content
Topic: [SOLVED] Help on Post/reply by email (Read 2663 times) previous topic - next topic
0 Members and 1 Guest are viewing this topic.

[SOLVED] Help on Post/reply by email

Sorry for bringing this up once again ... I have followed closely the discussion here, and I followed the advice(s) set out there.

After some fiddling with setting up mail forwarding at my hosting provider and the path to php binary, I think that incoming emails are now piped to the emailtopic.php script. At least I can see them coming in at the ElkArte section for "Moderation -> Unapproved Emails".

And here is the problem ... I do not get any information out of this mails, e.g. no "Subject", no "From", no "Body Text". The system alerts "Email ID not in Database", which might be an aftereffect of not detecting a sender address. The user I'm sending from IS registered in the database.

On a lighter note, also sending a bounce message from the admin page does fail with "Session verification failed. Please try logging out and back in again, and then try again."

Please advice what I could do to narrow down the problem. Any help is highly appreciated!

EDIT: Mark as "solved" ...
Last Edit: February 27, 2020, 05:38:59 am by JPalmer

[SOLVED] Help on Post/reply by email

Reply #1

After excessive debugging I found a few items that I want to put up for discussion.

In my opinion there are several problems in the logic for creating new topics from email.

1. Detecting and separating a mail message in headers and body
File / function:
Code: [Select]
EmailParse.class.php:327 / _split_headers()

Problem:
The function tries to detect beginning and end of the header section by using regular expressions. The problem is that this function is called recursively for parsing the sections of a multipart message. The sequence is:
  • for the whole message
  • for first section
  • for second section
  • ...

As a matter of fact, when called for a section, this will never include a header. Hence the existing code will always return without filling the body block, since this is done by the second regular expression that fills the reg exp result in the $match[] array.

Solution:
Just delete / comment the header start check:
Code: [Select]
// Do we even start with a header in this boundary section?
// if (!preg_match('~^[\w-]+:[ ].*?\r?\n~i', $this->raw_message))
// {
// return;
// }

2. Separating of the sections of a mail message
File / function:
Code: [Select]
EmailParse.class.php:751 / _boundary_split()

Problem:
The function tries to detect beginning and end of the header section by locating the delimiting '--<boundary>' pattern. The problem is that defined by the mail RFC1342 the multipart section is finally ended by adding two more hyphens at the end of the line, i.e. "--<boundary>--".
In the existing approach this last line causes creation of a section that has no content other than '--'.

Solution:
Modify to suppress the creation of a section not only for empty section, but also for closing section:
Code: [Select]
// Nothing or end of multipart split?
// if (empty($part))
if (empty($part) or (strcmp($part, '--') == 0))
{
continue;
}

3. Reformating the body part of a message
File / function:
Code: [Select]
EmailFormat.class.php:365 / _clean_up()

Problem:
The charset parameter is not processed correctly. Towards the end of the function a couple of special characters are replaced in a dedicated way. This must not be done if the text is UTF-8 encoded. Unfortunately this is checked only for the capital letter variant 'UTF-8'. In my case the character set came in as 'utf-8' which consequently destroyed the content of the body text.
Solution:
Modify the check to be case insensitive:
Code: [Select]
// And its 1252 variants
if (strcasecmp($charset, 'UTF-8') !== 0)

The last item was particularly tricky as I only stumbled accidentally over the use of some German "Umlaut" character which caused some mails to fail.

I have made this changes locally to my server. What is the correct way of bringing the suggested changes into future versions of ElkArte? On a lighter note, I feel a little perplexed that this has not been brought up so far. I wonder how the "add topic by email" feature could ever work correctly. Any opinion on this?

Re: [SOLVED] Help on Post/reply by email

Reply #2

Quote from: JPalmer – I have made this changes locally to my server. What is the correct way of bringing the suggested changes into future versions of ElkArte?
you can report on forum or also on github it's your choice

Quote from: JPalmer – On a lighter note, I feel a little perplexed that this has not been brought up so far. I wonder how the "add topic by email" feature could ever work correctly. Any opinion on this?
I don't have the knowledge to comment an discuss about coding, I'm not a coder but a simple forum admin.
Consider that most of forum admins are not using this feature and I guess this is the reason why it has not been reported until now, I manage more than one elkarte forum and I never used or tested this feature.
I'm pretty sure that @emanuele or @Spuds will asnswer soon
 
Thanks for your report
sorry for my bad english

Re: [SOLVED] Help on Post/reply by email

Reply #3

Quote from: radu81 –
Quote from: JPalmer – I have made this changes locally to my server. What is the correct way of bringing the suggested changes into future versions of ElkArte?
you can report on forum or also on github it's your choice

Thanks for your help! I have opened the respective issues on github,

Re: [SOLVED] Help on Post/reply by email

Reply #4

Quote from: JPalmer – Sorry for bringing this up once again ... I have followed closely the discussion here, and I followed the advice(s) set out there.

After some fiddling with setting up mail forwarding at my hosting provider and the path to php binary, I think that incoming emails are now piped to the emailtopic.php script. At least I can see them coming in at the ElkArte section for "Moderation -> Unapproved Emails".

And here is the problem ... I do not get any information out of this mails, e.g. no "Subject", no "From", no "Body Text". The system alerts "Email ID not in Database", which might be an aftereffect of not detecting a sender address. The user I'm sending from IS registered in the database.

On a lighter note, also sending a bounce message from the admin page does fail with "Session verification failed. Please try logging out and back in again, and then try again."

Please advice what I could do to narrow down the problem. Any help is highly appreciated!

EDIT: Mark as "solved" ...

I have run headsmack into the same problem (with v1.1.6). Replies from valid user with valid email address go to "moderation" as "Email ID not in Database"

After reading through this thread and the linked thread, I'm a bit confused regarding the fix, vs. troubleshooting the problem, or troubleshooting subsequent formatting issues once the "Email ID not in Database" issue is resolved.

Can the explicit fix for getting past the "Email ID not in Database" be articulated?

Thanks Muchly...

-Steeley

// Deep inside every dilemma lies a solution that involves explosives //

Re: [SOLVED] Help on Post/reply by email

Reply #5

Quote from: Steeley –
Quote from: JPalmer – Sorry for bringing this up once again ... I have followed closely the discussion here, and I followed the advice(s) set out there.

After some fiddling with setting up mail forwarding at my hosting provider and the path to php binary, I think that incoming emails are now piped to the emailtopic.php script. At least I can see them coming in at the ElkArte section for "Moderation -> Unapproved Emails".

And here is the problem ... I do not get any information out of this mails, e.g. no "Subject", no "From", no "Body Text". The system alerts "Email ID not in Database", which might be an aftereffect of not detecting a sender address. The user I'm sending from IS registered in the database.

On a lighter note, also sending a bounce message from the admin page does fail with "Session verification failed. Please try logging out and back in again, and then try again."

Please advice what I could do to narrow down the problem. Any help is highly appreciated!

EDIT: Mark as "solved" ...

I have run headsmack into the same problem (with v1.1.6). Replies from valid user with valid email address go to "moderation" as "Email ID not in Database"

After reading through this thread and the linked thread, I'm a bit confused regarding the fix, vs. troubleshooting the problem, or troubleshooting subsequent formatting issues once the "Email ID not in Database" issue is resolved.

Can the explicit fix for getting past the "Email ID not in Database" be articulated?

Thanks Muchly...

Hey Steeley, it has been a while, so that I had to get my memories together. For me the solution was to change the 3 source files mentioned earlier in this thread. I also have sent this changes to the github repository to have them included in future releases. Honestly, I don't know where this part of the story stands. I suspect the changes are not part of your v1.1.6.

Either way, I did do these changes in my installation and it works ever since. The html formatting of mails has room for improvement, but at least people can use the forum as kind of a mail-list. Please feel free to comment or ask questions.

Re: [SOLVED] Help on Post/reply by email

Reply #6

Quote from: JPalmer –
Hey Steeley, it has been a while, so that I had to get my memories together. For me the solution was to change the 3 source files mentioned earlier in this thread. I also have sent this changes to the github repository to have them included in future releases. Honestly, I don't know where this part of the story stands. I suspect the changes are not part of your v1.1.6.

Either way, I did do these changes in my installation and it works ever since. The html formatting of mails has room for improvement, but at least people can use the forum as kind of a mail-list. Please feel free to comment or ask questions.

...change the 3 source files mentioned earlier in this thread..

Bingo... Since the start of that message with those changes referred to "posting new topics" I wasn't sure if that was working the original generic reply via email issue or, once that was solved working subsequent new-topic formatting issues. Since I'm not a programmer, my coding experience is "basic'TM (figuratively and literally), and I play around with the code only when I know for sure that's what I need to do.

If you're not using 1.1.6, no - it's not incorporated in it either, and, like you, I was wondering how this hasn't been a bigger issue (it's fundamental to mail usage, after all).  I already ruled out email clients and various mail servers, perhaps the issue is unique to the server OS (linux / x86_64 / Apache / php 7.2)?

Anyway, I'm off to back up some files and edit some code.. if it works I owe ya a beer..


Last Edit: October 28, 2020, 02:52:59 am by Steeley

// Deep inside every dilemma lies a solution that involves explosives //

Re: [SOLVED] Help on Post/reply by email

Reply #7

YESSSSSSSS.... that does indeed solve the email reply posting problem.

Perversely however, now that email replies are posting, the sender gets a Mail-Delivery Failed message piped from emailpost.php..
(I could just tell users that's actually a "posting confirmation reply", but I think I'll go hunt it down and kill it instead..)

JP, next time you're up in the 'Pacific Northwet', you're not gonna be able to buy a beer..


// Deep inside every dilemma lies a solution that involves explosives //

 

Re: [SOLVED] Help on Post/reply by email

Reply #8

Quote from: Steeley – YESSSSSSSS.... that does indeed solve the email reply posting problem.

Perversely however, now that email replies are posting, the sender gets a Mail-Delivery Failed message piped from emailpost.php..
(I could just tell users that's actually a "posting confirmation reply", but I think I'll go hunt it down and kill it instead..)

JP, next time you're up in the 'Pacific Northwet', you're not gonna be able to buy a beer..



Deal! ;-)

Re: [SOLVED] Help on Post/reply by email

Reply #9

I was remiss for not posting the follow up on the fail message resolution. (Actually, I did post this resolution in this topic too.)

For the record,  the "Mail Delivery Failed"  message was being generated if "Guest" didn't have explicit permission to the board - emailpost.php accesses the forum as "guest". The emailed reply would post if the sender had access,  but the script would still generate a Fail message back at the sender if IT didn't have permission to access the board.

So, giving guest permission to all the boards solves the Fail message issue. Then, just setting guests unable to view the forum is sufficient to prevent unregistered folks from accessing the forum, and anyone sending email as an (unregistered) guest should be bounced by the member permissions settings.
Just have to make darn sure the members know not to forward forum messages around or their accounts could be compromised, but that's the nature of the beast regardless.

And JP, that is certainly a deal I look forward to. Thanks again.
Last Edit: November 21, 2020, 12:16:31 am by Steeley

// Deep inside every dilemma lies a solution that involves explosives //