Incorrect Message Date and Time - Messages Out of Order

[lawfulsoftware]

1.8.97 (Docker 1.17.25)

A reply to an email appears before the original email. The original email showed "1 min from now" (i.e., in the future).

The email headers include the following:

I suspect that the sender's system clock is somehow incorrect. Outlook correctly displays the time 13:49 PDT however FreeScout shows the message time as 13:56 PDT which explains why our reply at 13:54 PDT appears before the original email.

FreeScout is set to "Use date and time from mail headers". Can FreeScout use a more reliable time from the headers?

[freescout-helpdesk]

Unfortunately not. We'll remove this option from Mail Settings and allow to change it only via the parameter in .env file: APP_USE_MAIL_DATE_ON_FETCHING=true

It's recommended to use it only if you need to import past emails from your mail server.

[lawfulsoftware]

It does not need to be hidden in an environment variable and can still be useful if either FreeScout or the mail server is offline. Using the fetch time is likely to cause confusion if the fetch is delayed for any reason.

Is it possible to parse the headers and simply choose the earliest date and time?

[freescout-helpdesk]

No. Sender's mail server may specify absolutely any date in it's headers.

If this feature will continue causing troubles we'll remove it completely.

[lawfulsoftware]

It is an important feature and there are many possible solutions.

• Warn the administrator of this potential scenario prior to implementation
• Look for headers added by reliable email services such as Outlook and Gmail
• Reject dates that are in the future and use a date in the header that is in the past
• Look for three or more dates in the header and use the two that are closest together
• In cases where emails are out of order, allow the user to select the date that makes the most sense

https://stackoverflow.com/questions/35471184/why-do-received-email-headers-seem-to-be-out-of-order-chronologically

The most relevant date and time is when the recipient's mail server received the email. If the IMAP server is well known, it can be automatically detected based on the fetch settings and the proper header can be extracted.

[maltem-za]

@freescout-helpdesk I can kind of understand why you disabled the option and I'm not suggesting to change the current behaviour (datetime displayed in UI = fetch time). However, it could really be useful to be able to quickly see one of the datetimes from the headers, even knowing it could be wrong. I say "one of" because of course there are multiple we could choose from. The Sender Time Zone module already does something similar and uses the Date header. (By the way, that page states “Sender” here means the mail server which sent the email, but isn't the Date header set by the sender's mail client? With webmail client and server are basically the same of course...)

It seems to me like using the topmost Received header to (perhaps optionally) display the "actual" receipt datetime somewhere would be the most reliable option, as that's the server "closest" to the FreeScout instance. Yes, it could still be wrong, but would probably be least likely to be wrong. This could be displayed in a similar way to what the Sender Time Zone module is doing, for example. Potentially the APP_USE_MAIL_DATE_ON_FETCHING option could be improved and re-enabled in the UI in this way too.

I think this is also what @lawfulsoftware meant with "The most relevant date and time is when the recipient's mail server received the email". Personally I would avoid looking for headers added by "reliable" email services though (because it involves inspecting domain names, which can change quickly or can be spoofed, and who decides which services are reliable?). Similar for detecting the recipient mail server, which I don't think is necessary anyway as the headers are ordered. Trying to select the "best" date from the headers based on some rules also gets complicated quickly, and again, I think it can be much simpler than that.

I would be interested to hear your thoughts on this :)

[freescout-help]

It can be done via a custom module.