Delay in fetching emails from O365

[nhatznguyen]

I'm not a programmer/developer, so please don't roast me for trying to figure out why lately there's been a delay in fetching emails to all 30 mailboxes that we have. The delay time could be from 1 hour - 2 hours, I changed the fetch interval to 3 minutes, but nothing changed. When I do manual fetch, sometimes it fetches on the first or I have to do a second fetch. Every time I do manual fetch and got new emails, on the fetch log, I get this error message for all mailboxes.

Error fetching email | Error: connection setup failed; File: /var/www/xxxxxxxxxxxx/vendor/webklex/php-imap/src/Client.php (391)

Fetching emails setting:
IMAP
outlook.office365.com
993
OAuth credential
TLS

PHP version: 8.2.10
FreeScout version: 1.8.139
Database: MySQL
Are you using CloudFlare: Yes
Are you using non-official modules: No

When there's only a few mailboxes, there's no issue with delay. Is it because there's so many mailboxes that created the delay issue? If that's the case, what can I do to fix the delay for all mailboxes? All advises are welcome. Thank you in advance.

[freescout-help]

Try to increase Fetching Interval in "Manage > Settings > Mail Settings".

Try to increase the sleep time between connections in https://github.com/freescout-help-desk/freescout/blob/dist/app/Console/Commands/FetchEmails.php#L101. Try 100000, 200000 and so on.

[nhatznguyen]

Try to increase Fetching Interval in "Manage > Settings > Mail Settings".

Try to increase the sleep time between connections in https://github.com/freescout-help-desk/freescout/blob/dist/app/Console/Commands/FetchEmails.php#L101. Try 100000, 200000 and so on.

Thank you for the input. I increased to 5 seconds and that does help a lot. The fetch interval time is set for 5 minutes. Since the connection sleep time increased to 5 seconds, I noticed that the new mails got fetch at a random interval, sometime within 5 minutes, sometime within 15 minutes. Should I increase the sleep time more to keep the fetch time within 5 minutes of arrival?

[freescout-help]

Should I increase the sleep time more to keep the fetch time within 5 minutes of arrival?

No. Few seconds (5 seconds) is more than enough.

Now try to decrease connection sleep time: 4 seconds, 3 seconds, 2 seconds, 1 seconds and less and check how it goes.

[freescout-help]

Try to update FreeScout to the latest version and try this /app/Console/Commands/FetchEmails.php file and let us know the results.

[nhatznguyen]

Thank you for the advice. I have forwarded everything to the dev. I will update the status next week once the dev reply back.

[nhatznguyen]

Try to update FreeScout to the latest version and try this /app/Console/Commands/FetchEmails.php file and let us know the results.

so my dev replaced the FetchEmails.php file to this one this morning. I have been monitoring one of the mailboxes, and the email fetching is still delayed. In one instance where it's 15 minutes delay, in another instance there are 3 emails came in to the mailbox around 5-10 minutes apart but on FS those 3 emails got fetch at the same time which is almost 10 minutes from the last received email, like emails came in at 1451, 1455, 1504, got fetch on FS at 1512. And another instance where for almost an hour, no email was fetched till I do manual fetch.

[freescout-help]

In one instance where it's 15 minutes delay, in another instance there are 3 emails came in to the mailbox around 5-10 minutes apart but on FS those 3 emails got fetch at the same time which is almost 10 minutes from the last received email, like emails came in at 1451, 1455, 1504, got fetch on FS at 1512.

It's not quite clear what you are saying.

Are you still receiving "Error: connection setup failed" errors?
Set Fetching Interval back to 1 minute in "Manage > Settings > Mail Settings" and check how it goes.

And another instance where for almost an hour, no email was fetched till I do manual fetch.

This looks like whole another story. Again logs need to be checked.

[nhatznguyen]

In one instance where it's 15 minutes delay, in another instance there are 3 emails came in to the mailbox around 5-10 minutes apart but on FS those 3 emails got fetch at the same time which is almost 10 minutes from the last received email, like emails came in at 1451, 1455, 1504, got fetch on FS at 1512.

It's not quite clear what you are saying.

Are you still receiving "Error: connection setup failed" errors? Set Fetching Interval back to 1 minute in "Manage > Settings > Mail Settings" and check how it goes.

And another instance where for almost an hour, no email was fetched till I do manual fetch.

This looks like whole another story. Again logs need to be checked.

My apology for any confusion.

In the Fetch Error Log, yesterday it showed "Error: connection setup failed" for all mailboxes, today it showed "Error: flag could not be set" on one mailbox and no other error.

I have changed the Fetching Interval to 1 minute per advise.

I sent a test email to one of the mailbox around 10:53am, on Outlook it received 10:54am, on FS as of 11:05am still not show test email.

[nhatznguyen]

should I change these values to increase more time?

const MAX_SLEEP = 500000;

    // https://github.com/freescout-helpdesk/freescout/issues/2563
    // Add small delay between connections to avoid blocking by mail servers,
    // especially when there many mailboxes.
    // Microseconds: 1 second = 1 000 000 microseconds.
    $sleep = 20000;

$sleep += 20000;
if ($sleep > self::MAX_SLEEP) {
$sleep = self::MAX_SLEEP;
}

             // If mail server starts to block the connection
            // (when there are many mailboxes for example),
            // we increase connection sleep time and retry after sleep.
            // https://github.com/freescout-help-desk/freescout/issues/4227
            if (trim($e->getMessage()) == 'connection setup failed') {
                $sleep += 200000;

[nhatznguyen]

I'm still experiencing the fetching delay with mailboxes on FS. Please provide assistance.

[freescout-help]

Try to increase this sleep value: https://github.com/freescout-help-desk/freescout/blob/dist/app/Console/Commands/FetchEmails.php#L144 and const MAX_SLEEP = 500000; value.

[nhatznguyen]

Try to increase this sleep value: https://github.com/freescout-help-desk/freescout/blob/dist/app/Console/Commands/FetchEmails.php#L144 and const MAX_SLEEP = 500000; value.

I changed the values according to the advice. I will post an update tomorrow.

[freescout-help]

Try to set $sleep += 500000; for example and see how it goes.

[nhatznguyen]

Try to increase this sleep value: https://github.com/freescout-help-desk/freescout/blob/dist/app/Console/Commands/FetchEmails.php#L144 and const MAX_SLEEP = 500000; value.

Made this change, and so far still no improvement.

Try to set $sleep += 500000; for example and see how it goes.

Just made the change, will update tomorrow.

[nhatznguyen]

Try to set $sleep += 500000; for example and see how it goes.

after this change and also increased resources on the MySQL database, as of now the fetching seem to be normal, continue to monitor for the rest of today, will update again tomorrow

[freescout-help]

Ok. Can you also try to with the previous resources on the MySQL database. Otherwise we won't know if $sleep += 500000 helped or not.

[nhatznguyen]

Ok. Can you also try to with the previous resources on the MySQL database. Otherwise we won't know if $sleep += 500000 helped or not.

so, everything is still good from my last reply, unfortunately I'm not able to reverse MySQL resources, so your request can't be done, thank you very much for all your advises, much appreciate.

[freescout-help]

Try to return $sleep += 500000 back to $sleep += 200000 and check if the delay will return.