Statistics.state.requestsRetries only updates after request is fully handled

[metalwarrior665]

Which package is this bug report for? If unsure which one to select, leave blank

None

Issue description

The requestsRetries statistics are only updated once the request is fully handled (succeeds or fails). This creates a false notion that requests are not erroring as much. The problem is made worse by that after error, requests are pushed at the end of the queue which means for a long time, there will be requestsRetries: 0 while significant requestsFinished. At the end of the Crawler.run, the retries will show correct number if all the requests were finished but imagine if the crawler gets .teardown or crashes.

I don't see any use-case for having the retries added only once handled so I consider this a bug.

See also run on Apify https://console.apify.com/view/runs/cbHWZTetxgehJuK3b

Code sample

const crawler = new CheerioCrawler({
    requestHandler: async ({ crawler, request }) => {
        const { requestsRetries } = crawler.stats.state;
        const { requestRetryHistogram } = crawler.stats;
        log.info(`${request.url}: Crawler retry count: ${requestsRetries}, Request retry histogram: ${JSON.stringify(requestRetryHistogram)}`);

        if (Math.random() < 0.5) {
            throw new Error('50% Random error');
        }
    },
});

const requests = [];
for (let i = 0; i < 100; i++) {
    requests.push({ url: `https://www.example.com/${i}` });
}
await crawler.run(requests);

Package version

3.11.5

Node.js version

20.12.2

Operating system

No response

Apify platform

• Tick me if you encountered this issue on the Apify platform

I have tested this on the next release

No response

Other context

No response