CaptureRedirect=true does not capture ANY redirects, ResponseUri = null

[alvipeo]

This code load the doc and checks for Redirects:

        AsyncRetryPolicy policyRetry6TimesWithBackoff = Policy.Handle<HttpRequestException>()
            .WaitAndRetryAsync(6, attempt => TimeSpan.FromSeconds(2 * Math.Pow(2, attempt)),
                (exception, calculatedDuration) => { _logger.LogWarning(exception, $"Retrying to get page again in {calculatedDuration.TotalSeconds} sec..."); });

        HtmlWeb htmlWeb = new()
        {
            CaptureRedirect = true
        };
        HtmlDocument? pageDoc = null;
        try
        {
            await policyRetry6TimesWithBackoff.ExecuteAsync(
                async ct => { pageDoc = await htmlWeb.LoadFromWebAsync(pageUri, Encoding.UTF8, null, outerCancellationToken); }, outerCancellationToken);
        }
        catch (HttpRequestException ex)
        {
            _logger.LogError(ex, $"Eventually failed loading page {pageUri}");
            throw;
        }

        if (pageDoc == null)
            throw new ApplicationException($"Failed to load page {pageUri}");

        bool responseRedirected = htmlWeb.StatusCode is HttpStatusCode.PermanentRedirect or HttpStatusCode.TemporaryRedirect or HttpStatusCode.Redirect or HttpStatusCode.RedirectKeepVerb;

        if (htmlWeb.StatusCode != HttpStatusCode.OK && !responseRedirected)
        {
            brokenUris.Add(pageUri.AbsoluteUri);
            _logger.LogInformation($"Broken page: {pageUri}, code: {htmlWeb.StatusCode}");
            return null;
        }

        switch (responseRedirected)
        {
            case true when htmlWeb.ResponseUri != null && htmlWeb.ResponseUri != pageUri:
                // the page may have Redirect, so the Uri may change
                _logger.LogInformation($"Loaded {pageUri} but it was redirected to {htmlWeb.ResponseUri}");
                pageUri = htmlWeb.ResponseUri;
                break;
            case true:
                _logger.LogInformation($"Page {pageUri} has a redirect. Skipping...");
                return null;
            default:
                _logger.LogInformation("Loaded");
                break;
        }

But it ALWAYS returns 200 instead of actual 301/308 (which I HAVE and SEE in chrome dev tools). This is very bad joke...

.NET 6.0, HtmlAgilityPack 1.11.43

[JonathanMagnan]

Hello @alvipeo ,

Could you provide the URL of the page that is currently not captured correctly?

Unfortunately, we might also be limited of what the HttpWebRequest class support or not but we will be happy to check it to make sure there is no alternative.

Best Regards,

Jon

---

Sponsorship
Help us improve this library

Performance Libraries
context.BulkInsert(list, options => options.BatchSize = 1000);
Entity Framework Extensions • Dapper Plus

Runtime Evaluation
Eval.Execute("x + y", new {x = 1, y = 2}); // return 3
C# Eval Function

[alvipeo]

https://actes-types.com/avocat/france/rhone-alpes for example. Lots of them.

[JonathanMagnan]

Hello @alvipeo ,

Could you try the v1.11.45 and let us know if the issue has been correctly fixed?

---

Is this library useful to you? Please help us by becoming a sponsor to keep it alive and supported.