HttpClient / HttpWebRequest cannot download more than 2372B of file

[vtelenak]

Target name(s)

ESP32_REV0, ESP32_REV3

Firmware version

No response

Was working before? On which version?

No response

Device capabilities

System Information
HAL build info: nanoCLR running @ ESP32 built with ESP-IDF e7771c7
Target: ESP32_REV0
Platform: ESP32

Firmware build Info:
Date: Jun 17 2024
Type: MinSizeRel build, chip rev. >= 0, without support for PSRAM
CLR Version: 1.9.1.243
Compiler: GNU ARM GCC v12.2.0

OEM Product codes (vendor, model, SKU): 0, 0, 0

Serial Numbers (module, system):
00000000000000000000000000000000
0000000000000000

Target capabilities:
Has nanoBooter: NO
IFU capable: NO
Has proprietary bootloader: YES

AppDomains:

Assemblies:
NFAppAPTest, 1.0.0.0
mscorlib, 1.15.6.0
System.IO.FileSystem, 1.1.47.0
nanoFramework.System.Collections, 1.5.31.0
System.Device.Gpio, 1.1.41.0
System.Net.Http, 1.5.138.0
nanoFramework.Logging, 1.1.100.0
nanoFramework.Runtime.Native, 1.6.12.0
System.Threading, 1.1.32.63105
nanoFramework.System.Runtime, 1.0.27.0
nanoFramework.Runtime.Events, 1.11.18.0
nanoFramework.System.Text, 1.2.54.0
System.IO.Streams, 1.1.59.0
System.Net, 1.10.79.0

Native Assemblies:
mscorlib v100.5.0.19, checksum 0x445C7AF9
nanoFramework.Runtime.Native v100.0.9.0, checksum 0x109F6F22
nanoFramework.Hardware.Esp32 v100.0.10.0, checksum 0x6A20A689
nanoFramework.Hardware.Esp32.Rmt v100.0.4.0, checksum 0x608C5658
nanoFramework.Device.OneWire v100.0.4.0, checksum 0xB95C43B4
nanoFramework.Networking.Sntp v100.0.4.4, checksum 0xE2D9BDED
nanoFramework.ResourceManager v100.0.0.1, checksum 0xDCD7DF4D
nanoFramework.System.Collections v100.0.1.0, checksum 0x2DC2B090
nanoFramework.System.Text v100.0.0.1, checksum 0x8E6EB73D
nanoFramework.System.IO.Hashing v100.0.0.1, checksum 0xEBD8ED20
nanoFramework.System.Security.Cryptography v100.0.0.2, checksum 0xF4AEFE6C
nanoFramework.Runtime.Events v100.0.8.0, checksum 0x0EAB00C9
EventSink v1.0.0.0, checksum 0xF32F4C3E
System.IO.FileSystem v1.1.0.0, checksum 0xCC556D24
System.Math v100.0.5.5, checksum 0x9F9E2A7E
System.Net v100.2.0.1, checksum 0xD82C1452
System.Device.Adc v100.0.0.0, checksum 0xE5B80F0B
System.Device.Dac v100.0.0.6, checksum 0x02B3E860
System.Device.Gpio v100.1.0.6, checksum 0x097E7BC5
System.Device.I2c v100.0.0.2, checksum 0xFA806D33
System.Device.I2c.Slave v1.0.0.0, checksum 0x4238164B
System.Device.I2s v100.0.0.1, checksum 0x478490FE
System.Device.Pwm v100.1.0.4, checksum 0xABF532C3
System.IO.Ports v100.1.6.1, checksum 0xB798CE30
System.Device.Spi v100.1.2.0, checksum 0x3F6E2A7E
System.Runtime.Serialization v100.0.0.0, checksum 0x0A066871
System.Device.Wifi v100.0.6.4, checksum 0x00A058C6
Windows.Storage v100.0.3.0, checksum 0xF0C37E1B

++++++++++++++++++++++++++++++++
++ Memory Map ++
++++++++++++++++++++++++++++++++
Type Start Size
++++++++++++++++++++++++++++++++
RAM 0x3ffe46bc 0x0001b000
FLASH 0x00000000 0x00400000

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
++ Flash Sector Map ++
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
Region Start Blocks Bytes/Block Usage
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
0 0x00010000 1 0x1B0000 nanoCLR
1 0x001C0000 1 0x1E0000 Deployment
2 0x003C0000 1 0x040000 Configuration

+++++++++++++++++++++++++++++++++++++++++++++++++++
++ Storage Usage Map ++
+++++++++++++++++++++++++++++++++++++++++++++++++++
Start Size (kB) Usage
+++++++++++++++++++++++++++++++++++++++++++++++++++
0x003C0000 0x040000 (256kB) Configuration
0x00010000 0x1B0000 (1728kB) nanoCLR
0x001C0000 0x1E0000 (1920kB) Deployment

Deployment Map
Empty

Description

HttpWebRequest downloads a 2372B file and then stream.Read returns only 0.

HttpClient never finishes downloading.

How to reproduce

example below

DownloadFileA - HttpClient - https://www.nuget.org/packages/nanoFramework.System.Net.Http.Client/1.5.113

DownloadFileB - HttpWebRequest - https://github.com/nanoframework/Samples/blob/main/samples/HTTP/HttpWebRequest/Program.cs

Expected behaviour

No response

Screenshots

No response

Aditional information

public class Program {

public static void Main() {
    DownloadFileB("https://letsencrypt.org/docs/integration-guide/", "I:\\test.txt");
}


private const string LetsEncryptCARootCertificate = "-----BEGIN CERTIFICATE-----\r\nMIIFazCCA1OgAwIBAgIRAIIQz7DSQONZRGPgu2OCiwAwDQYJKoZIhvcNAQELBQAw\r\nTzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh\r\ncmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwHhcNMTUwNjA0MTEwNDM4\r\nWhcNMzUwNjA0MTEwNDM4WjBPMQswCQYDVQQGEwJVUzEpMCcGA1UEChMgSW50ZXJu\r\nZXQgU2VjdXJpdHkgUmVzZWFyY2ggR3JvdXAxFTATBgNVBAMTDElTUkcgUm9vdCBY\r\nMTCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAK3oJHP0FDfzm54rVygc\r\nh77ct984kIxuPOZXoHj3dcKi/vVqbvYATyjb3miGbESTtrFj/RQSa78f0uoxmyF+\r\n0TM8ukj13Xnfs7j/EvEhmkvBioZxaUpmZmyPfjxwv60pIgbz5MDmgK7iS4+3mX6U\r\nA5/TR5d8mUgjU+g4rk8Kb4Mu0UlXjIB0ttov0DiNewNwIRt18jA8+o+u3dpjq+sW\r\nT8KOEUt+zwvo/7V3LvSye0rgTBIlDHCNAymg4VMk7BPZ7hm/ELNKjD+Jo2FR3qyH\r\nB5T0Y3HsLuJvW5iB4YlcNHlsdu87kGJ55tukmi8mxdAQ4Q7e2RCOFvu396j3x+UC\r\nB5iPNgiV5+I3lg02dZ77DnKxHZu8A/lJBdiB3QW0KtZB6awBdpUKD9jf1b0SHzUv\r\nKBds0pjBqAlkd25HN7rOrFleaJ1/ctaJxQZBKT5ZPt0m9STJEadao0xAH0ahmbWn\r\nOlFuhjuefXKnEgV4We0+UXgVCwOPjdAvBbI+e0ocS3MFEvzG6uBQE3xDk3SzynTn\r\njh8BCNAw1FtxNrQHusEwMFxIt4I7mKZ9YIqioymCzLq9gwQbooMDQaHWBfEbwrbw\r\nqHyGO0aoSCqI3Haadr8faqU9GY/rOPNk3sgrDQoo//fb4hVC1CLQJ13hef4Y53CI\r\nrU7m2Ys6xt0nUW7/vGT1M0NPAgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBBjAPBgNV\r\nHRMBAf8EBTADAQH/MB0GA1UdDgQWBBR5tFnme7bl5AFzgAiIyBpY9umbbjANBgkq\r\nhkiG9w0BAQsFAAOCAgEAVR9YqbyyqFDQDLHYGmkgJykIrGF1XIpu+ILlaS/V9lZL\r\nubhzEFnTIZd+50xx+7LSYK05qAvqFyFWhfFQDlnrzuBZ6brJFe+GnY+EgPbk6ZGQ\r\n3BebYhtF8GaV0nxvwuo77x/Py9auJ/GpsMiu/X1+mvoiBOv/2X/qkSsisRcOj/KK\r\nNFtY2PwByVS5uCbMiogziUwthDyC3+6WVwW6LLv3xLfHTjuCvjHIInNzktHCgKQ5\r\nORAzI4JMPJ+GslWYHb4phowim57iaztXOoJwTdwJx4nLCgdNbOhdjsnvzqvHu7Ur\r\nTkXWStAmzOVyyghqpZXjFaH3pO3JLF+l+/+sKAIuvtd7u+Nxe5AW0wdeRlN8NwdC\r\njNPElpzVmbUq4JUagEiuTDkHzsxHpFKVK7q4+63SM1N95R1NbdWhscdCb+ZAJzVc\r\noyi3B43njTOQ5yOf+1CceWxG1bQVs5ZufpsMljq4Ui0/1lvh+wjChP4kqKOJ2qxq\r\n4RgqsahDYVvTH9w7jXbyLeiNdd8XM2w9U/t7y0Ff/9yi0GE44Za4rF2LN9d11TPA\r\nmRGunUHBcnWEvgJBQl9nJEiU0Zsnvgc/ubhPgXRR4Xq37Z0j4r7g1SgEEzwxA57d\r\nemyPxgcYxn/eR44/KJ4EBs+lVDR3veyJm+kXQ99b21/+jh5Xos1AnX5iItreGCc=\r\n-----END CERTIFICATE-----";

internal static void DownloadFileA(string url, string filePath) {
    var httpClient = new HttpClient();
    httpClient.HttpsAuthentCert = new X509Certificate(LetsEncryptCARootCertificate);
    httpClient.SslProtocols = System.Net.Security.SslProtocols.Tls12;

    HttpResponseMessage response = httpClient.Get(url);
    response.EnsureSuccessStatusCode();

    using FileStream fs = new FileStream(filePath, FileMode.Create, FileAccess.Write);
    response.Content.ReadAsStream().CopyTo(fs);
}

internal static void DownloadFileB(string url, string filePath) {

    var httpWebRequest = (HttpWebRequest)WebRequest.Create(url);
    httpWebRequest.Method = "GET";

    //////////////////////////////////////////////////////////////////////
    // need to set the SSL protocol that the connection is going to use //
    // *** this MANDATORY otherwise the authentication will fail ***    //
    //////////////////////////////////////////////////////////////////////
    httpWebRequest.SslProtocols = System.Net.Security.SslProtocols.Tls12;

    // if the request is to a secured server we need to make sure that we either:
    // 1. provide the root CA certificate 
    // 2. the device has already stored a root CA bundle that will use when performing the authentication
    httpWebRequest.HttpsAuthentCert = new X509Certificate(LetsEncryptCARootCertificate);

    int totalBytesRead = 0;

    // get the response as a HttpWebResponse
    // wrap the response object with a using statement to make sure that it's disposed
    using (var httpWebResponse = (HttpWebResponse)httpWebRequest.GetResponse()) {
        // wrap the response stream on a using statement to make sure that it's disposed
        using (var stream = httpWebResponse.GetResponseStream()) {
            // read response in chunks of 1k

            byte[] buffer = new byte[1024];
            int bytesRead = 0;

            Debug.WriteLine("Http response follows");
            Debug.WriteLine(">>>>>>>>>>>>>");

            using FileStream fs = new FileStream(filePath, FileMode.Create, FileAccess.Write);

            do {
                bytesRead = stream.Read(buffer, 0, buffer.Length);

                if (bytesRead > 0) {
                    totalBytesRead += bytesRead;
                    //Debug.Write(Encoding.UTF8.GetString(buffer, 0, bytesRead));

                    fs.Write(buffer, 0, bytesRead);
                }

                Debug.WriteLine("bytes read " + bytesRead);
            }
            while (totalBytesRead < httpWebResponse.ContentLength);

            fs.Flush();
            fs.Close();
        }

        Debug.WriteLine(">>>>>>>>>>>>>");
        Debug.WriteLine("End of Http response");
        Debug.WriteLine($"Read {totalBytesRead} bytes");
    }
}

}

[Ellerbach]

You are in a limited environment, constraints with CPU and memory. You are most likely hitting the memory allocation for those kind of tasks. If you need to read a very large file, then read by chunks like you are doing but write by chunks as well, don't feel up the memory like you are currently doing.

[vtelenak]

And how do I perceive memory now? It is read into a block that is 1KB and it is immediately stored on the Flash memory. First, it is an approach published in nanoframework samples. I need to do an OTA update and I am based on the article https://www.nanoframework.net/over-the-air-net-nanoframework-code-update-using-azure-iot/ I understand that I cannot work with large files, but over 2KB surely pe file with update will have.

[Ellerbach]

Adjust how you read the file, download chunk of the file and save it, download the next part. Exactly as you would do when you have a long list and you break it into pages.