[Bug] Using HttpCompletionOption.ResponseHeadersRead in HttpClient is throwing error as responce content size increase

[sasivishnu]

Description

We are using single instance HttpClient (Default HttpClient Implementation) in our app and SendAsync is passed with parameter HttpCompletionOption.ResponseHeadersRead. When response content size is less (ex: array of 10/20 Json objects), then API call is succeeding but if response content size is more, we are getting error in Android. We have released our app to production and encountered strange error in Android API call.

Could you please take a look & let me know what's going wrong?

Error Message:

Android.OS.NetworkOnMainThreadException: Exception of type 'Android.OS.NetworkOnMainThreadException' was thrown.
  at Java.Interop.JniEnvironment+InstanceMethods.CallIntMethod (Java.Interop.JniObjectReference instance, Java.Interop.JniMethodInfo method, Java.Interop.JniArgumentValue* args) [0x0006e] in <064cde681fba4dd08eaf8b1bdbbfb8c8>:0 
  at Java.Interop.JniPeerMembers+JniInstanceMethods.InvokeVirtualInt32Method (System.String encodedMember, Java.Interop.IJavaPeerable self, Java.Interop.JniArgumentValue* parameters) [0x0002a] in <064cde681fba4dd08eaf8b1bdbbfb8c8>:0 
  at Java.IO.InputStream.Read (System.Byte[] b, System.Int32 off, System.Int32 len) [0x00052] in <4a189ea3b82b48a089ac9002b2abc206>:0 
  at Android.Runtime.InputStreamInvoker.Read (System.Byte[] buffer, System.Int32 offset, System.Int32 count) [0x00006] in <4a189ea3b82b48a089ac9002b2abc206>:0 
  at System.IO.BufferedStream.Read (System.Byte[] array, System.Int32 offset, System.Int32 count) [0x000b9] in /Users/builder/jenkins/workspace/archive-mono/2019-08/android/release/external/corefx/src/System.Runtime.Extensions/src/System/IO/BufferedStream.cs:569 
  at System.Net.Http.DelegatingStream.Read (System.Byte[] buffer, System.Int32 offset, System.Int32 count) [0x00000] in /Users/builder/jenkins/workspace/archive-mono/2019-08/android/release/external/corefx/src/Common/src/System/IO/DelegatingStream.cs:88 
  at System.IO.StreamReader.ReadBuffer (System.Span`1[T] userBuffer, System.Boolean& readToUserBuffer) [0x000e5] in /Users/builder/jenkins/workspace/archive-mono/2019-08/android/release/external/corefx/src/Common/src/CoreLib/System/IO/StreamReader.cs:743 
  at System.IO.StreamReader.ReadSpan (System.Span`1[T] buffer) [0x00039] in /Users/builder/jenkins/workspace/archive-mono/2019-08/android/release/external/corefx/src/Common/src/CoreLib/System/IO/StreamReader.cs:390 
  at System.IO.StreamReader.Read (System.Char[] buffer, System.Int32 index, System.Int32 count) [0x00049] in /Users/builder/jenkins/workspace/archive-mono/2019-08/android/release/external/corefx/src/Common/src/CoreLib/System/IO/StreamReader.cs:364 
  at Newtonsoft.Json.JsonTextReader.ReadData (System.Boolean append, System.Int32 charsRequired) [0x00024] in <2073514815234917a5e8f91b0b239405>:0 
  at Newtonsoft.Json.JsonTextReader.ReadData (System.Boolean append) [0x00000] in <2073514815234917a5e8f91b0b239405>:0 
  at Newtonsoft.Json.JsonTextReader.ParseObject () [0x00057] in <2073514815234917a5e8f91b0b239405>:0 
  at Newtonsoft.Json.JsonTextReader.Read () [0x00053] in <2073514815234917a5e8f91b0b239405>:0 
  at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateObject (System.Object newObject, Newtonsoft.Json.JsonReader reader, Newtonsoft.Json.Serialization.JsonObjectContract contract, Newtonsoft.Json.Serialization.JsonProperty member, System.String id) [0x002b1] in <2073514815234917a5e8f91b0b239405>:0 
  at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject (Newtonsoft.Json.JsonReader reader, System.Type objectType, Newtonsoft.Json.Serialization.JsonContract contract, Newtonsoft.Json.Serialization.JsonProperty member, Newtonsoft.Json.Serialization.JsonContainerContract containerContract, Newtonsoft.Json.Serialization.JsonProperty containerMember, System.Object existingValue) [0x00161] in <2073514815234917a5e8f91b0b239405>:0 
  at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal (Newtonsoft.Json.JsonReader reader, System.Type objectType, Newtonsoft.Json.Serialization.JsonContract contract, Newtonsoft.Json.Serialization.JsonProperty member, Newtonsoft.Json.Serialization.JsonContainerContract containerContract, Newtonsoft.Json.Serialization.JsonProperty containerMember, System.Object existingValue) [0x0006d] in <2073514815234917a5e8f91b0b239405>:0 
  at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateList (System.Collections.IList list, Newtonsoft.Json.JsonReader reader, Newtonsoft.Json.Serialization.JsonArrayContract contract, Newtonsoft.Json.Serialization.JsonProperty containerProperty, System.String id) [0x00173] in <2073514815234917a5e8f91b0b239405>:0 
  at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateList (Newtonsoft.Json.JsonReader reader, System.Type objectType, Newtonsoft.Json.Serialization.JsonContract contract, Newtonsoft.Json.Serialization.JsonProperty member, System.Object existingValue, System.String id) [0x000dc] in <2073514815234917a5e8f91b0b239405>:0 
  at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal (Newtonsoft.Json.JsonReader reader, System.Type objectType, Newtonsoft.Json.Serialization.JsonContract contract, Newtonsoft.Json.Serialization.JsonProperty member, Newtonsoft.Json.Serialization.JsonContainerContract containerContract, Newtonsoft.Json.Serialization.JsonProperty containerMember, System.Object existingValue) [0x0007f] in <2073514815234917a5e8f91b0b239405>:0 
  at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.SetPropertyValue (Newtonsoft.Json.Serialization.JsonProperty property, Newtonsoft.Json.JsonConverter propertyConverter, Newtonsoft.Json.Serialization.JsonContainerContract containerContract, Newtonsoft.Json.Serialization.JsonProperty containerProperty, Newtonsoft.Json.JsonReader reader, System.Object target) [0x00065] in <2073514815234917a5e8f91b0b239405>:0 
  at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.PopulateObject (System.Object newObject, Newtonsoft.Json.JsonReader reader, Newtonsoft.Json.Serialization.JsonObjectContract contract, Newtonsoft.Json.Serialization.JsonProperty member, System.String id) [0x00280] in <2073514815234917a5e8f91b0b239405>:0 
  at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject (Newtonsoft.Json.JsonReader reader, System.Type objectType, Newtonsoft.Json.Serialization.JsonContract contract, Newtonsoft.Json.Serialization.JsonProperty member, Newtonsoft.Json.Serialization.JsonContainerContract containerContract, Newtonsoft.Json.Serialization.JsonProperty containerMember, System.Object existingValue) [0x00161] in <2073514815234917a5e8f91b0b239405>:0 
  at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal (Newtonsoft.Json.JsonReader reader, System.Type objectType, Newtonsoft.Json.Serialization.JsonContract contract, Newtonsoft.Json.Serialization.JsonProperty member, Newtonsoft.Json.Serialization.JsonContainerContract containerContract, Newtonsoft.Json.Serialization.JsonProperty containerMember, System.Object existingValue) [0x0006d] in <2073514815234917a5e8f91b0b239405>:0 
  at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize (Newtonsoft.Json.JsonReader reader, System.Type objectType, System.Boolean checkAdditionalContent) [0x000db] in <2073514815234917a5e8f91b0b239405>:0 
  at Newtonsoft.Json.JsonSerializer.DeserializeInternal (Newtonsoft.Json.JsonReader reader, System.Type objectType) [0x00054] in <2073514815234917a5e8f91b0b239405>:0 
  at Newtonsoft.Json.JsonSerializer.Deserialize (Newtonsoft.Json.JsonReader reader, System.Type objectType) [0x00000] in <2073514815234917a5e8f91b0b239405>:0 
  at Newtonsoft.Json.JsonSerializer.Deserialize[T] (Newtonsoft.Json.JsonReader reader) [0x00000] in <2073514815234917a5e8f91b0b239405>:0 
  at HttpTest.MainPage.MakeRequest[T] (System.String url, System.Net.Http.HttpCompletionOption responseHeadersRead) [0x00170] in C:\Users\vkumar1\source\repos\HttpTest\HttpTest\MainPage.xaml.cs:74 
  at HttpTest.MainPage.GetData (System.Net.Http.HttpCompletionOption responseContentRead, System.Boolean isSmallRequest) [0x0004d] in C:\Users\vkumar1\source\repos\HttpTest\HttpTest\MainPage.xaml.cs:98 
  at HttpTest.MainPage.Button_Clicked (System.Object sender, System.EventArgs e) [0x00050] in C:\Users\vkumar1\source\repos\HttpTest\HttpTest\MainPage.xaml.cs:55 
  --- End of managed Android.OS.NetworkOnMainThreadException stack trace ---
android.os.NetworkOnMainThreadException
	at android.os.StrictMode$AndroidBlockGuardPolicy.onNetwork(StrictMode.java:1513)
	at java.net.SocketInputStream.read(SocketInputStream.java:175)
	at java.net.SocketInputStream.read(SocketInputStream.java:144)
	at com.android.okhttp.okio.Okio$2.read(Okio.java:136)
	at com.android.okhttp.okio.AsyncTimeout$2.read(AsyncTimeout.java:211)
	at com.android.okhttp.okio.RealBufferedSource.read(RealBufferedSource.java:50)
	at com.android.okhttp.internal.http.Http1xStream$ChunkedSource.read(Http1xStream.java:439)
	at com.android.okhttp.okio.RealBufferedSource$1.read(RealBufferedSource.java:371)
	at mono.java.lang.RunnableImplementor.n_run(Native Method)
	at mono.java.lang.RunnableImplementor.run(RunnableImplementor.java:30)
	at android.os.Handler.handleCallback(Handler.java:873)
	at android.os.Handler.dispatchMessage(Handler.java:99)
	at android.os.Looper.loop(Looper.java:193)
	at android.app.ActivityThread.main(ActivityThread.java:6669)
	at java.lang.reflect.Method.invoke(Native Method)
	at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run(RuntimeInit.java:493)
	at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:858)

Same API call in our app with size of 100 array of json objects working fine when used with HttpCompletionOption.ResponseContentRead. So, there seems to be something going wrong when using HttpCompletionOption.ResponseHeadersRead.

As of now, iOS seems to be working. Only in Android, there is issue.

Steps to Reproduce

Try to receive large json object content size with HttpCompletionOption.ResponseHeadersRead option in HttpClient in Android. Attached reproduction repo sample to re-produce the issue. Even for smaller json object content size, it failing sometimes (harder to re-produce).

Code sample:

HttpRequestMessage request = new HttpRequestMessage(HttpMethod.Get, url);
            using (var response = await httpClient.SendAsync(request, HttpCompletionOption.ResponseHeadersRead, CancellationToken.None))
            {
                using (var stream = await response.Content.ReadAsStreamAsync())
                {
                    using (StreamReader reader = new StreamReader(stream))
                    using (JsonTextReader jsonReader = new JsonTextReader(reader))
                    {
                        return new JsonSerializer().Deserialize<T>(jsonReader);
                    }
                }
            }

Expected Behavior

Making GET Rest API call with HttpCompletionOption.ResponseHeadersRead should work fine and consistently across varying size of response content.

Actual Behavior

GET Rest API call with HttpCompletionOption.ResponseHeadersRead is throwing error when receiving large array of json objects.

Basic Information

• Version with issue: 4.5
• Last known good version: Unknown
• IDE:
• Platform Target Frameworks:
  • Android: Compiling with Android 9 SDK
• Nuget Packages:
• Affected Devices:

Screenshots

Reproduction Link

HttpTest.zip

Toggle to change API to fetch small number of items. By default (toggle false), it will fetch 50 objects (array) in Json. If you toggle to true and click the button, it will make API call to fetch 6 items in Json.

Workaround

Removing HttpCompletionOption.ResponseHeadersRead and using default HttpCompletionOption.ResponseContentRead works in Emulator.

[brendanzagaeski]

Cross-referencing note for the Xamarin.Android team

This issue appears similar to #4691

[grendello]

@sasivishnu The problem here is that while your request is made on a separate thread (the SendAsync call), you tell HttpClient not to fully download the request at that time (the HttpCompletionOption.ResponseHeadersRead option) and that download happens when you create and use the StreamReader - but it is back on the main/GUI thread, and thus Android throws the Java android/os/NetworkOnMainThreadException which we wrap in the managed Android.OS.NetworkOnMainThreadException one and throw it back to you. You have two options to fix the problem:

• either use HttpCompletionOption.ResponseContentRead (which might not be desirable as per your description)
• or call the deserializer (thus read the stream) in a separate task. Perhaps the deserializer has DeserializeAsync method? If not, just create an async method which will deserialize the stream in Task.Run and await on that method.

[sasivishnu]

@grendello thanks for quick check on this. Sorry for late reply (busy with personal life).

No DeserializeAsync method available in NewtonsoftJson. It was available in System.Text.Json but it won't work well in iOS (dotnet/runtime#31326).

Deserializing in seperate thread using Task.Run() did fix the issue. Sorry, I failed to notice that de-serialization was happening in main thread. I was checking every other reason except this :(

For anyone reference, I made below code change in above attached repo and it works.

        private async Task<T> MakeRequest<T>(string url, HttpCompletionOption responseHeadersRead)
        {
            HttpRequestMessage request = new HttpRequestMessage(HttpMethod.Get, url);
            using (var response = await httpClient.SendAsync(request, responseHeadersRead, CancellationToken.None))
            {
                using (var stream = await response.Content.ReadAsStreamAsync())
                {
                    return await Task.Run(() => DeserializeAsync<T>(stream));
                }
            }
        }

        private Task<T> DeserializeAsync<T>(Stream stream)
        {
            using (StreamReader reader = new StreamReader(stream))
            using (JsonTextReader jsonReader = new JsonTextReader(reader))
            {
                return Task.FromResult(new JsonSerializer().Deserialize<T>(jsonReader));
            }
        }