Azurite is incompatible with table API

[anmandav]

Which service(blob, file, queue, table) does this issue concern?

Table

Which version of the Azurite was used?

3.22.0

Where do you get Azurite? (npm, DockerHub, NuGet, Visual Studio Code Extension)

VisualStudio2022

What's the Node.js version?

NA

What problem was encountered?

Azurite seems to be somewhat incompatible with Table APi while storage emulator is. Since storage emulator is deprecated, it is not readily available on our build agents and azurite is not compatible.

Steps to reproduce the issue?

DurableTask framework is failing since table api is incompatible and this issue is same as previously mentioned issue #19 (Check ending discussion)
If possible, please provide the debug log using the -d parameter, replacing <pathtodebuglog> with an appropriate path for your OS, or review the instructions for docker containers:

-d "<pathtodebuglog>"

Please be sure to remove any PII or sensitive information before sharing!
The debug log will log raw request headers and bodies, so that we can replay these against Azurite using REST and create tests to validate resolution.

Have you found a mitigation/solution?

[blueww]

@anmandav

It looks the before issue is already resolved.
And the are many other customers also using Azurite table service, but not see this kind of issue.
So we will need more information to continue investigation.

Would you please share more details for this issue?

1. The Azurite debug log
2. The Expect behavior (responds) of Azurite
3. The actual behavior (responds) of Azurite
4. The detail repro steps:

• The command start Azurite (or how you start it)
• The step to send request and handle responds to Azurite

[anmandav]

@blueww

Repro'ed issue by running same setup using storage emulator and azurite. Adding details below:
We are running simple console app with below code with only one dependency(DTF) obtained from nuget -> "Microsoft.Azure.DurableTask.AzureStorage" version="1.14.0"

`
AzureStorageOrchestrationService storageService = new AzureStorageOrchestrationService(new AzureStorageOrchestrationServiceSettings{
StorageConnectionString = "UseDevelopmentStorage=true",
TaskHubName = "testAz",
PartitionCount = 1
});

        TaskHubWorker taskHubWorker = await new TaskHubWorker(storageService).AddTaskOrchestrations(typeof(TestOrchestration)).StartAsync();
        TaskHubClient taskHubClient = new TaskHubClient(storageService);

        var instance = await taskHubClient.CreateOrchestrationInstanceAsync(typeof(TestOrchestration), new SampleSchema { Id = 1, Name = "World" });
        var result = await taskHubClient.WaitForOrchestrationAsync(instance, TimeSpan.FromSeconds(10));
        Console.WriteLine(result.Output);

`

Response from storage emulator for the below request (http://127.0.0.1:10002/devstoreaccount1/testAzInstances?$filter=PartitionKey%20eq%20%27fd069298416245df8bfbab27a3ed8d50%27&$select=ExecutionId%2CName%2CVersion%2COutput%2CCustomStatus%2CCreatedTime%2CLastUpdatedTime%2CCompletedTime%2CRuntimeStatus%2CScheduledStartTime%2CGeneration%2CPartitionKey%2CRowKey%2CTimestamp%2CETag) is as follows:
Object Member: odata.metadata [Path with value [truncated]: /odata.metadata:http://127.0.0.1:10002/devstoreaccount1/$metadata#testAzInstances&$select=ExecutionId,Name,Version,Output,CustomStatus,CreatedTime,LastUpdatedTime,CompletedTime,RuntimeStatus,ScheduledStartTime] [Member with value [truncated]: odata.metadata:http://127.0.0.1:10002/devstoreaccount1/$metadata#testAzInstances&$select=ExecutionId,Name,Version,Output,CustomStatus,CreatedTime,LastUpdatedTime,CompletedTime,RuntimeStatus,ScheduledStartTim] String value [truncated]: http://127.0.0.1:10002/devstoreaccount1/$metadata#testAzInstances&$select=ExecutionId,Name,Version,Output,CustomStatus,CreatedTime,LastUpdatedTime,CompletedTime,RuntimeStatus,ScheduledStartTime,Generation,Partiti Key: odata.metadata [Path: /odata.metadata] Member: value Array Key: value [Path: /value]

Similar response for similar request (http://127.0.0.1:10002/devstoreaccount1/testAzInstances?$filter=PartitionKey%20eq%20%2751102896b2a445ae96bcba3d5334f020%27&$select=ExecutionId%2CName%2CVersion%2COutput%2CCustomStatus%2CCreatedTime%2CLastUpdatedTime%2CCompletedTime%2CRuntimeStatus%2CScheduledStartTime%2CGeneration%2CPartitionKey%2CRowKey%2CTimestamp%2CETag) from azurite:
Object Member: odata.metadata [Path with value: /odata.metadata:http://127.0.0.1:10002/devstoreaccount1/$metadata#Tables/@Element] [Member with value: odata.metadata:http://127.0.0.1:10002/devstoreaccount1/$metadata#Tables/@Element] String value: http://127.0.0.1:10002/devstoreaccount1/$metadata#Tables/@Element Key: odata.metadata [Path: /odata.metadata] Member: value Array Key: value [Path: /value]

If we see, the response body is having some wrong values which are being complained by DTF with exception:
The metadata URI 'http://127.0.0.1:10002/devstoreaccount1/$metadata#Tables/@Element' is not valid for the expected payload kind 'Feed'. ---> Microsoft.Data.OData.ODataException: The metadata URI 'http://127.0.0.1:10002/devstoreaccount1/$metadata#Tables/@Element' is not valid for the expected payload kind 'Feed'

Logs.zip

Adding logs zip folder where AzurtieDebug is azurtie debug logs and respective pcap files from Storage emulator and azurite emulator

[blueww]

@anmandav

Thanks for the responds!
Not sure if you have run this problem on the product Azure?

I have tried to run your program (with slightly change since build fail) on my machine, target both Azurite (version 3.25.1) and product Azure, both will fail with null reference.
The responds of both Azurite and product Azure looks similar.

So would you please see if the issue can repro on product Azure on your side, and since Azurite has a Table Query refactor recently, would you please also see if the issue repro on latest Azurite version (3.25.1)?

Product Azure:

GET https://[accountName].table.core.windows.net/testAzInstances?$filter=PartitionKey%20eq%20%27c0e970c75adc47ea9a6f9e0ff06556f4%27 HTTP/1.1
Host: [accountName].table.core.windows.net
Accept-Charset: UTF-8
MaxDataServiceVersion: 3.0;NetFx
Accept: application/json; odata=minimalmetadata
x-ms-client-request-id: 2c4e62ea-8894-4efb-ab46-ee47b4a7bef5
User-Agent: Azure-Storage/9.3.1 (.NET Core)
x-ms-version: 2018-03-28
x-ms-date: Mon, 14 Aug 2023 09:47:39 GMT
Authorization: SharedKey [accountName]:[hidden]


HTTP/1.1 200 OK
Cache-Control: no-cache
Transfer-Encoding: chunked
Content-Type: application/json;odata=minimalmetadata;streaming=true;charset=utf-8
Server: Windows-Azure-Table/1.0 Microsoft-HTTPAPI/2.0
x-ms-request-id: 9c920eca-0002-0069-1594-cee38e000000
x-ms-version: 2018-03-28
X-Content-Type-Options: nosniff
Date: Mon, 14 Aug 2023 09:48:31 GMT

2B6
{"odata.metadata":"https://[accountName].table.core.windows.net/$metadata#testAzInstances","value":[{"odata.etag":"W/\"datetime'2023-08-14T09%3A48%3A21.4338998Z'\"","PartitionKey":"c0e970c75adc47ea9a6f9e0ff06556f4","RowKey":"","Timestamp":"2023-08-14T09:48:21.4338998Z","Input":"{\"$type\":\"SampleSchema, ConsoleApp15\",\"Id\":1,\"Name\":\"World\"}","[email protected]":"Edm.DateTime","CreatedTime":"2023-08-14T09:47:27.0643634Z","Name":"DurableTask.Core.TaskOrchestration","Version":"","RuntimeStatus":"Pending","[email protected]":"Edm.DateTime","LastUpdatedTime":"2023-08-14T09:47:27.8326742Z","TaskHubName":"testAz","ExecutionId":"15dbf6d0e683471f95cd2faaf7591e61","Generation":0}]}
0

Azurite 3.25.1:

2023-08-14T09:41:22.780Z 8117a2ff-b69d-4bad-a106-697257a9d7fe info TableStorageContextMiddleware: RequestMethod=GET RequestURL=http://127.0.0.1/devstoreaccount1/testAzInstances?$filter=PartitionKey%20eq%20%271f3a45d353a24395bd5724866a28b162%27 RequestHeaders:{"host":"127.0.0.1:10002","accept-charset":"UTF-8","maxdataserviceversion":"3.0;NetFx","accept":"application/json; odata=minimalmetadata","x-ms-client-request-id":"c030a677-ff85-4697-8e7d-e107b354e29c","user-agent":"Azure-Storage/9.3.1 (.NET Core)","x-ms-version":"2018-03-28","x-ms-date":"Mon, 14 Aug 2023 09:41:22 GMT","authorization":"SharedKey devstoreaccount1:OKvFA/uMQv4RxWshKZwtsQpJ/K3Omlye5Xv8NwSimIQ="} ClientIP=127.0.0.1 Protocol=http HTTPVersion=1.1
2023-08-14T09:41:22.780Z 8117a2ff-b69d-4bad-a106-697257a9d7fe debug tableStorageContextMiddleware: Dispatch pattern string: /testAzInstances()
2023-08-14T09:41:22.780Z 8117a2ff-b69d-4bad-a106-697257a9d7fe info tableStorageContextMiddleware: Account=devstoreaccount1 tableName=testAzInstances
2023-08-14T09:41:22.780Z 8117a2ff-b69d-4bad-a106-697257a9d7fe verbose DispatchMiddleware: Dispatching request...
2023-08-14T09:41:22.780Z 8117a2ff-b69d-4bad-a106-697257a9d7fe info DispatchMiddleware: Operation=Table_QueryEntities
2023-08-14T09:41:22.780Z 8117a2ff-b69d-4bad-a106-697257a9d7fe verbose AuthenticationMiddlewareFactory:createAuthenticationMiddleware() Validating authentications.
2023-08-14T09:41:22.780Z 8117a2ff-b69d-4bad-a106-697257a9d7fe info TableSharedKeyLiteAuthenticator:validate() Start validation against account shared key authentication.
2023-08-14T09:41:22.780Z 8117a2ff-b69d-4bad-a106-697257a9d7fe info TableSharedKeyLiteAuthenticator:validate() Request doesn't include valid authentication header. Skip SharedKeyLite authentication.
2023-08-14T09:41:22.780Z 8117a2ff-b69d-4bad-a106-697257a9d7fe info TableSharedKeyAuthenticator:validate() Start validation against account shared key authentication.
2023-08-14T09:41:22.780Z 8117a2ff-b69d-4bad-a106-697257a9d7fe info TableSharedKeyAuthenticator:validate() [STRING TO SIGN]:"GET\n\n\nMon, 14 Aug 2023 09:41:22 GMT\n/devstoreaccount1/devstoreaccount1/testAzInstances"
2023-08-14T09:41:22.780Z 8117a2ff-b69d-4bad-a106-697257a9d7fe info TableSharedKeyAuthenticator:validate() Calculated authentication header based on key1: SharedKey devstoreaccount1:OKvFA/uMQv4RxWshKZwtsQpJ/K3Omlye5Xv8NwSimIQ=
2023-08-14T09:41:22.780Z 8117a2ff-b69d-4bad-a106-697257a9d7fe info TableSharedKeyAuthenticator:validate() Signature 1 matched.
2023-08-14T09:41:22.780Z 8117a2ff-b69d-4bad-a106-697257a9d7fe verbose DeserializerMiddleware: Start deserializing...
2023-08-14T09:41:22.780Z 8117a2ff-b69d-4bad-a106-697257a9d7fe info HandlerMiddleware: DeserializedParameters={"options":{"queryOptions":{"filter":"PartitionKey eq '1f3a45d353a24395bd5724866a28b162'"},"requestId":"c030a677-ff85-4697-8e7d-e107b354e29c"},"version":"2018-03-28"}
2023-08-14T09:41:22.781Z 8117a2ff-b69d-4bad-a106-697257a9d7fe debug TableHandler:queryEntities() Raw response string is "{\"odata.metadata\":\"http://127.0.0.1:10002/devstoreaccount1/$metadata#Tables/@Element\",\"value\":[{\"odata.etag\":\"W/\\\"datetime'2023-08-14T09%3A41%3A14.3762475Z'\\\"\",\"PartitionKey\":\"1f3a45d353a24395bd5724866a28b162\",\"RowKey\":\"\",\"Input\":\"{\\\"$type\\\":\\\"SampleSchema, ConsoleApp15\\\",\\\"Id\\\":1,\\\"Name\\\":\\\"World\\\"}\",\"[email protected]\":\"Edm.DateTime\",\"CreatedTime\":\"2023-08-14T09:41:13.8598841Z\",\"Name\":\"DurableTask.Core.TaskOrchestration\",\"Version\":\"\",\"RuntimeStatus\":\"Pending\",\"[email protected]\":\"Edm.DateTime\",\"LastUpdatedTime\":\"2023-08-14T09:41:14.3193085Z\",\"TaskHubName\":\"testAz\",\"ExecutionId\":\"4cf9f7cec88146ee8a6353b1ac06193a\",\"Generation\":0,\"Timestamp\":\"2023-08-14T09:41:14.3762475Z\"}]}"
2023-08-14T09:41:22.781Z 8117a2ff-b69d-4bad-a106-697257a9d7fe verbose SerializerMiddleware: Start serializing...
2023-08-14T09:41:22.781Z 8117a2ff-b69d-4bad-a106-697257a9d7fe info Serializer: Start returning stream body.
2023-08-14T09:41:22.781Z 8117a2ff-b69d-4bad-a106-697257a9d7fe info EndMiddleware: End response. TotalTimeInMS=1 StatusCode=200 StatusMessage=OK Headers={"server":"Azurite-Table/3.25.1","content-type":"application/json;odata=minimalmetadata","x-ms-client-request-id":"c030a677-ff85-4697-8e7d-e107b354e29c","x-ms-request-id":"8117a2ff-b69d-4bad-a106-697257a9d7fe","x-ms-version":"2023-01-03","date":"Mon, 14 Aug 2023 09:41:22 GMT"}

[ericle]

Seems like this may be a problem of Azurite not working with the OLD table API, but it does work with the NEW table API.

I was able to repro this problem -- "The metadata URI 'http://127.0.0.1:10002/devstoreaccount1/$metadata#Tables/@Element' is not valid for the expected payload kind 'Feed'" - with code using WindowsAzure.Storage 3.1.0.1. The code looks like this:

                TableQuerySegment<DynamicTableEntity> segment =
                    table.ExecuteQuerySegmented(tableQuery, continuationToken);
                foreach (DynamicTableEntity row in segment)
                {
                    ...
                }

The query is as follows (I selected 4 columns from the table):

    (PartitionKey eq '103') and (RowKey ge 'c7FFFFF36eFFBFi7FFFFEBD') and (RowKey lt 'c7FFFFF36eFFBFi7FFFFEBE')

However, the following code, with the same query, using Azure.Data.Tables 12.8.1, works fine:

    table.Query<Azure.Data.Tables.TableEntity>(filter: ..., maxPerPage: ...)

Issues like this are what finally got me to stop using the old Azure API and upgrade to the new one. (The upgrade is painful because the APIs are pretty different.)