Delta create_checkpoint() breaks Synapse SQL OPENROWSET #1611
Replies: 1 comment
-
@Trebla2010 unfortunately this seems like it might be the Synapse SQL delta integration having a bug in how it's handling the Delta protocol The
Sorry if this isn't the answer you were hoping for but our princess is in another castle |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
After running create_checkpoint(), I'm unable to run OPENROWSET. Getting below error.
I'm running this on Azure Synapse Serverless SQL.
SELECT
TOP 100 *
FROM
OPENROWSET(
BULK 'https://devstoragereporting.dfs.core.windows.net/enriched/TestData/',
FORMAT = 'DELTA'
) AS [result]
Resolving Delta logs on path 'ADLSProcessing/staging/TestData/' failed with error: Cannot parse field field 'parts' in json object.
I found that the file "_last_checkpoint" that gets created into the delta_log folder is what's breaking Synapse SQL. Fortunately, deleting this file restores functionality to my delta table and I keep the performance gain.
What is the purpose of the _last_checkpoint file? Is there a way to prevent this file from being generated?
Below is the content inside the _last_checkpoint file.
{"num_of_add_files":null,"parts":null,"size":132,"size_in_bytes":77024,"version":89}
Beta Was this translation helpful? Give feedback.
All reactions