Preserve column ordering #14
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This confirms that we need a new approach to creating the fields in the dataframe
The JSON records that DuckDB dumps are actually ordered maps. The object keys (and values) appear in the order that they were requested in the SELECT statement. Although JSON has no concept of ordering within objects, this is the simplest and quickest way to fix this problem. I'll try to investigate some other avenues for fixing this one day in the future. @ryantxu suggested a couple of things we could try: - We can implement a JSON parser that goes straight to data frame (like the one in the sdk) - or better may be using parquet output. if named based on a hash this could also be a cache system 'for free' Also note, there's another orderedmap Go package we could have used: https://github.com/wk8/go-ordered-map They seem to have a similar number of forks and stars on Github, and I just happened across the iancoleman one first.
samjewell
force-pushed
the
sj/preserve-column-ordering
branch
from
386ad93
to
daa82bb
Compare
scottlepp
approved these changes
Aug 8, 2024
samjewell
added a commit
to grafana/grafana
that referenced
this pull request
Aug 12, 2024
This fixed ordering of the columns **What is this feature, why do we need it?** See scottlepp/go-duck#14 for a description of the improvement we're including here **Who is this feature for?** Anyone who uses SQL Expressions. This is still an experimental feature, and only used by a very small number of instances.
mildwonkey
pushed a commit
to grafana/grafana
that referenced
this pull request
Aug 12, 2024
* Bump to go-duck v0.1.0 This fixed ordering of the columns **What is this feature, why do we need it?** See scottlepp/go-duck#14 for a description of the improvement we're including here **Who is this feature for?** Anyone who uses SQL Expressions. This is still an experimental feature, and only used by a very small number of instances. * Run `make update-workspace`
giorgioatmerqury
pushed a commit
to cybermerqury/grafana
that referenced
this pull request
Aug 21, 2024
* Bump to go-duck v0.1.0 This fixed ordering of the columns **What is this feature, why do we need it?** See scottlepp/go-duck#14 for a description of the improvement we're including here **Who is this feature for?** Anyone who uses SQL Expressions. This is still an experimental feature, and only used by a very small number of instances. * Run `make update-workspace`
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Currently SQL Expressions returns a data frame with columns that are in a different order to the order the user requested.
Not only is this a poor UX, it also blocks us from doing more exciting NoQL things with SQL Expressions, such as: