LOAD CSV improvements #887
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…troduction, minor fixes
rsill-neo4j
commented
Feb 9, 2024
Co-authored-by: Jessica Wright <[email protected]>
There was a problem hiding this comment.
I think a structure that would more effectively cater for the needs of the majority is one like:
- Import csv files into neo4j
- Load from local paths (one example + saying where they are sourced by default)
- Load from remote locations
- Create constraints
- Handle large amounts of data
- Import relationships and datasets coming from relational databases (expand on the section you have, this is important. It's often a source of confusion how people are supposed to handle csv files, what they should do if they get a number of csvs and what if they get one csv with all nodes and relationships inside it)
- Process data ahead of import
- Cast CSV columns to relevant Neo4j types
- Split list values
- Create additional node labels
- LOAD CSV options and functions
- {all subsections from CSV file format}
- Performance recommendations
- A full example
- (Other ways of importing data into Neo4j) (just to give pointers to
neo4j-admin importand drivers! Say that avoiding load csv and deferring the csv parsing+querying to your own app is a very valid choice (one that field team very often takes))
…ypher into LOAD-CSV-improvements
…ypher into LOAD-CSV-improvements
There was a problem hiding this comment.
Great work @rsill-neo4j!
I have a few editorial suggestions, and also 2 broader points:
- Maybe the "Uniqueness constraints" and "Handle large amounts of data" can go into a separate section at the end, called something along the lines of "Recommended/Best practices"?
- As discussed with @gem-neo4j , I think the "Performance" section could be cut from this page and perhaps be extended into its own tutorial.
Also, make sure to follow the new formating style of data types (e.g 'STRING' instead of 'string' (and 'STRING values' instead of 'strings').
Co-authored-by: Jens Pryce-Åklundh <[email protected]>
gem-neo4j
reviewed
Mar 6, 2024
Co-authored-by: Gem Lamont <[email protected]>
gem-neo4j
approved these changes
Mar 8, 2024
| From a graph perspective, these are nodes with different labels, so it takes different queries to load them. | ||
|
|
||
| The example executes multiple passes of `LOAD CSV` on that one file, and each pass focuses on the creation of _one_ entity type. | ||
| This is the most performant practice, see <<_separate_creation_of_nodes_and_relationships>>. |
There was a problem hiding this comment.
Looks the <<>> part isn't rendering correctly
There was a problem hiding this comment.
good catch :) it refers to a section we commented out. gonna update
|
Thanks for the documentation updates. The preview documentation has now been torn down - reopening this PR will republish it. |
JPryce-Aklundh
approved these changes
Mar 8, 2024
JPryce-Aklundh
added
the
cherry-pick-this-to-5.x
rsill-neo4j
added a commit
that referenced
this pull request
Mar 8, 2024
i think the first few (older) examples could be brought in line with the newer examples i added also, i'm pretty sure the tests will fail left and right - the expected query results were written by hand ✋ finally, the GitHub diff is not really pretty ( = readable). there ought to be a preview of the changes though --------- Co-authored-by: Jessica Wright <[email protected]> Co-authored-by: Stefano Ottolenghi <[email protected]> Co-authored-by: Jens Pryce-Åklundh <[email protected]> Co-authored-by: Gem Lamont <[email protected]>
rsill-neo4j
added
the
cherry-picked
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
i think the first few (older) examples could be brought in line with the newer examples i added
also, i'm pretty sure the tests will fail left and right - the expected query results were written by hand ✋
finally, the GitHub diff is not really pretty ( = readable). there ought to be a preview of the changes though