Bulk insertion into existing graph

[ahirel]

Hi!

I saw in the documentation that the GRAPH.BULK command was only allowed for importing data into a new graph. Would it be possible in the future to allow import into an existing one? If not, would it be possible, for instance, to bulk import into a new graph and "stitch" that new graph to the main graph?

Thank you

[swilly22]

Hey,
You can use falkordb-bulk-update to modify an existing graph.

[ahirel]

Thanks Roi, i just looked at the documentation, however i am working with Typescript and not Python so i cannot really use that lib.

What is the logic behind the update in the bulk-update library? How can it do some updates if the GRAPH.BULK command is explicitely forbidding it? Does it use another command?

Thanks

[swilly22]

If my memory serves me right, the bulk-update generates GRAPH.QUERY commands, it runs the given query for each row in the CSV file(s) replacing the place holders with the relevant values. although not as fast as the bulk-insert, we ought to introduce a command similar to LOAD CSV which does all of the work on the server side.

In case you're restricted to using Typescript I'll suggest processing the CSV file(s) your self and generating queries, this is practically what the current bulk-update does.

[ahirel]

Alright, thanks a lot :)

From what i remember reading some discussions in this repo, it was said that the advantage of bulk was to bypass the recalculation of some GraphBLAS related matrix, but also that is was generally a good practice to use reusable queries with escaped parameters.

So in the case of a big import (say several 10ks of nodes/edges at once), would it be more relevant to send one big query with n CREATE statements or n queries with one statement each?

[swilly22]

The advantages of using bulk-insert is mainly to avoid query processing, the data is laid out in an optimal form for ingestion.

When introducing a large set of modifications to the graph e.g. inserting new nodes/edges or updating a large portion of the graph in the form of queries, it really depends on the ability to express the change in a compact query

For example, suppose we want to increase the value of the score attribute of all nodes of type User in the graph by some constant, this can be easily expressed via the following query:

MATCH (u:User) SET u.score = u.score + 1

In such cases a single compact query is quite optimal, note it might take awhile to execute and memory consumption can spike.

On the other hand if the change can't be expressed in a compact form, I'll suggest breaking the change-set into a number of "batched" queries utilizing query parameters.

e.g.

CYPHER USER_ID_SET [1,2, 12, 52, 98, 102] MATCH (u:User) WHERE u.id in $USER_ID_SET SET u.score = u.score + 1

I hope this helps.

[ahirel]

Yes i understand, thank you very much!