Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

exchange add udf #2099

Merged
merged 1 commit into from
May 22, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -101,6 +101,9 @@ For different data sources, the vertex configurations are different. There are m
|`tags.fields`|list\[string\]|-|Yes|The header or column name of the column corresponding to properties. If there is a header or a column name, please use that name directly. If a CSV file does not have a header, use the form of `[_c0, _c1, _c2]` to represent the first column, the second column, the third column, and so on.|
|`tags.nebula.fields`|list\[string\]|-|Yes|Property names defined in NebulaGraph, the order of which must correspond to `tags.fields`. For example, `[_c1, _c2]` corresponds to `[name, age]`, which means that values in the second column are the values of the property `name`, and values in the third column are the values of the property `age`.|
|`tags.vertex.field`|string|-|Yes|The column of vertex IDs. For example, when a CSV file has no header, users can use `_c0` to indicate values in the first column are vertex IDs.|
|`tags.vertex.udf.separator`|string|-|No|Support merging multiple columns by custom rules. This parameter specifies the join character.|
|`tags.vertex.udf.oldColNames`|list|-|No|Support merging multiple columns by custom rules. This parameter specifies the names of the columns to be merged. Multiple columns are separated by commas.|
|`tags.vertex.udf.newColName`|string|-|No|Support merging multiple columns by custom rules. This parameter specifies the new column name. |
|`tags.batch`|int|`256`|Yes|The maximum number of vertices written into NebulaGraph in a single batch.|
|`tags.partition`|int|`32`|Yes|The number of Spark partitions.|

Expand Down
15 changes: 15 additions & 0 deletions docs-2.0/nebula-exchange/use-exchange/ex-ug-export-from-nebula.md
Original file line number Diff line number Diff line change
Expand Up @@ -228,6 +228,11 @@ As the data source, NebulaGraph stores the [basketballplayer dataset](https://do
nebula.fields: [target_nebula-field-0, target_nebula-field-1, target_nebula-field-2]
limit:10000
vertex: _vertexId # must be `_vertexId`
# udf:{
# separator:"_"
# oldColNames:[field-0,field-1,field-2]
# newColName:new-field
# }
batch: 2000
partition: 60
}
Expand All @@ -249,7 +254,17 @@ As the data source, NebulaGraph stores the [basketballplayer dataset](https://do
nebula.fields: [target_nebula-field-0, target_nebula-field-1, target_nebula-field-2]
limit:1000
source: _srcId # must be `_srcId`
# udf:{
# separator:"_"
# oldColNames:[field-0,field-1,field-2]
# newColName:new-field
# }
target: _dstId # must be `_dstId`
# udf:{
# separator:"_"
# oldColNames:[field-0,field-1,field-2]
# newColName:new-field
# }
ranking: source_nebula-field-2
batch: 2000
partition: 60
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -166,6 +166,11 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# Specify a column of data in the table as the source of vertex VID in the NebulaGraph.
vertex: {
field:playerid
# udf:{
# separator:"_"
# oldColNames:[field-0,field-1,field-2]
# newColName:new-field
# }
# policy:hash
}

Expand Down Expand Up @@ -233,11 +238,21 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# In source, use a column in the follow table as the source of the edge's source vertexes.
source: {
field:src_player
# udf:{
# separator:"_"
# oldColNames:[field-0,field-1,field-2]
# newColName:new-field
# }
}

# In target, use a column in the follow table as the source of the edge's destination vertexes.
target: {
field:dst_player
# udf:{
# separator:"_"
# oldColNames:[field-0,field-1,field-2]
# newColName:new-field
# }
}

# (Optional) Specify a column as the source of the rank.
Expand Down
15 changes: 15 additions & 0 deletions docs-2.0/nebula-exchange/use-exchange/ex-ug-import-from-csv.md
Original file line number Diff line number Diff line change
Expand Up @@ -187,6 +187,11 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# Currently, NebulaGraph {{nebula.release}} supports only strings or integers of VID.
vertex: {
field:_c0
# udf:{
# separator:"_"
# oldColNames:[field-0,field-1,field-2]
# newColName:new-field
# }
# policy:hash
}

Expand Down Expand Up @@ -286,9 +291,19 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# Currently, NebulaGraph {{nebula.release}} supports only strings or integers of VID.
source: {
field: _c0
# udf:{
# separator:"_"
# oldColNames:[field-0,field-1,field-2]
# newColName:new-field
# }
}
target: {
field: _c1
# udf:{
# separator:"_"
# oldColNames:[field-0,field-1,field-2]
# newColName:new-field
# }
}

# The delimiter specified. The default value is comma.
Expand Down
15 changes: 15 additions & 0 deletions docs-2.0/nebula-exchange/use-exchange/ex-ug-import-from-hbase.md
Original file line number Diff line number Diff line change
Expand Up @@ -200,6 +200,11 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# For example, if rowkey is the source of the VID, enter rowkey.
vertex:{
field:rowkey
# udf:{
# separator:"_"
# oldColNames:[field-0,field-1,field-2]
# newColName:new-field
# }
}


Expand Down Expand Up @@ -262,11 +267,21 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# In target, use a column in the follow table as the source of the edge's destination vertex.
source:{
field:rowkey
# udf:{
# separator:"_"
# oldColNames:[field-0,field-1,field-2]
# newColName:new-field
# }
}


target:{
field:dst_player
# udf:{
# separator:"_"
# oldColNames:[field-0,field-1,field-2]
# newColName:new-field
# }
}

# (Optional) Specify a column as the source of the rank.
Expand Down
15 changes: 15 additions & 0 deletions docs-2.0/nebula-exchange/use-exchange/ex-ug-import-from-hive.md
Original file line number Diff line number Diff line change
Expand Up @@ -238,6 +238,11 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# Specify a column of data in the table as the source of vertex VID in the NebulaGraph.
vertex:{
field:playerid
# udf:{
# separator:"_"
# oldColNames:[field-0,field-1,field-2]
# newColName:new-field
# }
}

# The number of data written to NebulaGraph in a single batch.
Expand Down Expand Up @@ -294,10 +299,20 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# In target, use a column in the follow table as the source of the edge's destination vertex.
source: {
field: src_player
# udf:{
# separator:"_"
# oldColNames:[field-0,field-1,field-2]
# newColName:new-field
# }
}

target: {
field: dst_player
# udf:{
# separator:"_"
# oldColNames:[field-0,field-1,field-2]
# newColName:new-field
# }
}

# (Optional) Specify a column as the source of the rank.
Expand Down
15 changes: 15 additions & 0 deletions docs-2.0/nebula-exchange/use-exchange/ex-ug-import-from-jdbc.md
Original file line number Diff line number Diff line change
Expand Up @@ -218,6 +218,11 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# Specify a column of data in the table as the source of VIDs in the NebulaGraph.
vertex: {
field:playerid
# udf:{
# separator:"_"
# oldColNames:[field-0,field-1,field-2]
# newColName:new-field
# }
}

# The number of data written to NebulaGraph in a single batch.
Expand Down Expand Up @@ -295,10 +300,20 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# In target, use a column in the follow table as the source of the edge's destination vertex.
source: {
field: src_player
# udf:{
# separator:"_"
# oldColNames:[field-0,field-1,field-2]
# newColName:new-field
# }
}

target: {
field: dst_player
# udf:{
# separator:"_"
# oldColNames:[field-0,field-1,field-2]
# newColName:new-field
# }
}

# (Optional) Specify a column as the source of the rank.
Expand Down
15 changes: 15 additions & 0 deletions docs-2.0/nebula-exchange/use-exchange/ex-ug-import-from-json.md
Original file line number Diff line number Diff line change
Expand Up @@ -215,6 +215,11 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# Currently, NebulaGraph {{nebula.release}} supports only strings or integers of VID.
vertex: {
field:id
# udf:{
# separator:"_"
# oldColNames:[field-0,field-1,field-2]
# newColName:new-field
# }
}

# The number of data written to NebulaGraph in a single batch.
Expand Down Expand Up @@ -299,9 +304,19 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# Currently, NebulaGraph {{nebula.release}} supports only strings or integers of VID.
source: {
field: src
# udf:{
# separator:"_"
# oldColNames:[field-0,field-1,field-2]
# newColName:new-field
# }
}
target: {
field: dst
# udf:{
# separator:"_"
# oldColNames:[field-0,field-1,field-2]
# newColName:new-field
# }
}


Expand Down
15 changes: 15 additions & 0 deletions docs-2.0/nebula-exchange/use-exchange/ex-ug-import-from-kafka.md
Original file line number Diff line number Diff line change
Expand Up @@ -159,6 +159,11 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# The key is the same as the value above, indicating that key is used as both VID and property name.
vertex:{
field:key
# udf:{
# separator:"_"
# oldColNames:[field-0,field-1,field-2]
# newColName:new-field
# }
}


Expand Down Expand Up @@ -222,11 +227,21 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# In target, use a column in the topic as the source of the edge's destination vertex.
source:{
field:timestamp
# udf:{
# separator:"_"
# oldColNames:[field-0,field-1,field-2]
# newColName:new-field
# }
}


target:{
field:offset
# udf:{
# separator:"_"
# oldColNames:[field-0,field-1,field-2]
# newColName:new-field
# }
}

# (Optional) Specify a column as the source of the rank.
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -176,6 +176,11 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# Specify a column of data in the table as the source of vertex VID in the NebulaGraph.
vertex:{
field: playerid
# udf:{
# separator:"_"
# oldColNames:[field-0,field-1,field-2]
# newColName:new-field
# }
}

# The number of data written to NebulaGraph in a single batch.
Expand Down Expand Up @@ -256,11 +261,21 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# In source, use a column in the follow table as the source of the edge's source vertex.
source:{
field: src_player
# udf:{
# separator:"_"
# oldColNames:[field-0,field-1,field-2]
# newColName:new-field
# }
}

# In target, use a column in the follow table as the source of the edge's destination vertex.
target:{
field: dst_player
# udf:{
# separator:"_"
# oldColNames:[field-0,field-1,field-2]
# newColName:new-field
# }
}

# (Optional) Specify a column as the source of the rank.
Expand Down
15 changes: 15 additions & 0 deletions docs-2.0/nebula-exchange/use-exchange/ex-ug-import-from-mysql.md
Original file line number Diff line number Diff line change
Expand Up @@ -204,6 +204,11 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# Specify a column of data in the table as the source of VIDs in the NebulaGraph.
vertex: {
field:playerid
# udf:{
# separator:"_"
# oldColNames:[field-0,field-1,field-2]
# newColName:new-field
# }
}

# The number of data written to NebulaGraph in a single batch.
Expand Down Expand Up @@ -273,10 +278,20 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
# In target, use a column in the follow table as the source of the edge's destination vertex.
source: {
field: src_player
# udf:{
# separator:"_"
# oldColNames:[field-0,field-1,field-2]
# newColName:new-field
# }
}

target: {
field: dst_player
# udf:{
# separator:"_"
# oldColNames:[field-0,field-1,field-2]
# newColName:new-field
# }
}

# (Optional) Specify a column as the source of the rank.
Expand Down
15 changes: 15 additions & 0 deletions docs-2.0/nebula-exchange/use-exchange/ex-ug-import-from-neo4j.md
Original file line number Diff line number Diff line change
Expand Up @@ -184,6 +184,11 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
nebula.fields: [age,name]
vertex: {
field:id
# udf:{
# separator:"_"
# oldColNames:[field-0,field-1,field-2]
# newColName:new-field
# }
}
partition: 10
batch: 1000
Expand Down Expand Up @@ -230,9 +235,19 @@ After Exchange is compiled, copy the conf file `target/classes/application.conf`
nebula.fields: [degree]
source: {
field: src
# udf:{
# separator:"_"
# oldColNames:[field-0,field-1,field-2]
# newColName:new-field
# }
}
target: {
field: dst
# udf:{
# separator:"_"
# oldColNames:[field-0,field-1,field-2]
# newColName:new-field
# }
}
#ranking: rank
partition: 10
Expand Down
Loading