Skip to content

Commit

Permalink
mutil task import
Browse files Browse the repository at this point in the history
  • Loading branch information
@xjlgod
xjlgod committed Mar 14, 2022
1 parent 73071be commit e6c69b8
Show file tree
Hide file tree
Showing 17 changed files with 935 additions and 265 deletions.
234 changes: 202 additions & 32 deletions server/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -29,15 +29,19 @@ $ ./nebula-studio-server -studio-config="./config/example-config.yaml"
| ConnectDB | /api-nebula/db/connect | POST |
| DisconnectDB | /api-nebula/db/disconnect | POST |
| ImportData | /api-nebula/task/import | POST |
| HandleImportAction | /api-nebula/import/action | POST |
| HandleImportAction | /api-nebula/task/import/action | POST |
| QueryImportStats | /api-nebula/task/import/stats | POST |
| DownloadConfig | /api-nebula/task/import/config | POST |
| DownloadLog | /api-nebula/task/import/log | GET |
| FilesIndex | /api/files | GET |
| FilesDestroy | /api/files/{id:string} | DELETE |
| FilesUpload | /api/files | PUT |
| ReadLog | /api/import/log | GET |
| CreateConfigFile | /api/import/config | POST |
| ReadErrLog | /api/import/err_log | GET |
| Callback | /api/import/finish | POST |
| GetWorkingDir | /api/import/working_dir | GET |
| Index | / | GET,HEAD |
| GetTaskDir | /api/import/task_dir | GET |
| GetTaskLogPaths | /api/import/task_err_log_paths/{id:string} | GET |

#### ExecNGQL API

Expand Down Expand Up @@ -100,7 +104,7 @@ Response:
The request json body:

```json
{"configBody":{"version":"v2","description":"web console import","clientSettings":{"retry":3,"concurrency":10,"channelBufferSize":128,"space":"basketballplayer","connection":{"user":"root","password":"123","address":"192.168.8.145:9669"}},"logPath":"E:\\NebulaProject\\test\\nebula-studio\\server\\data\\tasks/import.log","files":[{"path":"E:\\NebulaProject\\test\\nebula-studio\\server\\data\\upload\\player.csv","failDataPath":"E:\\NebulaProject\\test\\nebula-studio\\server\\data\\tasks/err/data 1Fail.csv","batchSize":10,"type":"csv","csv":{"withHeader":false,"withLabel":false},"schema":{"type":"vertex","vertex":{"vid":{"index":0,"type":"string"},"tags":[{"name":"player","props":[{"name":"name","type":"string","index":1},{"name":"age","type":"int","index":2}]}]}}}]},"configPath":""}
{"configBody":{"version":"v2","description":"web console import","clientSettings":{"retry":3,"concurrency":10,"channelBufferSize":128,"space":"basketballplayer","connection":{"user":"root","password":"123","address":"192.168.8.145:9669"}},"logPath":"E:\\NebulaProject\\test\\nebula-studio\\server\\data\\tasks/import.log","files":[{"path":"E:\\NebulaProject\\test\\nebula-studio\\server\\data\\upload\\player.csv","failDataPath":"E:\\NebulaProject\\test\\nebula-studio\\server\\data\\tasks/err/data 1Fail.csv","batchSize":10,"type":"csv","csv":{"withHeader":false,"withLabel":false},"schema":{"type":"vertex","vertex":{"vid":{"index":0,"type":"string"},"tags":[{"name":"player","props":[{"name":"name","type":"string","index":1},{"name":"age","type":"int","index":2}]}]}}}]},"configPath":"","name":"task1"}
```

Response:
Expand All @@ -119,35 +123,153 @@ Response:

The request json body:

```json
{"taskID":"1","taskAction":"actionQuery"}
```

Response:

```json
{
"code": 0,
"message": "Import task 4 submit successfully",
"data": [
"4"
]
"code": 0,
"message": "Processing a task action successfully",
"data": {
"results": [
{
"taskID": 1,
"name": "task1",
"space": "test",
"nebulaAddress": "192.168.8.243:9669",
"createdTime": 1644386055,
"updatedTime": 1644386056,
"user": "root",
"taskStatus": "statusAborted",
"taskMessage": "failed to open connection, error: incompatible version between client and server: Graph client version(3.0.0) is not accepted, current graph client white list: 2.6.1:2.5.0:2.5.1:2.6.0. ",
"stats": {
"numFailed": 0,
"numReadFailed": 0,
"totalCount": 0,
"totalBatches": 0,
"totalLatency": 0,
"totalReqTime": 0,
"totalBytes": 0,
"totalImportedBytes": 0
}
}
],
"msg": "Task query successfully"
}
}
```

#### QueryImportStats API

The request json body:

```json
{"taskId":"1"}
```

Response:

```json
{
"code": 0,
"message": "Processing a task action successfully",
"data": {
"results": [
{
"taskID": "4",
"code": 0,
"message": "Processing a task action successfully",
"data": {
"taskID": 3,
"name": "task1",
"space": "test",
"nebulaAddress": "192.168.8.233:9669",
"createdTime": 1646989643,
"updatedTime": 1646989646,
"user": "root",
"taskStatus": "statusFinished",
"taskMessage": ""
}
],
"msg": "Task query successfully"
}
"taskMessage": "",
"stats": {
"numFailed": 0,
"numReadFailed": 0,
"totalCount": 52,
"totalBatches": 10,
"totalLatency": 30089,
"totalReqTime": 532718,
"totalBytes": 1583,
"totalImportedBytes": 1583
}
}
}
```

#### DownloadConfig API

The request json body:

```json
{
"taskID": "1"
}
```

Response:

```
version: v2
description: web console import
removeTempFiles: null
clientSettings:
retry: 3
concurrency: 10
channelBufferSize: 128
space: test
connection:
user: user
password: password
address: host
postStart: null
preStop: null
logPath: E:\NebulaProject\新建文件夹 (4)\nebula-studio\server\data\tasks\1/import.log
files:
- path: E:\NebulaProject\新建文件夹 (4)\nebula-studio\server\data\upload\player.csv
failDataPath: E:\NebulaProject\新建文件夹 (4)\nebula-studio\server\data\tasks\1/err/数据源
1Fail.csv
batchSize: 60
limit: null
inOrder: null
type: csv
csv:
withHeader: false
withLabel: false
delimiter: null
schema:
type: vertex
edge: null
vertex:
vid:
index: 0
function: null
type: string
tags:
- name: player
props:
- name: name
type: string
index: 1
- name: age
type: int
index: 2
```

#### DownloadLog API

The request :

http://localhost:9000/api-nebula/task/import/log/?pathName=E:\NebulaProject\nebula-studio\server\data\tasks\1\err\err 1.log

Response：

a file

#### FilesIndex API

Response:
Expand Down Expand Up @@ -212,31 +334,37 @@ Response:
```
#### ReadLog API
Use params:
http://localhost:7001/api/import/log?dir=E:\NebulaProject\test\nebula-studio\server\data\tasks&startByte=0&endByte=1000000&taskId=4
http://localhost:9000/api/import/log?limit=2&path=data/tasks/1/import.log&taskId=1&offset=1
Response:

```json
{
"code": 0,
"message": "",
"data": "2022/01/11 16:33:48 [INFO] clientmgr.go:28: Create 10 Nebula Graph clients\u003cbr /\u003e2022/01/11 16:33:48 [INFO] reader.go:64: Start to read file(0): E:\\NebulaProject\\test\\nebula-studio\\server\\data\\upload\\player.csv, schema: \u003c :VID(string),player.name:string,player.age:int \u003e\u003cbr /\u003e2022/01/11 16:33:48 [INFO] reader.go:180: Total lines of file(E:\\NebulaProject\\test\\nebula-studio\\server\\data\\upload\\player.csv) is: 52, error lines: 0\u003cbr /\u003e2022/01/11 16:33:48 [INFO] statsmgr.go:62: Done(E:\\NebulaProject\\test\\nebula-studio\\server\\data\\upload\\player.csv): Time(1.45s), Finished(52), Failed(0), Read Failed(0), Latency AVG(6182us), Batches Req AVG(48887us), Rows AVG(35.87/s)\u003cbr /\u003e"
}xxxxxxxxxx {  "code": 0,  "message": "",  "data": "2022/01/11 16:33:48 [INFO] clientmgr.go:28: Create 10 Nebula Graph clients\u003cbr /\u003e2022/01/11 16:33:48 [INFO] reader.go:64: Start to read file(0): E:\\NebulaProject\\test\\nebula-studio\\server\\data\\upload\\player.csv, schema: \u003c :VID(string),player.name:string,player.age:int \u003e\u003cbr /\u003e2022/01/11 16:33:48 [INFO] reader.go:180: Total lines of file(E:\\NebulaProject\\test\\nebula-studio\\server\\data\\upload\\player.csv) is: 52, error lines: 0\u003cbr /\u003e2022/01/11 16:33:48 [INFO] statsmgr.go:62: Done(E:\\NebulaProject\\test\\nebula-studio\\server\\data\\upload\\player.csv): Time(1.45s), Finished(52), Failed(0), Read Failed(0), Latency AVG(6182us), Batches Req AVG(48887us), Rows AVG(35.87/s)\u003cbr /\u003e"}{    "code": 0,    "message": "",    "data": "hell"}
```
#### CreateConfigFile API
Create a config file in config_dir
```json
{"config":{"version":"v2","description":"web console import","clientSettings":{"retry":3,"concurrency":10,"channelBufferSize":128,"space":"basketballplayer","connection":{"user":"root","password":"123","address":"192.168.8.145:9669"}},"logPath":"E:\\NebulaProject\\test\\nebula-studio\\server\\data\\tasks/import.log","files":[]},"mountPath":"E:\\NebulaProject\\test\\nebula-studio\\server\\data\\tasks"}
"code": 0,
"message": "",
"data": [
"bbbbbbbbbbb",
"ccccccccccc"
]
}
```
#### ReadErrLog API

Use params:

http://localhost:9000/api/import/err_log?offset=1&ReadJSON=1&path=E:\\NebulaProject\nebula-studio\\server\\data\\tasks\\1\\err\\err 1.log

Response:

```json
{
"code": 0,
"message": "",
"data": ""
"data": [
"bbbbbbbbbb"
]
}
```

#### Callback API

The request json body:
Expand Down Expand Up @@ -266,9 +394,51 @@ Response:
"code": 0,
"message": "",
"data": {
"taskDir": "E:\\NebulaProject\\test\\nebula-studio\\server\\data\\tasks",
"uploadDir": "E:\\NebulaProject\\test\\nebula-studio\\server\\data\\upload"
}
}
```

#### GetTaskDir API

Response:

```json
{
"code": 0,
"message": "",
"data": {
"taskDir": "E:\\NebulaProject\\nebula-studio\\server\\data\\tasks\\1"
}
}
```

#### GetTaskLogPaths API

Request:

http://localhost:9000/api/import/task_err_log_paths/1

Response:

```json
{
"code": 0,
"message": "",
"data": [
{
"name": "import.log",
"path": "E:\\NebulaProject\\nebula-studio\\server\\data\\tasks\\1\\import.log"
},
{
"name": "err 1.log",
"path": "E:\\NebulaProject\\nebula-studio\\server\\data\\tasks\\1\\err\\err 1.log"
},
{
"name": "err 2.log",
"path": "E:\\NebulaProject\\nebula-studio\\server\\data\\tasks\\1\\err\\err 2.log"
}
]
}
```

2 changes: 1 addition & 1 deletion server/config/example-config.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,4 +4,4 @@ web:
# tasks_dir:
# sqlitedb_file_path:
# ip:
port: 7001
# port: 7001
10 changes: 8 additions & 2 deletions server/go.mod
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,12 +3,16 @@ module github.com/vesoft-inc/nebula-studio/server
go 1.17

require (
github.com/axgle/mahonia v0.0.0-20180208002826-3358181d7394
github.com/kataras/iris/v12 v12.2.0-alpha4
github.com/mattn/go-sqlite3 v2.0.3+incompatible
github.com/saintfish/chardet v0.0.0-20120816061221-3af4cd4741ca
github.com/vesoft-inc/nebula-http-gateway/ccore v0.0.0-20220124093655-f7f5b4c63e6a
github.com/vesoft-inc/nebula-importer v1.0.1-0.20211213064541-05a8646be295
github.com/vesoft-inc/nebula-importer v1.0.1-0.20220314055243-ece1a79d37e5
go.uber.org/zap v1.17.0
gopkg.in/yaml.v2 v2.4.0
gorm.io/driver/sqlite v1.2.6
gorm.io/gorm v1.22.5
)

require (
Expand All @@ -29,6 +33,8 @@ require (
github.com/gorilla/css v1.0.0 // indirect
github.com/iris-contrib/jade v1.1.4 // indirect
github.com/iris-contrib/schema v0.0.6 // indirect
github.com/jinzhu/inflection v1.0.0 // indirect
github.com/jinzhu/now v1.1.4 // indirect
github.com/json-iterator/go v1.1.12 // indirect
github.com/kataras/blocks v0.0.4 // indirect
github.com/kataras/golog v0.1.7 // indirect
Expand All @@ -47,7 +53,7 @@ require (
github.com/tdewolff/minify/v2 v2.9.22 // indirect
github.com/tdewolff/parse/v2 v2.5.21 // indirect
github.com/valyala/bytebufferpool v1.0.0 // indirect
github.com/vesoft-inc/nebula-go/v2 v2.5.2-0.20211210024917-9461e07cdca2 // indirect
github.com/vesoft-inc/nebula-go/v3 v3.0.0-20220214062853-d0c59964d0af // indirect
github.com/vmihailenco/msgpack/v5 v5.3.4 // indirect
github.com/vmihailenco/tagparser/v2 v2.0.0 // indirect
github.com/yosssi/ace v0.0.5 // indirect
Expand Down
Loading

0 comments on commit e6c69b8

Please sign in to comment.