Some bug fix for load task and schema manage #246

Linary · 2020-09-29T09:22:46Z

No description provided.

javeme · 2020-09-30T10:10:41Z

hubble-be/src/main/java/com/baidu/hugegraph/entity/load/LoadTask.java

+            this.fileReadLines = this.context().newProgress().totalInputReaded();
+            this.lastDuration += this.context().summary().totalTime();
+            this.currDuration = 0L;
+            // how to save db


javeme · 2020-09-30T10:11:17Z

hubble-be/src/main/java/com/baidu/hugegraph/entity/load/LoadTask.java

+        while (this.status.inRunning()) {
+            try {
+                Thread.sleep(1000);
+            } catch (InterruptedException e) {


rename to ignored

javeme · 2020-09-30T10:11:54Z

hubble-be/src/main/java/com/baidu/hugegraph/handler/LoadTaskExecutor.java

        log.info("LoadTask is executing run task:{}", task.getId());
        task.run();
+        log.info("Execute callback");


update log to:
log.info("Executing task: {}", task.getId());
log.info("Executed task: {}", task.getId());

javeme · 2020-10-16T12:05:55Z

hubble-be/src/main/java/com/baidu/hugegraph/service/load/FileMappingService.java

        try {
            // transferTo should accept absolute path
            srcFile.transferTo(destFile.getAbsoluteFile());
            result.setStatus(FileUploadResult.Status.SUCCESS);
+            log.info("Uploaded file part {}", partName + "-" + index);


"Uploaded file part {}-{}",

javeme · 2020-10-16T12:06:55Z

hubble-be/src/main/java/com/baidu/hugegraph/service/load/FileMappingService.java

@@ -160,7 +168,8 @@ public boolean tryMergePartFiles(String dirPath, int total) {
                // Rename file to dest file
                FileUtils.moveFile(partFiles[0], newFile);
            } catch (IOException e) {
-                throw new InternalException("load.upload.move-file.failed");
+                log.error(e);


add some message for log

javeme · 2020-10-16T12:07:05Z

hubble-be/src/main/java/com/baidu/hugegraph/service/load/FileMappingService.java

@@ -178,19 +187,22 @@ public boolean tryMergePartFiles(String dirPath, int total) {
                    try (InputStream is = new FileInputStream(partFile)) {
                        IOUtils.copy(is, os);
                    } catch (IOException e) {
+                        log.error(e);


javeme · 2020-10-16T12:07:11Z

hubble-be/src/main/java/com/baidu/hugegraph/service/load/FileMappingService.java

            }
        }
        // Delete origin directory
        try {
            FileUtils.forceDelete(dir);
        } catch (IOException e) {
-            throw new InternalException("load.upload.delete-temp-dir.failed");
+            log.error(e);


javeme · 2020-10-16T12:08:28Z

hubble-be/src/main/java/com/baidu/hugegraph/service/algorithm/AsyncTaskService.java

-        Iterator<Task> tasks = list.iterator();
-        while (tasks.hasNext()) {
-            Task task = tasks.next();
+        for (Task task : list) {


rename list to tasks

javeme · 2020-10-16T12:12:33Z

hubble-be/src/main/java/com/baidu/hugegraph/service/load/LoadTaskService.java

+                    LoadContext context = task.context();
+                    task.setFileReadLines(context.newProgress().totalInputReaded());
+                    task.setCurrDuration(context.summary().totalTime());
+                    if (this.update(task) != 1) {


what does 1 mean, prefer boolean or enum

1 means operate affected rows. it defined by mybatis-plus framework.

javeme · 2020-10-19T06:11:15Z

hubble-be/src/main/java/com/baidu/hugegraph/service/load/FileMappingService.java

@@ -178,19 +188,23 @@ public boolean tryMergePartFiles(String dirPath, int total) {
                    try (InputStream is = new FileInputStream(partFile)) {
                        IOUtils.copy(is, os);
                    } catch (IOException e) {
+                        log.error("Failed copy file stream from {} to {}",


Failed to copy

javeme · 2020-10-19T06:11:24Z

hubble-be/src/main/java/com/baidu/hugegraph/service/load/FileMappingService.java

                    }
                }
            } catch (IOException e) {
-                throw new InternalException("load.upload.merge-file.failed");
+                log.error(e);


javeme · 2020-10-19T06:14:09Z

hubble-be/src/main/java/com/baidu/hugegraph/service/load/LoadTaskService.java

-            throw new InternalException("entity.update.failed", task);
+        task.getOptions().incrementalMode = true;
+        task.restoreContext();
+        task.lock.lock();


prefer to add method task.lock()/unlock()

javeme · 2020-10-19T06:17:58Z

hubble-be/src/main/java/com/baidu/hugegraph/entity/load/LoadTask.java

 public class LoadTask implements Runnable {

    @TableField(exist = false)
    @JsonIgnore
-    private transient HugeGraphLoader loader;
+    public transient final Lock lock = new ReentrantLock();


set private

javeme · 2020-10-19T06:25:36Z

hubble-be/src/main/java/com/baidu/hugegraph/controller/load/FileUploadController.java

+                              @PathVariable("jobId") int jobId,
+                              @RequestParam("name") String fileName) {
+        String token = this.service.generateFileToken(fileName);
+        this.uploadingTokenLocks.put(token, new ReentrantReadWriteLock());


check exist

javeme · 2020-10-19T06:32:35Z

hubble-be/src/main/java/com/baidu/hugegraph/service/load/LoadTaskService.java

-        Iterator<Map.Entry<Integer, LoadTask>> iter = taskContainer.entrySet()
-                                                                   .iterator();
+        Iterator<Map.Entry<Integer, LoadTask>> iter;
+        iter = this.taskContainer.entrySet().iterator();


runningTasks

javeme · 2020-10-20T03:37:50Z

hubble-be/src/main/java/com/baidu/hugegraph/controller/load/FileUploadController.java

+        for (String fileName : fileNames) {
+            String token = this.service.generateFileToken(fileName);
+            Ex.check(!this.uploadingTokenLocks.containsKey(token),
+                     "load.upload.file.token.existed");


check put() return null

hubble-be/src/main/java/com/baidu/hugegraph/service/load/LoadTaskService.java

javeme · 2020-10-23T13:00:56Z

hubble-be/src/main/java/com/baidu/hugegraph/controller/load/FileMappingController.java

@@ -286,6 +291,17 @@ public void loadParameter(@RequestBody LoadParameter newEntity) {
        }
    }

+    @PutMapping("finish")


prefer rename to another name, like ready-loading

javeme · 2020-10-23T13:02:34Z

hubble-be/src/main/java/com/baidu/hugegraph/controller/load/FileUploadController.java

@@ -240,6 +247,17 @@ public Boolean delete(@PathVariable("connId") int connId,
        }
    }

+    @PutMapping("finish")


javeme · 2020-10-26T13:11:14Z

hubble-be/src/main/java/com/baidu/hugegraph/options/HubbleOptions.java

+            new ConfigOption<>(
+                    "upload_file.max_uploading_time",
+                    "The maximum allowable uploading time(second) for file " +
+                    "uploads, the uploaded parts will be cleared if exceed " +


uploaded file parts

javeme · 2020-10-26T13:16:55Z

hubble-be/src/main/java/com/baidu/hugegraph/service/load/FileMappingService.java

+                try {
+                    FileUtils.forceDelete(new File(filePath));
+                } catch (IOException e) {
+                    log.warn("Failed to delete uploading timeout file {}",


delete expired uploading files

javeme · 2020-10-26T13:23:09Z

hubble-be/src/main/java/com/baidu/hugegraph/service/load/LoadTaskService.java

-            throw new InternalException("entity.update.failed", task);
-        }
+        // executed in other threads
+        this.taskExecutor.execute(task, () -> this.update(task));


should handle exception?

should handle exception if the update is failed?

javeme · 2020-10-26T13:23:38Z

hubble-be/src/main/java/com/baidu/hugegraph/service/load/LoadTaskService.java

+        try {
+            if (this.update(task) != 1) {
+                throw new InternalException("entity.update.failed", task);
+            }


can we wrap these 3 lines with a method

hubble-be/src/main/java/com/baidu/hugegraph/service/load/LoadTaskService.java

javeme · 2020-10-26T13:26:24Z

hubble-be/src/main/java/com/baidu/hugegraph/service/load/LoadTaskService.java

-        if (this.update(task) != 1) {
-            throw new InternalException("entity.update.failed", task);
+        task.getOptions().incrementalMode = true;
+        task.restoreContext();


move line 200~2201 to try block

javeme · 2020-10-26T13:27:56Z

hubble-be/src/main/java/com/baidu/hugegraph/service/query/ExecuteHistoryService.java

@@ -99,7 +101,8 @@ public ExecuteHistory get(int connId, int id) {
            try {
                Task task = client.task().get(history.getAsyncId());
                history.setDuration(task.updateTime() - task.createTime());
-                history.setAsyncStatus(AsyncTaskStatus.valueOf(task.status().toUpperCase()));
+                history.setAsyncStatus(AsyncTaskStatus.valueOf(


prefer add a method setAsyncStatus(string)

javeme · 2020-10-28T06:06:17Z

hubble-be/src/main/java/com/baidu/hugegraph/service/load/LoadTaskService.java

-        }
+        this.save(task);
+        // executed in other threads
+        this.taskExecutor.execute(task, () -> this.update(task));


should handle exception if the update is failed?

no need, it doesn't affect other tasks to use the thread pool

zhoney · 2020-10-29T12:20:07Z

hubble-be/src/main/java/com/baidu/hugegraph/controller/load/FileMappingController.java

@@ -130,18 +132,14 @@ public FileMapping fileSetting(@PathVariable("id") int id,
        if (mapping == null) {
            throw new ExternalException("load.file-mapping.not-exist.id", id);
        }
-        // unescape \\t to \t
-        newEntity.unescapeDelimiterIfNeeded();
+        // change format to TEXT if needed


// Change format to TEXT if needed

zhoney · 2020-10-29T12:22:14Z

hubble-be/src/main/java/com/baidu/hugegraph/entity/load/FileMapping.java

+
+    @JsonIgnore
+    public Set<String> getVertexMappingLabels() {
+        if (this.getVertexMappings() != null) {


if (this.getVertexMappings() != null) { return new HashSet<>(); } return this.getVertexMappings().stream() .map(ElementMapping::getLabel) .collect(Collectors.toSet());

zhoney · 2020-10-29T12:22:23Z

hubble-be/src/main/java/com/baidu/hugegraph/entity/load/FileMapping.java

+
+    @JsonIgnore
+    public Set<String> getEdgeMappingLabels() {
+        if (this.getEdgeMappings() != null) {


zhoney · 2020-10-29T12:23:10Z

hubble-be/src/main/java/com/baidu/hugegraph/entity/load/LoadTask.java

-                this.status = LoadStatus.FAILED;
+        this.lock.lock();
+        try {
+            // Pay attention to whether the user stops actively or


zhoney · 2020-10-29T12:23:51Z

hubble-be/src/main/java/com/baidu/hugegraph/entity/load/LoadTask.java

        float rate;
-        if (this.fileReadLines == null || this.duration == null ||
-            this.duration == 0L) {
+        if (readLines == 0L || duration == 0L) {
            rate = 0;


rate = 0.0F

Linary force-pushed the 1.5.0-bug-fix branch 3 times, most recently from 7e582ac to efe4ed9 Compare September 29, 2020 14:14

javeme reviewed Sep 30, 2020

View reviewed changes

Linary force-pushed the 1.5.0-bug-fix branch 2 times, most recently from 9c40dc3 to e794dd3 Compare October 16, 2020 09:58

javeme reviewed Oct 16, 2020

View reviewed changes

Some bug fix for load task and schema manage

f92fac4

Linary force-pushed the 1.5.0-bug-fix branch from e794dd3 to f92fac4 Compare October 19, 2020 06:05

javeme reviewed Oct 19, 2020

View reviewed changes

Linary force-pushed the 1.5.0-bug-fix branch from 664633b to 2a314fb Compare October 20, 2020 03:28

javeme reviewed Oct 20, 2020

View reviewed changes

Linary force-pushed the 1.5.0-bug-fix branch 2 times, most recently from d67f793 to 670d887 Compare October 22, 2020 06:04

tiny improve

abf84a3

Linary force-pushed the 1.5.0-bug-fix branch from 670d887 to abf84a3 Compare October 22, 2020 07:20

add API for JobManager status controling

653896d

javeme reviewed Oct 23, 2020

View reviewed changes

Linary force-pushed the 1.5.0-bug-fix branch from de9dbe8 to a92e9b3 Compare October 26, 2020 11:22

javeme reviewed Oct 26, 2020

View reviewed changes

Use background thread to delete unfinished files

405b75c

Linary force-pushed the 1.5.0-bug-fix branch from a92e9b3 to 405b75c Compare October 27, 2020 02:39

wrap save/update/remove method, and rename finish API to next-step

eaf4494

javeme reviewed Oct 28, 2020

View reviewed changes

tiny improve

2ae54a7

javeme previously approved these changes Oct 28, 2020

View reviewed changes

zhoney suggested changes Oct 29, 2020

View reviewed changes

tiny improve

236472f

Linary dismissed javeme’s stale review via 236472f October 30, 2020 02:38

zhoney approved these changes Oct 30, 2020

View reviewed changes

javeme approved these changes Oct 30, 2020

View reviewed changes

Linary merged commit 872887b into master Oct 30, 2020

Linary deleted the 1.5.0-bug-fix branch October 30, 2020 05:08

OshotOkill pushed a commit to OshotOkill/hugegraph-hubble that referenced this pull request Nov 3, 2020

Some bug fix for load task and schema manage (hugegraph#246)

8b401f5

OshotOkill pushed a commit to OshotOkill/hugegraph-hubble that referenced this pull request Nov 3, 2020

Some bug fix for load task and schema manage (hugegraph#246)

3f14a56

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Some bug fix for load task and schema manage #246

Some bug fix for load task and schema manage #246

Linary commented Sep 29, 2020

javeme Sep 30, 2020

javeme Sep 30, 2020

javeme Sep 30, 2020

javeme Oct 16, 2020

javeme Oct 16, 2020

javeme Oct 16, 2020

javeme Oct 16, 2020

javeme Oct 16, 2020

javeme Oct 16, 2020

Linary Oct 19, 2020

javeme Oct 19, 2020

javeme Oct 19, 2020

javeme Oct 19, 2020

javeme Oct 19, 2020

javeme Oct 19, 2020

javeme Oct 19, 2020

javeme Oct 20, 2020

javeme Oct 23, 2020

javeme Oct 23, 2020

javeme Oct 26, 2020

javeme Oct 26, 2020

javeme Oct 26, 2020

javeme Oct 27, 2020

javeme Oct 26, 2020

javeme Oct 26, 2020

javeme Oct 26, 2020

javeme Oct 28, 2020

Linary Oct 28, 2020

zhoney Oct 29, 2020

zhoney Oct 29, 2020

zhoney Oct 29, 2020

zhoney Oct 29, 2020

zhoney Oct 29, 2020

Some bug fix for load task and schema manage #246

Some bug fix for load task and schema manage #246

Conversation

Linary commented Sep 29, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment