Fix two bug about single tasks and executor #12

Linary · 2018-10-17T09:47:00Z

When a single insert task times out, it may still cause statistical errors.
TashManager.shutdown() ignored singleExecutor

Fix #11
Change-Id: I9a468241f369f4c7e73a95ba9a5772dac90e0ce1

zhoney · 2018-10-18T11:47:34Z

src/main/java/com/baidu/hugegraph/loader/executor/LoadOptions.java

@@ -110,6 +110,10 @@ private LoadOptions() {}
               description = "Setting the interval time before retrying")
    public int retryInterval = 10;

+    @Parameter(names = {"--continue-when-single-task-timeout"}, arity = 1,
+               description = "Setting the interval time before retrying")


no need to add this argument. Loader will wait until all elements inserted except insert errors element.

zhoney · 2018-10-18T11:52:35Z

src/main/java/com/baidu/hugegraph/loader/task/TaskManager.java

@@ -286,7 +151,7 @@ private void printProgress(String type, long frequency, int batchSize) {
        }
    }

-    private void ensurePoolAvailable() {
+    protected void ensurePoolAvailable() {
        while (this.batchService.isShutdown()){


thread pool can be used immediately after created. no need to wait here.

zhoney · 2018-10-18T11:58:57Z

src/main/java/com/baidu/hugegraph/loader/executor/LoadOptions.java

+    @Parameter(names = {"--continue-when-single-task-timeout"}, arity = 1,
+               description = "Setting the interval time before retrying")
+    public boolean continueWhenSingleTaskTimeout = true;
+
    @Parameter(names = {"--test-mode"}, arity = 1,
               description = "Whether the hugegraph-loader work in test mode")
    public boolean testMode = false;


also add “--help” subcommand to print usage information

zhoney · 2018-10-18T12:01:36Z

src/main/java/com/baidu/hugegraph/loader/task/TaskManager.java

+    protected final int futureNum;
+    protected final Semaphore available;
+    protected final ListeningExecutorService batchService;
+    protected final ExecutorService singleExecutor;


use multi-thread to improve single-insertion performance

javeme · 2018-10-19T07:16:48Z

src/main/java/com/baidu/hugegraph/loader/HugeGraphLoader.java

@@ -161,7 +151,10 @@ private void createSchema() {
        groovyExecutor.execute(script, client);
    }

-    private void loadVertices() {
+    private void loadVertices(LoadSummary summary) {
+        Instant begTime = Instant.now();


javeme · 2018-10-19T07:24:42Z

src/main/java/com/baidu/hugegraph/loader/task/TaskManager.java

            if (this.singleTasks.longValue() != 0L) {
-                return false;
+                throw new IllegalStateException(


E.checkState

javeme · 2018-10-19T07:38:22Z

src/main/java/com/baidu/hugegraph/loader/task/EdgeInsertTask.java


-    InsertEdgeTask(List<Edge> batch) {
+    EdgeInsertTask(List<Edge> batch) {


javeme · 2018-10-19T07:38:38Z

src/main/java/com/baidu/hugegraph/loader/task/EdgeInsertTask.java

@@ -24,9 +24,9 @@
 import com.baidu.hugegraph.loader.executor.LoadOptions;
 import com.baidu.hugegraph.structure.graph.Edge;

-public class InsertEdgeTask extends InsertTask<Edge> {
+public class EdgeInsertTask extends InsertTask<Edge> {


InsertionTask

javeme · 2018-10-19T07:40:16Z

src/main/java/com/baidu/hugegraph/loader/task/TaskManager.java

+        this.available.acquire(this.futureNum);
+    }
+
+    private void waitSingleTaskFinished() throws InterruptedException {


waitSingleTasksFinished

javeme · 2018-10-19T07:41:44Z

src/main/java/com/baidu/hugegraph/loader/task/TaskManager.java

+
+    private void waitSingleTaskFinished() throws InterruptedException {
+        long total = this.singleTasks.longValue();
+        for (long i = 0; i < total; i++) {


i < this.singleTasks.longValue()

Linary · 2018-10-23T02:34:00Z

Already edited

javeme · 2018-10-23T03:00:49Z

src/main/java/com/baidu/hugegraph/loader/task/TaskManager.java

        try {
            // Wait batch task finished
-            this.available.acquire(this.futureNum);
+            this.batchSemaphore.acquire(options.numThreads);
+            LOG.info("Batch tasks of {} finished", type);


Batch-mode tasks

javeme · 2018-10-23T03:02:16Z

src/main/java/com/baidu/hugegraph/loader/HugeGraphLoader.java

+                try {
+                    reader.close();
+                } catch (Throwable e) {
+                    LOG.warn("Failed to close parser for edge source {}",


Failed to close reader

javeme · 2018-10-23T03:05:22Z

src/main/java/com/baidu/hugegraph/loader/HugeGraphLoader.java

-                if (++this.parseFailureNum >= options.maxParseErrors) {
-                    exitWithInfo("vertices", options.maxParseErrors);
+                if (++this.parseFailureNum >= this.options.maxParseErrors) {
+                    LOG.error("Too many vertices parse error ... Stopping");


delete it since LOG.error() is called in printError()

javeme · 2018-10-23T03:05:40Z

src/main/java/com/baidu/hugegraph/loader/HugeGraphLoader.java

-                if (++this.parseFailureNum >= options.maxParseErrors) {
-                    exitWithInfo("edges", options.maxParseErrors);
+                if (++this.parseFailureNum >= this.options.maxParseErrors) {
+                    LOG.error("Too many edges parse error ... Stopping");


javeme · 2018-10-23T03:06:18Z

src/main/java/com/baidu/hugegraph/loader/task/TaskManager.java

-                return false;
-            }
+            this.singleSemaphore.acquire(options.numThreads);
+            LOG.info("Single tasks of {} finished", type);


Single-mode tasks

javeme · 2018-10-23T03:06:31Z

src/main/java/com/baidu/hugegraph/loader/task/TaskManager.java

        } catch (InterruptedException e) {
-            LOG.error("Tasks is interrupted.");
+            LOG.error("The batch tasks are interrupted");


ditto, check somewhere else

javeme · 2018-10-23T03:12:57Z

src/main/java/com/baidu/hugegraph/loader/task/TaskManager.java

+            public void onFailure(Throwable t) {
+                singleSemaphore.release();
+            }
+        });


move line 282 to LoadUtil

src/main/java/com/baidu/hugegraph/loader/HugeGraphLoader.java

javeme · 2018-10-23T03:20:35Z

src/main/java/com/baidu/hugegraph/loader/HugeGraphLoader.java

@@ -100,43 +107,29 @@ private void parseAndCheckOptions(String[] args) {
    }

    private LoadSummary load() {
+        LoadOptions options = LoadOptions.instance();


remove it, and remove instance() method from class LoadOptions too

1. When a single insert task times out, it may still cause statistical errors. 2. TaskManager.shutdown() ignored singleExecutor Fix #12 Change-Id: I9a468241f369f4c7e73a95ba9a5772dac90e0ce1

Linary · 2018-10-26T02:15:34Z

already edited

javeme · 2018-10-26T02:34:04Z

src/main/java/com/baidu/hugegraph/loader/HugeGraphLoader.java

        GroovyExecutor groovyExecutor = new GroovyExecutor();
        groovyExecutor.bind("schema", client.schema());
        String script;
        try {
            script = FileUtils.readFileToString(schemaFile, "UTF-8");
        } catch (IOException e) {
            throw new LoadException("Read schema file '%s' error",
-                                    e, options.schema);
+                                    e, this.options.schema);


move '%s' to the end

Will be modified in the issue #15

javeme · 2018-10-26T02:37:49Z

src/main/java/com/baidu/hugegraph/loader/util/LoaderUtil.java

    public static String backward(long word) {
        StringBuilder backward = new StringBuilder();
        for (int i = 0, len = String.valueOf(word).length(); i <= len; i++) {
            backward.append("\b");
        }
        return backward.toString();
    }
+


move backward() to the bottom of file

javeme · 2018-10-26T02:44:24Z

src/main/java/com/baidu/hugegraph/loader/task/TaskManager.java

+        this.batchSemaphore = new Semaphore(options.numThreads);
+        this.singleSemaphore = new Semaphore(options.numThreads);
+        this.batchService = MoreExecutors.listeningDecorator(
+                            Executors.newFixedThreadPool(options.numThreads));


ExecutorUtil.newFixedThreadPool

javeme · 2018-10-26T02:46:47Z

src/main/java/com/baidu/hugegraph/loader/task/TaskManager.java

-                // That's fine, just continue.
-            }
+        if (this.batchService.isShutdown()) {
+            throw new LoadException("Thread pool has been closed");


add singleService check too

edited

javeme · 2018-10-29T08:57:03Z

src/main/java/com/baidu/hugegraph/loader/task/TaskManager.java

+        this.batchSemaphore = new Semaphore(options.numThreads);
+        this.singleSemaphore = new Semaphore(options.numThreads);
+        this.batchService = MoreExecutors.listeningDecorator(
+                            Executors.newFixedThreadPool(options.numThreads));


javeme · 2018-10-29T09:01:44Z

src/main/java/com/baidu/hugegraph/loader/util/LoaderUtil.java

+    }
+
+    public static void printInBackward(long count) {
+        System.out.print(String.format(" %d%s", count,


remove the first space and let backward() return the size without space
remember update loadVertices and loadEdges

done

zhoney · 2018-11-09T09:08:17Z

src/main/java/com/baidu/hugegraph/loader/HugeGraphLoader.java

+                    reader.close();
+                } catch (Throwable e) {
+                    LOG.warn("Failed to close reader for vertex source {}",
+                             source, e);


LOG.warn("Failed to close reader for vertex source {} with exception {}",source, e)

zhoney · 2018-11-09T09:57:50Z

src/main/java/com/baidu/hugegraph/loader/HugeGraphLoader.java

+                try {
+                    reader.close();
+                } catch (Throwable e) {
+                    LOG.warn("Failed to close reader for edge source {}",


1. When a single insert task times out, it may still cause statistical errors. 2. TaskManager.shutdown() ignored singleExecutor Fix #11 Change-Id: I9a468241f369f4c7e73a95ba9a5772dac90e0ce1

Fix #12 Change-Id: I2ae4e4e6e3d1749981155e477887a8a4269a26ba

zhoney requested changes Oct 18, 2018

View reviewed changes

zhoney previously requested changes Oct 18, 2018

View reviewed changes

Linary force-pushed the split-taskmanager branch from 94048a1 to 4afccad Compare October 18, 2018 12:21

javeme requested changes Oct 19, 2018

View reviewed changes

Linary force-pushed the split-taskmanager branch from 4afccad to 4e800e3 Compare October 19, 2018 09:48

javeme requested changes Oct 23, 2018

View reviewed changes

Linary force-pushed the split-taskmanager branch 2 times, most recently from 487255e to 40e96b3 Compare October 23, 2018 07:10

Linary force-pushed the split-taskmanager branch from 40e96b3 to 5b59b0b Compare October 23, 2018 08:39

javeme approved these changes Oct 26, 2018

View reviewed changes

javeme requested changes Oct 26, 2018

View reviewed changes

javeme previously requested changes Oct 26, 2018

View reviewed changes

Linary force-pushed the split-taskmanager branch from 5b59b0b to 84c5868 Compare October 26, 2018 02:56

javeme previously requested changes Oct 29, 2018

View reviewed changes

Linary mentioned this pull request Oct 30, 2018

hugeloader的导入原理是相当于 hugeclient的batch 方式吗,还是single方式? apache/incubator-hugegraph#135

Closed

Linary force-pushed the split-taskmanager branch from 84c5868 to a2ba0c9 Compare October 30, 2018 03:22

javeme previously approved these changes Nov 6, 2018

View reviewed changes

zhoney requested changes Nov 9, 2018

View reviewed changes

Linary dismissed javeme’s stale review via ed1d263 November 23, 2018 06:36

Linary force-pushed the split-taskmanager branch from a2ba0c9 to ed1d263 Compare November 23, 2018 06:36

Fix two bug about single tasks and executor

8421f70

1. When a single insert task times out, it may still cause statistical errors. 2. TaskManager.shutdown() ignored singleExecutor Fix #11 Change-Id: I9a468241f369f4c7e73a95ba9a5772dac90e0ce1

Linary force-pushed the split-taskmanager branch from ed1d263 to 8421f70 Compare November 23, 2018 06:43

zhoney approved these changes Nov 23, 2018

View reviewed changes

javeme approved these changes Nov 23, 2018

View reviewed changes

javeme merged commit a0b27a1 into master Nov 23, 2018

javeme deleted the split-taskmanager branch November 23, 2018 08:03

imbajin pushed a commit that referenced this pull request Apr 14, 2022

Fix get_ip failed when doesn't install ifconfig

9151e83

Fix #12 Change-Id: I2ae4e4e6e3d1749981155e477887a8a4269a26ba

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix two bug about single tasks and executor #12

Fix two bug about single tasks and executor #12

Linary commented Oct 17, 2018 •

edited by javeme

Loading

zhoney Oct 18, 2018 •

edited

Loading

zhoney Oct 18, 2018

zhoney Oct 18, 2018

zhoney Oct 18, 2018

javeme Oct 19, 2018

javeme Oct 19, 2018

javeme Oct 19, 2018

javeme Oct 19, 2018

javeme Oct 19, 2018

javeme Oct 19, 2018

Linary commented Oct 23, 2018

javeme Oct 23, 2018

javeme Oct 23, 2018

javeme Oct 23, 2018

javeme Oct 23, 2018

javeme Oct 23, 2018

javeme Oct 23, 2018

javeme Oct 23, 2018

javeme Oct 23, 2018

Linary commented Oct 26, 2018

javeme Oct 26, 2018

Linary Oct 26, 2018

javeme Oct 26, 2018

javeme Oct 26, 2018

javeme Oct 29, 2018

javeme Oct 26, 2018

javeme Oct 29, 2018

javeme Oct 29, 2018

zhoney Nov 9, 2018

zhoney Nov 9, 2018


		InsertEdgeTask(List<Edge> batch) {
		EdgeInsertTask(List<Edge> batch) {

Fix two bug about single tasks and executor #12

Fix two bug about single tasks and executor #12

Conversation

Linary commented Oct 17, 2018 • edited by javeme Loading

zhoney Oct 18, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Linary commented Oct 23, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Linary commented Oct 26, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Linary commented Oct 17, 2018 •

edited by javeme

Loading

zhoney Oct 18, 2018 •

edited

Loading