Using HDFS to backup and restore notebook #1600

aspen01 · 2016-11-04T15:41:02Z

What is this PR for?

This PR supports using HDFS to backup and restore notebook.
It is similar to #1479.
However this PR just use WebHDFS API, so we don't need to care about hadoop libraries' dependency.

What type of PR is it?

Improvement

Todos

- Task

What is the Jira issue?

https://issues.apache.org/jira/browse/ZEPPELIN-1515

How should this be tested?

Set the variables in zeppelin-site.xml

zeppelin.notebook.storage : org.apache.zeppelin.notebook.repo.HDFSNotebookRepo
hdfs.url : Hadoop WebHDFS URL. default: http://localhost:50070/webhdfs/v1/
hdfs.user : HDFS user. default: hdfs
hdfs.maxlength : Maximum number of lines of results fetched. default: 100000
hdfs.notebook.dir : notebook location directory in HDFS. default: /tmp

After zeppelin daemon start, check the notebook directories in hdfs.notebook.dir.

Screenshots (if appropriate)

Questions:

Does the licenses files need update? No
Is there breaking changes for older versions? No
Does this needs documentation? Yes

bzz · 2016-11-24T02:19:49Z

@aspen01 thank you for contribution!

Could you please double-check that

all new files, as any other file in Apache Zeppelin, have ASF's Apache 2.0 license header?
all Java files adhere project's code conventions?

As for copyright notices, could you please help to understand if your contribution contains any third-party works or is that all code, developed for the ASF?

That would determine, if the existing copyright notice from the files should be moved to root NOTICE file or not. Thanks!

khalidhuseynov · 2016-11-28T06:38:45Z

zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/repo/HDFSNotebookRepo.java

+  public synchronized void save(Note note, AuthenticationInfo subject) throws IOException {
+    super.save(note, subject);
+    if (this.enableWebHDFS)
+      uploadNoteListToHDFS(note);


what do you think about:

whether makes sense to pass user from subject instead of getting user from environment inside HDFSCommand

from this function name seems like uploading whole list instead of a single note

If this feature save a notebook to hdfs directly, I think your opinion is right. But now the feature is just backup the notebook to hdfs, so I tried to think of the two functions(connecting hdfs and save the notebook) separately.

I'll fix it. Thanks.

khalidhuseynov · 2016-11-28T06:39:47Z

zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/repo/HDFSNotebookRepo.java

+      String notebookStatus = this.hdfsCmd.runCommand(this.hdfsCmd.getFileStatus, notebook, null);
+      fileStatus = gson.fromJson(notebookStatus, OneFileStatus.class);
+    } catch (Exception e) {
+      logger.warn("Warning: ", e);


maybe some description to give here

khalidhuseynov · 2016-11-28T06:44:01Z

zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/repo/HDFSNotebookRepo.java

+      }
+    }
+    catch (Exception e) {
+      logger.error("Exception: ", e);


a bit more description would help here as well

I'll add more description. Thanks.

placeybordeaux · 2016-12-05T18:50:03Z

zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/repo/HDFSNotebookRepo.java

@@ -0,0 +1,241 @@
+/**
+ * Copyright 2015 NAVER Corp. All rights Reserved.


Is this file licensed for use in an Apache project?

It is wrong license. I'll remove.

Please make sure all the code is distributed under one of the ASF category-a licenses

khalidhuseynov · 2016-12-08T02:16:36Z

@aspen01 I wonder if any progress on this one?

zjffdu · 2016-12-11T02:45:10Z

docs/storage/storage.md

+</property>
+
+<property>
+  <name>hdfs.url</name>


I think it is better to add prefix zeppelin.notebook, because hdfs is used in many places. Adding prefix can help user to understand this this is for notebook storage.

HDFSCommand use hdfs.url property already. So I used the same property name.
If adding prefix can help user to understand, I'll add prefix.

zjffdu · 2016-12-11T03:10:36Z

zeppelin-zengine/pom.xml

@@ -42,6 +42,18 @@
    </dependency>

    <dependency>
+      <groupId>${project.groupId}</groupId>


I don't think it is a good idea to make zengine to depend on one specific interpreter.

I agree with you. But To use HDFSCommand functions, I had to add dependency.
Could you give me advice to solve this?

Indeed, Zengine should not have such dependency.

@aspen01 if HDFSCommand have to be shared:

one option is to move it to zeppelin-interpreter, as both file and zeppelin-zengine already depend on it

another option to consider would be - extract a new maven sub-module out of zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/repo/, move HDFSCommand there and make other modules depend on it.

Second option is more involved and deserves a separate issue \w discussion. First one looks more feasible

aspen01 · 2016-12-19T13:20:11Z

@bzz Thank you for advice. But the build is failed even though I fixed license and checkstyle.

bzz · 2016-12-20T01:10:31Z

I see, thank you for taking care! This looks like something failing on the TravisCI side though.

Could you please rebase this branch on latest master and force-push it here again? This will trigger CI \w latest code\fixes from master.

placeybordeaux · 2016-12-21T20:47:20Z

Looks like you're missing a hotfix commit:

1c7d8fb

bzz · 2016-12-22T03:07:44Z

Thank you @aspen01, @placeybordeaux

In order to be merged, this branch have to be rebased on top of the master, which itself already includes CI fixes, i.e 1c7d8fb.

This branch must not include any other commits, except for ones that implement "HDFS to backup and restore notebook" - edae7bc, d94eb16 need to be removed.

bzz · 2016-12-22T03:23:38Z

zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/repo/HDFSNotebookRepo.java

+      return "/" + newUrl.replaceAll("^hdfs://", "").split("/", 2)[1];
+    }
+    else
+      return newUrl;


Please, make sure all the code follows project style guide

aspen01 · 2016-12-22T03:32:25Z

@placeybordeaux Thank you.
@bzz Do you mean I have to merge these commits to master not branch-0.6?

add header for uploading file using REST API

aspen01 · 2016-12-28T10:00:34Z

I committed this branch on latest master and force-push it.

placeybordeaux · 2017-01-10T21:41:07Z

Not sure why the CI hasn't kicked in, but I tried compiling this on my local and I am getting some test failures:

Results :

Failed tests:
  NotebookTest.testAbortParagraphStatusOnInterpreterRestart:760 expected:<ABORT> but was:<RUNNING>

Tests in error:
  HeliumApplicationFactoryTest.testUnloadOnInterpreterUnbind:232 » ClassCast jav...
  HeliumApplicationFactoryTest.testLoadRunUnloadApplication:148 » ClassCast java...
  HeliumApplicationFactoryTest.testUnloadOnInterpreterRestart:299 » ClassCast ja...
  HeliumApplicationFactoryTest.testUnloadOnParagraphRemove:193 » ClassCast java....

Tests run: 149, Failures: 1, Errors: 4, Skipped: 0

Not sure if these are related to the PR at all.

aspen01 · 2017-01-11T14:21:01Z

@placeybordeaux How did you build a package?

$ mvn clean package -Pspark-1.6 -Phadoop-2.6 -Pyarn -Ppyspark -Psparkr -Pscala-2.10

...


Results :

Tests run: 66, Failures: 0, Errors: 0, Skipped: 0

...

[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 19:23 min
[INFO] Finished at: 2017-01-11T23:00:58+09:00
[INFO] Final Memory: 160M/815M
[INFO] ------------------------------------------------------------------------

placeybordeaux · 2017-01-11T23:43:02Z

I built with mvn clean package, but got the same results with mvn clean package -Pspark-1.6 -Phadoop-2.6 -Pyarn -Ppyspark -Psparkr -Pscala-2.10. It's likely this is just a problem on my side.

keithchambers · 2017-02-04T16:24:25Z

Is this ready to be merged?

EronWright · 2017-03-10T23:04:23Z

Seems overcomplicated for HDFSNotebookRepo to extend VFSNotebookRepo, and to be synchronizing the local FS with the remote FS. Just implement NotebookRepo directly.

aspen01 · 2017-05-10T13:47:38Z

@EronWright Thank you for your feedback. When I first developed the feature, I assumed that HDFS would not be used as default storage because remote FS failure could affect zeppelin usage. So extended VFSNotebookRepo to keep the default storage space on local FS and HDFS as backup storage for failover. This makes the implementation is simple, but it may have created complexity.
I'm using this feature, but I am also considering using a third-party storage such as HBase, because the latency of HDFS affects the latency of zeppelin notebooks.

close #83 close #86 close #125 close #133 close #139 close #146 close #193 close #203 close #246 close #262 close #264 close #273 close #291 close #299 close #320 close #347 close #389 close #413 close #423 close #543 close #560 close #658 close #670 close #728 close #765 close #777 close #782 close #783 close #812 close #822 close #841 close #843 close #878 close #884 close #918 close #989 close #1076 close #1135 close #1187 close #1231 close #1304 close #1316 close #1361 close #1385 close #1390 close #1414 close #1422 close #1425 close #1447 close #1458 close #1466 close #1485 close #1492 close #1495 close #1497 close #1536 close #1545 close #1561 close #1577 close #1600 close #1603 close #1678 close #1695 close #1739 close #1748 close #1765 close #1767 close #1776 close #1783 close #1799

aspen01 mentioned this pull request Nov 4, 2016

Notebook: HDFS as a backend storage (Read & Write Mode) #1479

Closed

1 task

khalidhuseynov reviewed Nov 28, 2016

View reviewed changes

placeybordeaux reviewed Dec 5, 2016

View reviewed changes

zjffdu reviewed Dec 11, 2016

View reviewed changes

bzz reviewed Dec 22, 2016

View reviewed changes

aspen01 added 7 commits December 28, 2016 18:20

Using HDFS to backup and restore notebook

fd94a51

add hadoop pseudo auth in GETFILESTATUS, LISTSTATUS

c8643a1

add header for uploading file using REST API

always add hadoop pseudo authentication

fdb48f3

add document for HDFSNotebookRepo

5ce0c39

adding Apache 2.0 license header and exception message

a955a21

fix checkstyle for hdfs notebook

815a1ee

change method for getting note's id

123a074

aspen01 force-pushed the branch-0.6 branch from d94eb16 to 123a074 Compare December 28, 2016 09:55

aspen01 changed the base branch from branch-0.6 to master December 28, 2016 09:56

asfgit closed this in c38a0a0 May 9, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Using HDFS to backup and restore notebook #1600

Using HDFS to backup and restore notebook #1600

aspen01 commented Nov 4, 2016

bzz commented Nov 24, 2016 •

edited

Loading

khalidhuseynov Nov 28, 2016 •

edited

Loading

aspen01 Dec 16, 2016

khalidhuseynov Nov 28, 2016

khalidhuseynov Nov 28, 2016

aspen01 Dec 16, 2016

placeybordeaux Dec 5, 2016

aspen01 Dec 16, 2016

bzz Dec 22, 2016

khalidhuseynov commented Dec 8, 2016

zjffdu Dec 11, 2016

aspen01 Dec 16, 2016

zjffdu Dec 11, 2016

aspen01 Dec 16, 2016

bzz Dec 22, 2016

aspen01 commented Dec 19, 2016

bzz commented Dec 20, 2016 •

edited

Loading

placeybordeaux commented Dec 21, 2016

bzz commented Dec 22, 2016

bzz Dec 22, 2016

aspen01 commented Dec 22, 2016 •

edited

Loading

aspen01 commented Dec 28, 2016

placeybordeaux commented Jan 10, 2017

aspen01 commented Jan 11, 2017

placeybordeaux commented Jan 11, 2017

keithchambers commented Feb 4, 2017

EronWright commented Mar 10, 2017

aspen01 commented May 10, 2017

		@@ -0,0 +1,241 @@
		/**
		* Copyright 2015 NAVER Corp. All rights Reserved.

Using HDFS to backup and restore notebook #1600

Using HDFS to backup and restore notebook #1600

Conversation

aspen01 commented Nov 4, 2016

What is this PR for?

What type of PR is it?

Todos

What is the Jira issue?

How should this be tested?

Screenshots (if appropriate)

Questions:

bzz commented Nov 24, 2016 • edited Loading

khalidhuseynov Nov 28, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

khalidhuseynov commented Dec 8, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aspen01 commented Dec 19, 2016

bzz commented Dec 20, 2016 • edited Loading

placeybordeaux commented Dec 21, 2016

bzz commented Dec 22, 2016

Choose a reason for hiding this comment

aspen01 commented Dec 22, 2016 • edited Loading

aspen01 commented Dec 28, 2016

placeybordeaux commented Jan 10, 2017

aspen01 commented Jan 11, 2017

placeybordeaux commented Jan 11, 2017

keithchambers commented Feb 4, 2017

EronWright commented Mar 10, 2017

aspen01 commented May 10, 2017

bzz commented Nov 24, 2016 •

edited

Loading

khalidhuseynov Nov 28, 2016 •

edited

Loading

bzz commented Dec 20, 2016 •

edited

Loading

aspen01 commented Dec 22, 2016 •

edited

Loading