ZEPPELIN-885 Add option to send email in scheduler execution #918

vgmartinez · 2016-05-25T21:50:29Z

What is this PR for?

Add the option to send email when a note is runing using the scheduler.
The option to send an email when you finish executing a note or when an error occurs in one paragraph is a good utility for the user.

What type of PR is it?

Improvement

Todos

What is the Jira issue?

ZEPPELIN-885

How should this be tested?

Configure smtp properties in zeppelin-site.xml
Set email in scheduler and add cron expresion to execute note.

Screenshots (if appropriate)

Questions:

Does the licenses files need update? No
Is there breaking changes for older versions? No
Does this needs documentation? No

cloverhearts · 2016-05-26T02:44:13Z

Thank you for good feature.
I like this feature.

but, i got following problem.

Zeppelin Server log.

ERROR [2016-05-26 11:27:01,752] ({DefaultQuartzScheduler_Worker-2} Notebook.java[sendEmail]:594) - Error: 
org.apache.commons.mail.EmailException: Sending the email to the following server failed : smtp.googlemail.com:465
    at org.apache.commons.mail.Email.sendMimeMessage(Email.java:1421)
    at org.apache.commons.mail.Email.send(Email.java:1448)
    at org.apache.zeppelin.notebook.Notebook.sendEmail(Notebook.java:591)
    at org.apache.zeppelin.notebook.Notebook$CronJob.execute(Notebook.java:513)
    at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
    at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
Caused by: javax.mail.SendFailedException: Send failure (javax.mail.AuthenticationFailedException: 535-5.7.8 Username and Password not accepted. Learn more at
535 5.7.8  https://support.google.com/mail/answer/14257 z12sm1446736pfj.1 - gsmtp
)
    at javax.mail.Transport.send(Transport.java:163)
    at javax.mail.Transport.send(Transport.java:48)
    at org.apache.commons.mail.Email.sendMimeMessage(Email.java:1411)
    ... 5 more
Caused by: javax.mail.AuthenticationFailedException: 535-5.7.8 Username and Password not accepted. Learn more at
535 5.7.8  https://support.google.com/mail/answer/14257 z12sm1446736pfj.1 - gsmtp

    at com.sun.mail.smtp.SMTPTransport$Authenticator.authenticate(SMTPTransport.java:892)
    at com.sun.mail.smtp.SMTPTransport.authenticate(SMTPTransport.java:814)
    at com.sun.mail.smtp.SMTPTransport.protocolConnect(SMTPTransport.java:728)
    at javax.mail.Service.connect(Service.java:274)
    at javax.mail.Service.connect(Service.java:91)
    at javax.mail.Service.connect(Service.java:76)
    at javax.mail.Transport.send(Transport.java:94)
    ... 7 more
ERROR [2016-05-26 11:27:03,392] ({DefaultQuartzScheduler_Worker-2} Notebook.java[sendEmail]:594) - Error: 
org.apache.commons.mail.EmailException: Sending the email to the following server failed : smtp.googlemail.com:465
    at org.apache.commons.mail.Email.sendMimeMessage(Email.java:1421)
    at org.apache.commons.mail.Email.send(Email.java:1448)
    at org.apache.zeppelin.notebook.Notebook.sendEmail(Notebook.java:591)
    at org.apache.zeppelin.notebook.Notebook$CronJob.execute(Notebook.java:539)
    at org.quartz.core.JobRunShell.run(JobRunShell.java:202)
    at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573)
Caused by: javax.mail.SendFailedException: Send failure (javax.mail.AuthenticationFailedException: 535-5.7.8 Username and Password not accepted. Learn more at
535 5.7.8  https://support.google.com/mail/answer/14257 f9sm1425655pfd.56 - gsmtp
)
    at javax.mail.Transport.send(Transport.java:163)
    at javax.mail.Transport.send(Transport.java:48)
    at org.apache.commons.mail.Email.sendMimeMessage(Email.java:1411)
    ... 5 more
Caused by: javax.mail.AuthenticationFailedException: 535-5.7.8 Username and Password not accepted. Learn more at
535 5.7.8  https://support.google.com/mail/answer/14257 f9sm1425655pfd.56 - gsmtp

    at com.sun.mail.smtp.SMTPTransport$Authenticator.authenticate(SMTPTransport.java:892)
    at com.sun.mail.smtp.SMTPTransport.authenticate(SMTPTransport.java:814)
    at com.sun.mail.smtp.SMTPTransport.protocolConnect(SMTPTransport.java:728)
    at javax.mail.Service.connect(Service.java:274)
    at javax.mail.Service.connect(Service.java:91)
    at javax.mail.Service.connect(Service.java:76)
    at javax.mail.Transport.send(Transport.java:94)
    ... 7 more

My Environment : Mac OS elcapitan, chrome lastests.
my note context : "println("hellow");

cron time 1m;

email : onStart : [email protected]
email : onError : [email protected]
email : onSuccess : [email protected]

astroshim · 2016-05-26T04:03:19Z

I tested and working well.
@cloverhearts maybe you need to set up the smtp informations. you can set the variables in the "conf/zeppelin-site.xml".

felixcheung · 2016-05-26T06:38:27Z

cool - should we have the info in documentation?

cloverhearts · 2016-05-26T06:47:11Z

@astroshim Oh, I see. Thank you!

bzz · 2016-05-26T07:46:18Z

@vgmartinez Thank you for contribution!

Before a deep dive into the code, I have a question - could you pelase explain the use-case you have in mind for this feature?

I'm trying to understand in particular what is the benefit of introducing complexity of SNTP support in the Zeppelin codebase (not sure that Notebook.java is the best place for it either), versus just scheduling a notebook using existing scheduller, that just loads the SNTP library (using %dep) and then sends any template out.

See #8 from this notebook for an example of the data product, that uses this approach for sending out email notifications.

vgmartinez · 2016-05-26T08:02:59Z

@cloverhearts @astroshim @felixcheung thks, add more info and doc ASAP.

hi @bzz, I think having the option to send an email when you finish executing a note or when an error occurs in one paragraph is a good utility for the user.
e.g if you have cron expression to run every monday is useful to receive an email when it starts, when finished or when there is an error.
what do you think?

Leemoonsoo · 2016-05-26T22:09:41Z

@vgmartinez Thanks for contribution.

@vgmartinez and folks, how about having generalized event hook instead of just handling email after scheduler execution?

Zeppelin can generate events,

Scheduler execution (start, success, fail)
Paragraph execution (CRUD, status change)
Notebook (CRUD)
etc

And if there're some extensible mechanism that allow user plugin custom event handler. So possibly not only sending email, but also execute shell script or any other action will be possible for all type of events.

What do you think?

vgmartinez · 2016-05-29T20:21:27Z

Hi @Leemoonsoo, I totally agree with you. I'll think a bit about this topic, if you already have some idea we can debate.

thanks for the suggestion.

bzz · 2016-05-31T02:23:05Z

@vgmartinez thank you for explaining the use case, it makes perfect sense and is not covered by the case I was referring to. May it's worth putting it in JIRA\PR description as well, how do you think?

@Leemoonsoo suggestion sounds really nice indeed. Even something simpler and less general with just an email would work as a first step, but email facilities better be placed outside of the Notebook.java. I presume it may very well be some kind of 'client' of the Notebook.java API inside zengine.

Leemoonsoo · 2016-05-31T03:58:45Z

@vgmartinez I think move email facilities from Notebook.java to new class, like @bzz mentioned, would be the first step to generalize it. Doesn't need to generalize everything from the beginning, but i think it worth to show possibilities of implementing more events and more actions. And taking code out from Notebook.java will help.

vgmartinez · 2016-05-31T21:59:02Z

@bzz @Leemoonsoo thks for review.
@bzz I'm sorry I have not explained well. I've added more description to JIRA/PR
I created a new class for events outside notebook, to use for other types of event notifications. Please take a look.

Leemoonsoo · 2016-05-31T22:37:42Z

zeppelin-zengine/src/main/java/org/apache/zeppelin/notebook/Notebook.java

  private StdSchedulerFactory quertzSchedFact;
  private org.quartz.Scheduler quartzSched;
  private JobListenerFactory jobListenerFactory;
  private NotebookRepo notebookRepo;
  private SearchService notebookIndex;
  private NotebookAuthorization notebookAuthorization;
+  private static EventNotification notification;


are there any special reasons making notification and conf static?

Leemoonsoo · 2016-05-31T22:37:53Z

Do you mind add new configurations to https://github.com/apache/incubator-zeppelin/blob/master/docs/install/install.md#zeppelin-configuration as well??

vgmartinez · 2016-06-06T22:16:40Z

@Leemoonsoo sorry for delay. I added the configurations in the file install.md and delete the static modifier.

Leemoonsoo · 2016-06-07T18:50:03Z

@vgmartinez No worries. CI build fails with error

[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 02:30 min
[INFO] Finished at: 2016-06-06T22:47:47+00:00
[INFO] Final Memory: 38M/471M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.rat:apache-rat-plugin:0.11:check (verify.rat) on project zeppelin-zengine: Too many files with unapproved license: 1 See RAT report in: /home/travis/build/apache/incubator-zeppelin/zeppelin-zengine/target/rat.txt -> [Help 1]

Could you take a look?

vgmartinez · 2016-06-13T21:10:23Z

@Leemoonsoo CI pass...

Leemoonsoo · 2016-06-14T22:45:59Z

@vgmartinez Thanks for update.

I tried and found 'Cancel' button is not actually discarding chances in dialog. Check following screenshot

vgmartinez · 2016-11-28T09:59:49Z

Hi @Leemoonsoo,
I have refactored the code to be able to add more channels of notifications, e.g slack, SNS (in the next PR)...
Please take a look...

thks

close #83 close #86 close #125 close #133 close #139 close #146 close #193 close #203 close #246 close #262 close #264 close #273 close #291 close #299 close #320 close #347 close #389 close #413 close #423 close #543 close #560 close #658 close #670 close #728 close #765 close #777 close #782 close #783 close #812 close #822 close #841 close #843 close #878 close #884 close #918 close #989 close #1076 close #1135 close #1187 close #1231 close #1304 close #1316 close #1361 close #1385 close #1390 close #1414 close #1422 close #1425 close #1447 close #1458 close #1466 close #1485 close #1492 close #1495 close #1497 close #1536 close #1545 close #1561 close #1577 close #1600 close #1603 close #1678 close #1695 close #1739 close #1748 close #1765 close #1767 close #1776 close #1783 close #1799

Leemoonsoo reviewed May 31, 2016
View reviewed changes

vgmartinez force-pushed the email_option branch from b3cdefa to 128bb58 Compare June 6, 2016 22:06

vgmartinez force-pushed the email_option branch 2 times, most recently from 92edccf to 8ab7fe6 Compare June 13, 2016 19:28

vgmartinez force-pushed the email_option branch from 8ab7fe6 to 3dbdba2 Compare November 11, 2016 14:00

vgmartinez added 4 commits November 27, 2016 22:43

rebase with master

93ed834

rebase master

8fbabfb

fix bug in angular copy

adae5e7

rebase with master

8d50f1a

vgmartinez force-pushed the email_option branch from 1469364 to 8d50f1a Compare November 27, 2016 21:56

vgmartinez added 2 commits November 27, 2016 23:02

back change in tutorial note

8fe4505

fix end line and add isError funtions

8bca02c

asfgit closed this in c38a0a0 May 9, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

ZEPPELIN-885 Add option to send email in scheduler execution #918

ZEPPELIN-885 Add option to send email in scheduler execution #918

vgmartinez commented May 25, 2016 •

edited

Loading

cloverhearts commented May 26, 2016

astroshim commented May 26, 2016 •

edited

Loading

felixcheung commented May 26, 2016

cloverhearts commented May 26, 2016

bzz commented May 26, 2016 •

edited

Loading

vgmartinez commented May 26, 2016

Leemoonsoo commented May 26, 2016

vgmartinez commented May 29, 2016

bzz commented May 31, 2016

Leemoonsoo commented May 31, 2016

vgmartinez commented May 31, 2016

Leemoonsoo May 31, 2016

Leemoonsoo commented May 31, 2016

vgmartinez commented Jun 6, 2016

Leemoonsoo commented Jun 7, 2016

vgmartinez commented Jun 13, 2016 •

edited

Loading

Leemoonsoo commented Jun 14, 2016

vgmartinez commented Nov 28, 2016

ZEPPELIN-885 Add option to send email in scheduler execution #918

ZEPPELIN-885 Add option to send email in scheduler execution #918

Conversation

vgmartinez commented May 25, 2016 • edited Loading

What is this PR for?

What type of PR is it?

Todos

What is the Jira issue?

How should this be tested?

Screenshots (if appropriate)

Questions:

cloverhearts commented May 26, 2016

astroshim commented May 26, 2016 • edited Loading

felixcheung commented May 26, 2016

cloverhearts commented May 26, 2016

bzz commented May 26, 2016 • edited Loading

vgmartinez commented May 26, 2016

Leemoonsoo commented May 26, 2016

vgmartinez commented May 29, 2016

bzz commented May 31, 2016

Leemoonsoo commented May 31, 2016

vgmartinez commented May 31, 2016

Leemoonsoo May 31, 2016

Choose a reason for hiding this comment

Leemoonsoo commented May 31, 2016

vgmartinez commented Jun 6, 2016

Leemoonsoo commented Jun 7, 2016

vgmartinez commented Jun 13, 2016 • edited Loading

Leemoonsoo commented Jun 14, 2016

vgmartinez commented Nov 28, 2016

vgmartinez commented May 25, 2016 •

edited

Loading

astroshim commented May 26, 2016 •

edited

Loading

bzz commented May 26, 2016 •

edited

Loading

vgmartinez commented Jun 13, 2016 •

edited

Loading