Email notification upon replies

[TWiStErRob]

Usually commenting solutions allow for users to be notified via email about replies to their comments, or in case of a flat commenting stream when anyone posts a new comment. Is it possible to do this with staticman?

[eduardoboucas]

This is currently in development and will be released soon!

[illus0r]

@eduardoboucas looking forward to it! I've just discovered, I've received a first comment on my blog. This comment is useful, but I'm not able to say “Thank you” to it's author.

[eduardoboucas]

The feature is pretty much ready. I could use with some help with testing. Would any of you be interested in being the guinea pig? 😄

[illus0r]

@eduardoboucas ready!

[eduardoboucas]

Okay, here's how to get notifications working.

Emails are sent using Mailgun. There is an account associated with the public instance of Staticman, but it comes with a limit of 10,000 emails a month. This limit shouldn't be an issue in the immediate future, but users are encouraged to use their own Mailgun accounts instead. They are free (for the first 10k emails), but you do need a custom domain to use them with. I can possibly provide *.staticman.net subdomains.

To enable notifications, you need a notifications block in your site config and you must be using the config file format introduced in v2 (i.e. having a staticman.yml file, instead of using Jekyll's _config.yml). See the sample config file for more info.

If you're using your own Mailgun account, you'll need to add a Mailgun API key and domain to the config, which need to be encrypted. To encrypt those, you can use the following endpoint: https://api.staticman.net/v2/encrypt/{TEXT TO BE ENCRYPTED}.

Next, your entries will need to specify that a user wishes to subscribe to a specific entry. This is an example:

<form method="POST" action="https://api.staticman.net/v2/entry/joebloggs/my-site/master/comments">
 <input type="hidden" name="options[origin]" value="http://mysite.com/post1.html">
 <input type="hidden" name="options[parent]" value="867bd1e0-921c-11e6-930f-79eeedf443ea">
 
 <input type="text" name="fields[name]" placeholder="Title">
 <input type="text" name="fields[email]" placeholder="Email">
 <textarea name="fields[body]"></textarea>
 <input type="checkbox" name="options[subscribe]" value="email">
 
 <button type="submit">Go!</button>
</form>

In the from above, you can see the following:

• options[origin] contains the full URL of the entry. This will be included in the email sent to subscribers, allowing them to open the post directly.
• options[parent] is a unique identifier to the entry the user is subscribing to. This could be the title of a post (although it needs to be URL-friendly, so this needs to be a slug) or the _id field (which is now automatically added to every entry) if you want to subscribe to another comment.
• options[subscribe] contains the name of the field corresponding to the email address of the subscriber.

Finally, and to make sure we don't send emails asking users to click on links that have been spoofed, you need to add an allowedOrigins field (see docs) to your site config, where you'll specify which domains are valid origins. If you add only mysite.com, then an entry where options[origin] is http://myothersite.com/post1.html will be rejected.

I'm sure there are still a few rough edges, so if any of you can try implementing this and share your feedback that'd be very useful.

Thanks!

[dancwilliams]

Works great! Thanks for your work on this!

[zongren]

Is this working on public staticman instance?
I tried it a couple of times and failed.
Here is my staticman.yml:

allowedFields: ["name", "email", "url", "message", "origin", "parent", "subscribe"]
requiredFields: ["name", "message"]
format: "json"
generatedFields:
  date:
    type: "date"
    options:
      format: "iso8601"
moderation: true
branch: "master"
path: _data/{options.slug}
filename: comment-{@timestamp}
transforms:
  email: md5
name: "宗仁的博客"
allowedOrigins: ["localhost", "zongren.me"]
notifications: 
  enabled: true
  apiKey: "qPB1uWby7FS6T0wgLqUfjcQkkVtLkcergXOqiEoZyTo5yqMmGU/cuzOD825KOZkvbE7m0mOYo2LKPj82v+BcQDxxcIULev8lwpQ1KZJwjv6Ei3f1HbFyIq5N2Ehmya3PyPGga3IaedFVTPFrue67DQ2W5+tu8xJX1S2PahUEgAA="
  domain: "FkFQQFNFX96xVIOYvgtB8IxeBP60lZa/sUhHv0Y3KWV3EdRjMLED0zZr6nGGC5opytzczKvQtR2Y6YJvKQ2ltOBV1aFuAvsN/HPnOZ4e5JMBI+BjGWWlaqUsKp/mNGq/q9oWDk3FT8tdfw7UBqa8lC99eYa+QMXj/k+gpPx5ki0="

and comment form:

<form class="comment-form" method="POST" action="https://api.staticman.net/v2/entry/<%- theme.staticman %>">
    <input type="hidden" name="options[origin]" value="<%- config.url %><%- config.root %><%- item.path %>">
    <input type="hidden" name="options[parent]" value="<%- item.slug %>">
    <input type="hidden" name="options[redirect]" value="<%- config.url %><%- config.root %><%- item.path %>">
    <input type="hidden" name="options[slug]" value="<%- item.slug %>">
    <span class="comment-form__input-wrapper">
      <input autocomplete="off" spellcheck="false" class="comment-form__input" name="fields[name]" type="text" placeholder="请输入姓名"/>
    </span>
    <span class="comment-form__input-wrapper">
      <input autocomplete="off" spellcheck="false" class="comment-form__input" name="fields[email]" type="email" placeholder="请输入邮箱（可选）"/>
    </span>
    <span class="comment-form__input-wrapper">
      <input autocomplete="off" spellcheck="false" class="comment-form__input" name="fields[url]" type="url" placeholder="请输入你的网站（可选）"/>
    </span>
    <span class="comment-form__input-wrapper">
    <textarea class="comment-form__input comment-form__input--textarea" name="fields[message]" placeholder="请输入评论内容（支持Markdown）"></textarea>
    </span>
    <input type="checkbox" name="options[subscribe]" value="email" id="subscribe" /><label for="subscribe">订阅此文章的评论</label>
    <button class="comment-form__input comment-form__input--button">提交评论</button>
  </form>

And there are no logs on my mailgun dashboard.

[eduardoboucas]

@zongren link to the repo, please?

[zongren]

@eduardoboucas the repository is at gitlab

[zongren]

And I used encrypted "zongren.me" as notifications.domain.

[wes-brooks]

Hi Eduardo!
Any chance you could check the logs of the staticman service to help me understand why my attempts to get reply notifications aren't working? I think I've configured staticman properly and have a working mailgun account, but I must be doing something wrong.

The repo is wrbrooks.github.io. Thanks in advance!
-W

[eduardoboucas]

@wrbrooks I don't see any errors in the logs. Can you link to the page you're using to submit entries?

[justinrummel]

Hello @eduardoboucas I'm having issues as well w/ Staticman triggering emails with Mailgun. I'm able to validate with Mailgun's example curl statement but cannot trigger anything from my primary test page: https://www.justinrummel.com/macworld-2010-pictures/

(API Key removed)

justinrummel@Rummel-MBPr ~> curl -s --user 'api:key-12341234123412341234123412341234' \
                                https://api.mailgun.net/v3/mg.justinrummel.com/messages \
                                -F from='Excited User <[email protected]>' \
                                -F [email protected] \
                                -F subject='Hello' \
                                -F text='Testing some Mailgun awesomness!'
{
  "id": "<[email protected]>",
  "message": "Queued. Thank you."
justinrummel@Rummel-MBPr ~>

[eduardoboucas]

@justinrummel Staticman uses the Mailing Lists functionality from Mailgun, where each thread/post is created as a mailing list. If you go to your Mailgun account and navigate to Mailing Lists, do you see any lists created? If so, do you see the email addresses corresponding to the users that have subscribed to that thread?

[justinrummel]

I see one mailing list that I think was created from my API tests, but not from the site.

[chuckmasterson]

I've been planning how to integrate email notifications onto my blog. I have my comment fields defined somewhat idiosyncratically, so the author's email is authoremail instead of just email. I suppose that probably wouldn't work, since I haven't found where to set the name of the field that Staticman looks for... do you know of a way to make it work, or will I just need to sed all my authoremails into emails?

Also, I saw in your reply to #72 (which brilliantly solved a problem I was having, thanks) that the notifications feature isn't considered fully stable yet. Do you have a sense of how likely it is that an existing staticman notification configuration would continue working as the project continues toward stability?

[eduardoboucas]

@chuckmasterson You can specify the name of the field that contains the email address. It's the value of options[subscribe]. In your case, that would be:

<input type="hidden" name="options[subscribe]" value="authoremail">

Regarding the stability of the email notification system, I have to admit I've been struggling to find enough time to dedicate to it. In any case, I think the existing configuration format is flexible enough to accommodate any changes in the near future, so I don't foresee any breaking changes.

[chuckmasterson]

Completely understand about having time. If I knew the first thing about Node.js I'd try contributing, but alas.

I tried setting up with the advice you gave and got as far as getting some mailing lists created. But nothing triggered sending to these lists. I'll look at it again soon... if you find the time to read this and reply, a couple questions that might help me out are:

• How does Staticman create the "Alias Address" in Mailgun for each mailing list? Mine looks like a random series of hex digits, but it seems like it should correspond to the _id of some post, or at least something that I see in the YAML of one of the comments.
• How does Staticman determine what (if any) Alias Address to send notification emails to? I'm not quite wrapping my head around how this is achieved with just the options[parent] field. Does it look at all the files in the post's comment directory and see if any have an _id that matches options[parent]? If so, how does it work if you give it a slug as option[parent], since you aren't telling it whether it's identifying the page or the parent's _id? And how does it make the leap from there to knowing where to send the email?

[chuckmasterson]

After writing that, I took a crack at reading the code anyhow, and I think I may have understood enough to make a suggestion that would get us partway to a solution, though not all the way. (I would write this as a pull request but my understanding of Node.js is so weak that I would almost certainly break more than I'd fix.)

Suggestion:

Use a different variable besides options.parent to control subscriptions, perhaps a new one named options.subscribeTo.

Reasoning:

Though I haven't found anywhere you gave explicit directions on how to use options.parent, @mmistakes is already using it in threaded commenting, and a few other people have copied his strategy (I'm one). This strategy relies on options.parent being written into the YAML comment file as _parent. In this strategy _parent (options.parent) is the _id of a child comment's parent.

From what I understand of the Staticman code, Staticman expects options.parent to be some identifier of a post that the user is currently subscribing to. This conflicts with the other usage. If a user is writing a top-level comment on a post, then that comment's _parent will be null, and hence even if the user clicks the "subscribe me" checkbox, no mailing list is created because this condition doesn't get satisfied:

if (subscriptions && options.parent && options.subscribe && this.fields[options.subscribe]) {
  subscriptions.set(options.parent, this.fields[options.subscribe]).catch(err => {
    console.log(err.stack || err)
  })
}

If I follow the logic aright, the only subscriptions that would work with Staticman as it stands are where a user makes a child comment and clicks "subscribe me" - in which case they would get subscribed to all replies to their comment's parent. No one can subscribe to replies to their own comment, only to replies to some top-level comment that already exists.

Although my and mmistakes' use of options.parent here is apparently nonstandard, I think there's reason to favor it over what's currently in the code:

• It's already employed in several sites.
• The name parent describes a child comment's parent more intuitively than it describes "what comment the user is currently subscribing to".
• I don't think you'd be breaking any currently functioning sites. It seems like all the Jekyll pioneers here who have tried implementing Mailgun notifications have gotten stuck at about the same stage I have, and the only one I'm aware of who apparently has it working is mmistakes. (I've left a comment on his site to try to see if notification works in the strange way I've predicted, but it's in moderation right now.) The sites of @justinrummel and @zongren currently have no "subscribe" options and @wrbrooks 's does but I can't test it because no comments are going through at all for me (hey wrbrooks, did you know that?).

I'm not sure what format would make sense for the hypothetical options.subscribeTo as I still don't fully understand how Staticman currently interprets options.parent.

I'm still confused on one point: you mentioned in #76 that options fields aren't written to the comment file, but options.parent sure is getting written. I'm happy with that, but it is also a bit unusual; maybe it would make sense for it to go into the documentation eventually that "options fields generally don't get written to the entry, with the exception of options.parent, which is prefixed with an underscore (_parent) to denote the difference."

I'm happy to be a guinea pig on this feature as I'd definitely like to have it, but I'm also going to be on extended travels with little to no coding starting April 1, so I hope we can all get it figured out by then.

And thanks for Staticman! I hope this helped and wasn't too full of misunderstandings of how the code works.

[mmistakes]

@chuckmasterson I just merged in your comment. My understanding is you should get a notification email once a new comment is merged in. I'm going to do that now and make it off the main thread (not a child of your comment to see if your hypothesis is true).

My site takes about 20 minutes to build as it crunches through a bazillion images, so give it that long to see if anything shows up.

[eduardoboucas]

Thanks for the suggestions, @chuckmasterson. Let me try to answer some of the questions.

How does Staticman create the "Alias Address" in Mailgun for each mailing list? Mine looks like a random series of hex digits, but it seems like it should correspond to the _id of some post, or at least something that I see in the YAML of one of the comments.

Each mailing list has a corresponding email address. This email address is a MD5 hash of the GitHub username, repository and entry id concatenated together. This happens here.

How does Staticman determine what (if any) Alias Address to send notification emails to?

Whenever we process an entry, we take its id, generate the hash described above and check if there's any mailing list with a matching address. If there is, we fire notifications to all the addresses in it. Otherwise, there's nothing to do.

From what I understand of the Staticman code, Staticman expects options.parent to be some identifier of a post that the user is currently subscribing to. This conflicts with the other usage.

My thinking was that options[parent] describes the "thread" that you want to subscribe to. If you're writing a top-level comment as you mentioned, this parent would be an identifier of the post you're commenting on (it could be a title, slug, or anything else that uniquely identifies the entry). If you're commenting on an existing comment, then the parent would be the id of the comment.

Theoretically, it could go on how many levels necessary, but as @mmistakes will be able to tell you, it's probably not a good idea to nest further than 2 levels because otherwise the site generation process becomes too complex.

So the general idea is: options[parent] can be whatever — a post title, slug or a Staticman entry id. All it does is create a mailing list with subscribers for that particular parent, which we use whenever we want to process that entry.

Does this help?

[eduardoboucas]

And sorry if this has caused confusion and made you have to guess what Staticman does. My intention was to document this whenever I had the chance to make it fully stable, but I understand that people want to use this as soon as it's available, so from now on I'll make sure to document things as they get deployed to the live API.

[eduardoboucas]

Also:

I'm still confused on one point: you mentioned in #76 that options fields aren't written to the comment file, but options.parent sure is getting written. I'm happy with that, but it is also a bit unusual; maybe it would make sense for it to go into the documentation eventually that "options fields generally don't get written to the entry, with the exception of options.parent, which is prefixed with an underscore (_parent) to denote the difference."

This is very true. This should be in the documentation.

[wes-brooks]

After some further investigation, I figured out how to get notifications to send when moderation is enabled: you have to add a webhook to your repository, pointed at

https://api.staticman.net/v1/webhook

Note that this is true even if you're using the v2 API for comments.

This fix is kind of obvious in retrospect (notifications only go out when the comment is accepted, but how was staticman going to know that the comment had been accepted?), but the documentation only mentions the webhook API in reference to automagically deleting the staticman_ branches after they are merged.

I think there should be a clarification in the documentation section about reply notifications to say that when moderation is enabled, sending notifications requires that you add the staticman webhook to your website's repository.

[nakoo]

@wrbrooks Perfect. I finally made it. Thanks for goodness.

[binarymist]

The main problem I'm (and my commenters) seeing is the multiple Staticman notifications for every comment: #182

[chmac]

@eduardoboucas Haven't read this issue in detail yet, but I could put some time into this if you like. I've just started thinking that I really want reply notifications and saw this and #35. Will wait for the go ahead, and if you're happy, put some effort into this. I'm offline next week, and only online a few days the following week, so might be 3-6 weeks before I get to it...

[samarulmeu]

I finally managed to get notification working with mailgun (in fact only adding the emails to a mailing list). Most of the trouble and stress was because I was trying to use a domain in the EU region. There is a difference in the API base URL: https://api.eu.mailgun.net/v3/ vs https://api.mailgun.net/v3/.
Is there a way to specify the EU region?

[gabeluci]

I managed to setup notifications per-thread (i.e. people will only be notified if someone replied to a thread they started or replied to, rather than any thread on the post), but only because I'm running my own instance of Staticman and I can change the code.

The key was setting the parent property to the auto-generated id if the parent is empty. I did this in the processEntry function of /lib/Staticman.js:

Staticman.prototype.processEntry = function (fields, options) {
  if (!options.parent) {
    options.parent = this.uid
  }
  ...

That may not be the best place to do it, but it works. Maybe that's something that can be worked into the master code, even as a config option. I'm sure others could benefit.

Then in my site code, I leave options[parent] blank for a top-level comment, and I set it to the _id of the parent if it's a reply. I completely got rid of the replying_to property and instead just check if _parent == _id to determine if it's a top-level comment or a reply.

Feel free to look at my code. It is based on the good work of @mmistakes.

The only thing outstanding that I might fix is the wording of the initial email to the original commenter. The email is valuable when moderation is on since it basically tells the person that their comment was approved. But I'd rather it actually say that than say "Someone replied to a comment you subscribed to".

But that's for another day.

[binarymist]

But no PR submitted @gabeluci?

[gabeluci]

No, no PR. I don't know the code base well enough to say that's the best place in the code to do it, or if it will break how others are using Staticman. @eduardoboucas may decide it's better to make it a configurable option, if anything.

[binarymist]

Understood @gabeluci. Looking at the commits vs issues and PR's, it's starting to look like staticman is no longer being maintained. Is this correct @eduardoboucas? If so, have you considered passing the battern?

[purpleidea]

Looking at the commits vs issues and PR's, it's starting to look like staticman is no longer being maintained.

I'm sure @eduardoboucas is busy with other things as well, but send a patch, do a fork, or even host an alternative mail server if you'd like! At least from my POV it's pretty feature complete. The biggest trouble I have is integration with gohugo.

[binarymist]

Oh odd. What exactly are you struggling with there @purpleidea, I have it integrated (https://binarymist.io/blog/2018/02/24/hugo-with-staticman-commenting-and-subscriptions/)

[purpleidea]

@binarymist

I gave up after getting annoyed with the available functions in hugo, and not finding clear docs on how to write one. Code is here:

https://github.com/purpleidea/purpleidea.com

Patches welcome! :)

[eduardoboucas]

Looking at the commits vs issues and PR's, it's starting to look like staticman is no longer being maintained. Is this correct @eduardoboucas? If so, have you considered passing the battern?

The time I've been able to dedicate to the project hasn't been enough, sure. But what exactly do you mean by passing the batten? The repository is open, anyone can contribute with code. I'm more than happy to give write access to people that wish to become regular maintainers (as I've done in the past).

Most issues have to do with new feature requests, most of which I don't want to commit to precisely because I don't have enough time to implement and maintain them. Most PRs are from people wanting to add their site to the README (which, looking back, wasn't the best of ideas).

Maybe what I need to do is go through the list of issues/PRs, close what needs to be closed, merge what needs to be merged, and then start a GitHub Project to give some visibility on what needs to be addressed and in which priority. That should give users (and contributors) an indication on what is being worked on and what we need help with.

Thanks for the patience.

[gabeluci]

I ended up writing a whole article about how I setup my site to use Staticman. More relevant to this thread is the section about comment threading.

[binarymist]

What I mean @eduardoboucas is that it seems that @gabeluci has an answer for this issue in code, and it'd be great to see some colaboration between you two so that the work could be included in Staticman. Respect to both parties involved.

[binarymist]

@purpleidea That's why I provided the available functions and clear docs for you, if they're less than clear please leave a comment on the blog post and I'll do my best to help.

[gabeluci]

I don't think my way is the only way to do it. I think you could do per-thread notifications on the public API as-is. You would have to:

1. For top-level comments, generate a unique identifier in JavaScript (this might help) and use that for the value of parent. That value would be used for replies too. That solves the issue of subscribing the top-level commenter.
2. Add a second field called something like isTopLevel that is true for top-level comments and false for replies. This would be used solely for displaying the comments, since the first loop only looks at top-level comments.

[chmac]

I'm also a committer and have deploy privileges. I'd be open to putting some time into getting this live. Is anyone ready to submit a PR? I'm ready to review it, loop in @eduardoboucas, and if all's well, deploy.

@samarulmeu It would probably be easier to open a new issue about the switch to the EU region. There's a few things getting mixed together here.

[gabeluci]

@chmac Are you talking about setting parent to the generated id? I could attempt a PR for that, but it depends on if it should be a configurable option. Doesn't hurt to make it an option I guess.

I did submit an unrelated PR for something else related to notifications.

[chmac]

@gabeluci Aha, I see your PR. To be honest, I'm not 100% on top of the status of this issue. I'll try to schedule some time to dig into it in the next few weeks. I also now see there's quite a few open PRs which could be reviewed...

[gabeluci]

Well my PR doesn't really have anything to do with this issue, except being related to notifications.