[Proposal]: Marked roadmap update and details

[joshbruce]

Gonna use this initial comment to be more in conversation facilitation and spark mode before adding my personal comments and thoughts.

#1073 brought up some really interesting points and user feedback, which should prove beneficial to us moving forward. We are roughly the third crew to take on marked. Marked also has a large user base who has become accustomed to marked not changing (being "stable" in that sense), despite still being a zero major release. Some of the areas of new conversation have revolved around the scope of marked and future vision.

1. What's in a version for us? Consider some of the following.

• Firefox 1.0 was released in 2002. Current version 58.0.2.¹
• Safari 1.0 was released in 2003. Current version 11.0.3.²
• Chrome 1.0 was released in 2008. Current version 64.0.3x.³
• macOS was released as version 10 (OS X) in 2001. Since then it has not had a major release; however, has been rebranded from Mac OS X to macOS and looks and feels completely different than the one released in 2001.⁴
• iOS was released in 2007. Current version 11.2.6.⁵

2. What are the user and product considerations of various versioning and release strategies?

• Angular has shifted to a time-based semantic versioning construct. So, version numbers are incremented based on time, not necessarily the content of the change.⁶
• Straight semantic versioning, on the other hand, is primarily about communicating intent through a number. And seems to be the standard approach. ⁷

3. What is the product purpose of marked? what problem(s) does marked solve? what problem(s) does markdown solve? and how do they work together?

Footnotes

1. https://en.wikipedia.org/wiki/Firefox ↩

2. https://en.wikipedia.org/wiki/Safari_(web_browser) ↩

3. https://en.wikipedia.org/wiki/Google_Chrome ↩

4. https://en.wikipedia.org/wiki/MacOS ↩

5. https://en.wikipedia.org/wiki/IOS ↩

6. https://github.com/angular/angular/blob/master/docs/RELEASE_SCHEDULE.md#more-info--resources ↩

7. https://semver.org ↩

[Feder1co5oave]

I think each one chooses the versioning scheme that best suits his/her needs.
We're on npm so we must respect semver, but I honestly don't care about all its intricacies and details, as long as our versioning scheme works for us and for our users!
In the linked issue I proposed a change to our current scheme: to bump a minor version for every significant change we make (that changes significantly the output, that is), and aim to the 1.0 as the "no known issues" (API backward compatible) release, instead of releasing patches until 0.4. Knowing that after the 1.0 development will focus primarily on patches for it (so 1.0.1, 1.0.2...), and working on extendability, modularization, developer experience and API for the future 2.0. In other words we treat our 0.x as if it were a 1.x.

When we release 1.0 I think we can mark 0.3.x as deprecated and declare 1.0 to be API compatible with it so people will feel safe to upgrade even though it is a major? (same thing goes if we stick to the 0.4.0 approach we had since now, I believe)

To be precise, I think the more conservative approach here would be to bump the minor instead of the patch, since we declare an incompatibility, as slight as it might be. On the other side, bumping patches while releasing breaking changes (see #1073 and #1059, even though this last one relied on some marked internals) might be considered incorrect by many.

[UziTech]

+1 on the "no known issues" being 1.0

from the semver.org FAQ

If your software is being used in production, it should probably already be 1.0.0. If you have a stable API on which users have come to depend, you should be 1.0.0. If you’re worrying a lot about backwards compatibility, you should probably already be 1.0.0.

I'm pretty sure all of those apply to us.

[Feder1co5oave]

Uhm good point! I think 1.0 is overrated, somehow 😄

[joshbruce]

tl;dr

So, maybe another question is: What event is more important to spark a version change and when?

I tend to lean toward semver with purpose and goals (milestones) as well. With a zero major, users should be prepared for drastic changes; however, given marked's longevity in the marketplace, I definitely see where the expectation has become that it is stable.

I should also note that time is irrelevant to me on this score and I think tying time to the versioning of software is a horrible idea. :)

If you look at the history of 8fold Component, 1.0 was released in October of last year...3.0 was released in November. The point here being that the zero major is what we make of it, and I think there are three things, given our context, that we should still use the zero major to accomplish.

1. No known issues.
2. Product scope and purpose.
3. Community, process, and tooling.

---

For me there are two battles I never appreciated in software development:

1. Version wars: It's on release 50; so, it must be 50 times more stable (or better) than version 1. This is what it started feeling like after we managed to kill IE6 when it came to browsers. Safari is only on version 11 while Chrome is on version 64...despite Safari being around five years longer...feels like marketing to me and I have a very low tolerance for marketing for its own sake. :)
2. Feature wars: Can it do this? Well this other thing can; so, this other thing is better. iPhone versus Android...I can modify the source code for Android, can you do that with iOS...wrong question, in my opinion, more appropriate question imho is should I be able to given the users this product is geared toward?

With all that said. I can see where our users might be a bit confused. We've been on a zero major release for 6 or 7 years. Having said that, for half that time, development and releases pretty much stagnated and marked almost died really. Then we came along. :)

Here's my thoughts on the roadmap given the conversations I'm seeing and the tickets that have been floating around.

1. The zero major should remain as long as a) we still have known issues against CommonMark and GFM and b) the capabilities (feature set) and distilled purpose of marked is still in major flux.
2. The zero major should focus, at least, on finishing what @chjj started and others continued before we get too crazy. (This may include the architecture changes currently slated as the focus of 0.5.0.)
3. Assuming number one and two are true, thinking the following makes sense based on feedback we've had on this subject so far (noting that all of these goals are always happening, just maybe not the focus):

• 0.3.0: All about no known issues compared to CommonMark and GitHub Flavored Markdown (without just cause). Security, bad regex, and so on.
• 0.4.0: Solidifying marked's purpose and removing features that don't fit that purpose. Giving people more stuff is always easier than taking something away and marked has definitely gone beyond what I think is desired for sustainable development. (It happens, people take on a project, want to please all the users ever and compete against the feature-sets of others.)
• 0.5.0: Solidifying marked contribution processes and tooling. A good, simple flow for creating releases, CI, so on and so forth. If we all disappeared tomorrow, could someone come in and take the torch while still respecting marked's users and history?
• 1.0: We might call the @Feder1co5oave and @dominikwilkowski release. :) "we should accelerate the bump to 1.0, declaring it the new "no defects" release instead of the 0.4" - if we do the 1.0 bump before deprecations, we introduce the possibility of having a lot of new majors based on semver...we broke the API in a non-backward compatible fashion.
• 2.0: We might call the @Nalinc release. Fundamentally changes the architecture of the library.

[joshbruce]

Dude! People are so afraid of the 1.0 it's creepy. It's almost like they think, "Oh, now I have a 1.0 - the world can't change. I'm committed. I'm married to this project. Oh my god, nooooo!!!!!" Meanwhile, I'm over here like, "Does it work? Yes. Is anything in here an absolutely horrible idea? No. 1.0...signed, sealed, delivered." :)

[UziTech]

My idea of 1.0 is that I am 100% sure I know what everything is supposed to do and >50% sure everything does what it is supposed to do 😆

[joshbruce]

In defense of our profession and the developers I know, a lot of developers I know don't do well with conflict. So, the 1.0 represents the point of no return for them. "I don't want to change the API or update the version of JS or deprecate a feature because that will piss off my users." I suffer from it a little as well...one reason I was fine with doing all those majors to the component project so quickly was because I pretty much knew all the users...we don't have that luxury with marked.

[Feder1co5oave]

Consider this: for semver, 0.4.0 is incompatible with 0.3.0 in the same way 1.0 is too

[joshbruce]

@Feder1co5oave: I may not be following.

[Feder1co5oave]

I'm not sure either. It's 5am and I'm cooked.

[dominikwilkowski]

@Feder1co5oave
From the specs:

MINOR version when you add functionality in a backwards-compatible manner

So I think it says that minor version is compatible with other minor versions given that they are in the same major one.
I could be reading this wrong though :) (3:30PM in Oz)

[Feder1co5oave]

@dominikwilkowski
Uhm yes, but by incompatible I meant in the ^-operator way:

Allows changes that do not modify the left-most non-zero digit in the [major, minor, patch] tuple. In other words, this allows patch and minor updates for versions 1.0.0 and above, patch updates for versions 0.X >=0.1.0, and no updates for versions 0.0.X.
So in 0.x a minor upgrade would mean breaking changes, whereas in 1.x it wouldn't.

edit: If you have marked ^0.3.15 as a dependency in your package.json, npm won't upgrade it to 0.4, or 1.0.

[dominikwilkowski]

Yeah so we need to be careful to not confuse the npm implementation of semver and the actual specs.
I do see it as an important consideration as that's the ecosystem we live in but strictly speaking from semver perspective minor versions are compatible.

But yes you're right that would be true for default npm installs.

As an aside: I was caught in this bug because we test our software on older node versions (6) and in node 6 there is no support for the package.lock file which means my dependency installed as "marked": "^0.3.9", resulted in the install of 0.3.16 which means npm failed to adheare to it's own rules. So let's not trust their implementation but the specs?

[Feder1co5oave]

As an aside: I was caught in this bug because we test our software on older node versions (6) and in node 6 there is no support for the package.lock file which means my dependency installed as "marked": "^0.3.9", resulted in the install of 0.3.16 which means npm failed to adheare to it's own rules. So let's not trust their implementation but the specs?

In my opinion this should be the correct behavior of npm. We released 0.3.15 so we declared it
^-compatible with 0.3.9, and npm upgraded to it automatically.

The package-lock.json mechanism is there to make sure things run smoothly even when semver was not abided (as in this case - or maybe we didn't? I don't even know, anymore), but I don't like it because it doesn't work all the time. You frequently have merge conflicts in it because of git and the solution usually is to delete it and recreate it. And that defeats its purpose.

[dominikwilkowski]

To your point. You could say on 0 major even semver says 0.3 is incompatible with 0.4. You could also say any update can be incompatible as semver is not very clear on that. The only guidance that I could find it has on magic 0 is https://semver.org/#spec-item-4:

Major version zero (0.y.z) is for initial development. Anything may change at any time. The public API should not be considered stable.

It's a good point though, the difference between making updates to 0.x and bumping major seem the same so why not go major right away?

[Feder1co5oave]

the difference between making updates to 0.x and bumping major seem the same so why not go major right away?

This is exactly my point, I just couldn't be this concise.

[joshbruce]

Back to semver for a second...that's why the zero major is kind of the game changer and why I agree that we should push to get a relatively stable-ish 1.0 out there. Then we'll actually be able to communicate properly using semver. The zero major is literally there to say, "We could break all the things! You might not want to use this in production." - unfortunately, our users have...and some will get hit with the negative side of that.

Fair question on not going right away:

Let's say we do 1.0 - like right now.

Deprecate id generation in headers = 2.0 (see #1043)
Deprecate pedantic = 3.0 (see #1043)
Update everything to ES6 and transpile to ES5 = 4.0
Update the architecture to use classes = 5.0

Blah, blah, blah...possibility of a lot of minors and patches in there along the way. For me it's not the question of the mathematics (the spec) of it, it's more of a user experience consideration and communication problem.

0.x = Anything can change at any time without notice.
1.0 = We think the original intent and spirit of marked has been achieved (nothing more, nothing less, and holding on to the single responsibility principle a bit tighter)...at this point, we are decidedly inside semver.
2.0 = This latest crew has established and implemented its current vision based on all the feedback ever.

If we could go back in time (say three years or so), I'm sure we would have tried to convince @chjj to do a 1.0 once the CLI, client-side work, and separation of the lexers, parsers, renderers and so on was in place...at which point, this conversation becomes quite different. :)

[dominikwilkowski]

Deprecate id generation in headers = 2.0 (see #1043)
Deprecate pedantic = 3.0 (see #1043)
Update everything to ES6 and transpile to ES5 = 4.0
Update the architecture to use classes = 5.0

Two things here:

1. I actually don't mind that as it helps you communicate those changes
2. You can always create a major branch and smush some of them if not all together into one

I don't want to push this too hard as I know it will make some contributors nervous but I do think it will make it easier for adopters.

[Feder1co5oave]

Let's say we do 1.0 - like right now.
Deprecate id generation in headers = 2.0 (see #1043)
Deprecate pedantic = 3.0 (see #1043)
Update everything to ES6 and transpile to ES5 = 4.0
Update the architecture to use classes = 5.0

More like:
Solve security issues, comply with specification, build a solid test base = 1.0
Deprecate id generation, pedanic, upgrade to es6, re-engineer = 2.0

---

0.x = Anything can change at any time without notice.

Except nothing changed. For years.

1.0 = We think the original intent and spirit of marked has been achieved (nothing more, nothing less, and holding on to the single responsibility principle a bit tighter)...at this point, we are decidedly inside semver.

More like: here is a library whose API is unchanged from years ago, it is now safe, and complies with modern and much discussed non-ambiguous specifications.

2.0 = This latest crew has established and implemented its current vision based on all the feedback ever.

More like: we rewrote big chunks of the whole thing to make it the best it can be. Now use it with joy (and help us maintain it).

If we could go back in time (say three years or so), I'm sure we would have tried to convince @chjj to do a 1.0 once the CLI, client-side work, and separation of the lexers, parsers, renderers and so on was in place...at which point, this conversation becomes quite different. :)

I can honestly understand why he didn't feel like releasing a 1.0, considering the only thing he based everything was that daringfireball page and markdown.pl.

[joshbruce]

@dominikwilkowski: Not sure I follow this one - I don't want to push this too hard as I know it will make some contributors nervous but I do think it will make it easier for adopters.

1. Unfortunately, we don't know what other users expect here. So far, feedback has been positive and we've only had a couple of questions regarding concept of breaking changes. Not sure we would get the same under a non-zero-major...could be wrong, of course.
2. Agreed on the smushing. Could also make deprecation a minor instead of a major (macOS 10)...goes back to the "what is a release to us" concept. semver does allow for wiggle room and custom definitions (the Angular major seems to be a slight departure from the strictest interpretation).

@Feder1co5oave: Think we're saying almost the same thing here, brother...just different ways, which is good...and we're both kinda fried.

Totally agree on that 0.x being a problem and a saving grace. Totally agree on what 1.0 should mean (do we include features that were added in that weird non-changing time??). Totally agree on the 2.0 description.

[Feder1co5oave]

Sorry, I **am** fried and am reiterating myself a bit

[joshbruce]

Team flow tangent

This discussion got me thinking about flow. (The org team discussion areas have got me thinking about Slack and transparency, but that's getting way too ahead of ourselves...making this one public to go beyond the marked team.) :)

Given the combination of milestones and projects offered by GitHub, it might be a good idea to use milestones as literal mile markers (end of a longer/bigger journey) while using projects to get into the nitty gritty details. (And, of course, none of this would be permanent, I hate process for its own sake almost as much as marketing for its own sake. lol)

Milestone 1.0 release:

• No known issues.
• Product scope and purpose.
• Community, process, and tooling.

Let's say we had a project board for each one of those "big ideas" that are part of the 1.0 milestone...what work would be in each project?

• No known issues project board.
  • The consolidation tickets @Feder1co5oave and I've set up and he's been kicking butt on.
• Product scope and purpose project board.
  • Updates to the README. (Tagging @styfle who suggested breaking it up a bit.)
  • Communication strategies with users.
  • This Issue and other issues that discuss what marked should and shouldn't be (Add linksInNewTab config option #1030, for example).
  • The vision alignment issues.
• Community, process, and tooling project board.
  • Travis CI.
  • This Issue (can't have the same issue in two projects, but this is hypothetical).
  • Adding contributors.
  • Adding owners to NPM.
  • Deprecation strategy.
  • Code of conduct.
  • So on and so forth

Not saying we would need to create tickets and projects for everything - just saying that, if there is an Issue (or we create an Issue) that could be tied to the project or milestone go for it. Further, if a lot of issues can be grouped together into a "theme" - maybe make a project board. Finally, if those project boards can be lumped together into something of an "epic" - maybe make a milestone (right now they're tied to major releases, but may be different in the future - just epics). (Totally stealing and inspired by these ideas from a lot of Agile Software Development practices and this roadmap: https://github.com/uswds/uswds/milestone/52)

The boards can be automated - not totally sure what that looks and feels like yet - so, they would act more as information radiators for us regarding progress not administrative overhead we have to intentionally go and update. Just a matter of updating labels and other dropdown menus, if I'm reading the docs properly.

[joshbruce]

Given that non-contributors can see the boards, could also help improve communication with users who have a more than passing level of interest in the future of marked...despite not being able to actually interact with the boards themselves.

[joshbruce]

Another nice thing about project specifically, we can start with just a note...not an Issue or PR. Transparent communication with users - without committing - then you can convert a note into an Issue. (I'm really trying to find counter arguments to trying this I swear. Just really impressed with the progress GitHub has made in its ability to actually do project management.)

Think I'll start setting things up tomorrow - unless y'all totally think this is a horrible idea - but don't want to do it while I should probably be sleeping. Cheers.

[joshbruce]

Starting to toy around. Let me know if anyone is in violent disagreement with what's in the description here: https://github.com/markedjs/marked/milestone/4

Also, @dominikwilkowski, forgot to mention in case you didn't see this:
https://github.com/markedjs/marked/blob/master/RELEASE.md

[styfle]

I’m a little late to the game so I put a few 👍 on the comments I agree with.

I think the conclusion is that we stick with semver because that is what npm users expect and really, that’s how it’s built.

[joshbruce]

Closing - see #1106 - time to get back to checking email...woohoo!