v2: decide on import path and make a new release

[kolyshkin]

The latest release is from September 2018, maybe it's time to make a new one?

The primary reasons for a new release are #509 and #515; this will make the package not depend on any other packages. Including #586 would be nice, too.

[dmitshur]

This should be done.

In order to be able to publish new v2 versions, a blocking task is to resolve the current lack of clarity about what import path should be suggested and continued to be used for v2: github.com/russross/blackfriday/v2 or gopkg.in/russross/blackfriday.v2. The go.mod file says one thing, the README on master branch says something that isn't very consistent.

By making a newer release, some decision would be committed to, and ideally that decision should be a good one given the constraints this project is currently operating with. Some of the decisions made in the past predate a good understanding of modules and best practices regarding v2 versions, which has made this decision more difficult to make conclusively.

I suspect that in the current situation, the best course of action is to continue to use the github.com/russross/blackfriday/v2 import path, as defined by the module statement in go.mod file in v2 branch:

blackfriday/go.mod

Line 1 in 3b2b20c

module github.com/russross/blackfriday/v2

I started talking to @rtfb about resolving this situation a number of months ago, but then unfortunately dropped the ball on it due to lack of bandwidth. But I wanted to post this comment to shed light on what I believe is a large contributing factor to no new v2 releases.

[russross]

I'm happy with either import path and agree that we should post a new release soon. I haven't been following developments in the module system very closely--is there one approach that is more forward looking/will fit in better with the community than the other?

[SamWhited]

There's also a third option which I always personally prefer: using a new vanity import path (eg. blackfriday.example.net/v2).

This requires that someone own a domain name and setup an HTTP server (or use a service like Netlify or GitHub Pages) which is more work for the maintainers, however, it gives you the benefit of owning your package import instead of relying on GitHub or gopkg.in never changing or going under or being changed to github.microsoft.com or similar. While this may not seem likely, other scenarios could be imagined where GitHub becomes impractical in the future, for example a price change, adding more extreme rate limiting that causes fetching packages to break more often, the author just decides they don't like GitHub and want to use , etc.
Using a vanity import keeps things flexible in the future, and avoids unforeseen future issues at the cost of a tiny bit of infrastructure work. I would be willing to help set this up if you want.

[kolyshkin]

This requires that someone own a domain name and setup an HTTP server

... and in a couple years it will be down and we won't be able to reach a person who hosted it. Speaking from experience :-\

[SamWhited]

I would think relying on GitHub would be the bigger risk here. They could go under, or Microsoft could change their pricing and make it harder for open source projects to host on it, or some overzealous company could file a patent claim against something as trivial as tree traversal and Microsoft could take down the repo out of an abundance of caution, etc.

The package proxy smooths out this sort of thing either way though.

[russross]

I'd rather stick with github. It's pretty standard and predictable without requiring any maintenance. The Go project is hosted there as are most libraries in widespread use, so in the unlikely event that Microsoft pulls the rug out from under everyone, updating blackfriday will just be a small part of a much larger disruption.

[laher]

Assuming that we don't need to debate the module path any further ... I believe all this really needs is a README update to replace the 'gopkg.in' recommendation, and a tag on the v2 branch - v2.0.2/v2.1.0?

Or, are you considering a merge into master first?

If it's just the tag & readme, I can make a PR for the readme update.

FWIW I agree that github seems the pragmatic choice - I don't believe blackfriday has any need for gopkg.in any more now that modules have versioned paths - and also hasn't github already been chosen in go.mod, in both the v2 branch and the v2.0.1 tag?

[mwat56]

@laher

Assuming that we don't need to debate the module path any further ... I believe all this really needs is a README update to replace the 'gopkg.in' recommendation, and a tag on the v2 branch - v2.0.2/v2.1.0?

… and copy the recent files to a ./v2/ subdirectory while leaving the v1.x.x files were they are. (see Go Modules: v2 and Beyond )

[laher]

@mwat56 it's worth raising that point, yup, especially this part:

if v2+ source code is on the repository's default branch (usually master), tools that are not version-aware — including the go command in GOPATH mode — may not distinguish between major versions.

I guess the question is, would it be OK to just break non-module-aware users that aren't pinning versions somehow? And, I guess the answer for such an established project like blackfriday, might be 'nope, not for some time to come'? Or not? pained thinking emoji

[mwat56]

@laher

[...]
I guess the question is, would it be OK to just break non-module-aware users that aren't pinning versions somehow?

Why break anything at all? Just leave the current files as v1.x.y and copy the newest version of each file into a new subdirectory ./v2/ and assign there a version tag v2.x.y

That way the non-version aware projects (are there still some of those after discussing versioning for years?) are using the v1.x.y and the version-aware projects can use the v2 branch if they want.

Sure, it will probably take an hour to do the copying and check-in into a new v2 branch. But in the end all those require/replace hacks won't be necessary anymore.

[depado]

Hey, any news on this?
I think it can be as straightforward as @mwat56 suggested and it would finally fix all those replace tricks we've been doing to point to gopkg.in. This is a real pain point right now.

[khuddlefish]

I'd like to +1 this. Just tried to go get this using the path in the README and had all sorts of issues. go get github.com/russross/blackfriday/v2 worked for me. Can we change the recommendation to use that?

[danneu]

go get github.com/russross/blackfriday/v2 worked for me. Can we change the recommendation to use that?

Yeah, current readme install instruction leads to this output:

$ go get "gopkg.in/russross/blackfriday.v2"
go: gopkg.in/russross/blackfriday.v2 upgrade => v2.0.1
go get: gopkg.in/russross/[email protected]: parsing go.mod:
	module declares its path as: github.com/russross/blackfriday/v2
	        but was required as: gopkg.in/russross/blackfriday.v2

[russross]

I haven't followed all of the module changes and import path stuff. If someone who is up-to-date on it will submit a PR I'll merge it and we can get this one resolved.

[dmitshur]

Picking up from #587 (comment).

Current State

To recap, the current (unfortunate) state regarding blackfriday v2 and its import path is:

• The README (on v2 branch) says conflicting things like import "gopkg.in/russross/blackfriday.v2" and "With package management you should import github.com/russross/blackfriday and specify that you're using version 2.0.0."
• Version v2.0.0 is available only under the gopkg.in/russross/blackfriday.v2 import path and doesn't work under the github.com/russross/blackfriday/v2 import path.
• Version v2.0.1 is available only under the github.com/russross/blackfriday/v2 import path and doesn't work under gopkg.in one.
• Latest commit on v2 branch (pseudo-version v2.0.2-0.20200714164129-acedacffef10 as of this writing) is available only under the github.com/russross/blackfriday/v2 import path and doesn't work under the gopkg.in one.

Unfortunately, some projects use the github.com/russross/blackfriday/v2 import path, while some other projects use the gopkg.in/russross/blackfriday.v2 one, so whatever decision we make involves causing some projects the need to change import path they use to be able to receive newer versions of blackfriday v2.

Some Data

To get some (imperfect) information about which of the two v2 import paths are being used by public projects in active development, I wrote a small program (source) that walks over the module index, considers only the latest version of each unique module, and checks whether it requires github.com/russross/blackfriday/v2 or gopkg.in/russross/blackfriday.v2. When considering the entire 2020 year so far:

total modules: 1111930
unique modules: 192022
modules that require github.com/russross/blackfriday (v1): 469
modules that require github.com/russross/blackfriday/v2: 202
modules that require gopkg.in/russross/blackfriday.v1 modules: 3
modules that require gopkg.in/russross/blackfriday.v2 modules: 81

When looking only at modules that have had new versions published within the last month:

total modules: 104163
unique modules: 34127
modules that require github.com/russross/blackfriday (v1): 117
modules that require github.com/russross/blackfriday/v2: 68
modules that require gopkg.in/russross/blackfriday.v1: 1
modules that require gopkg.in/russross/blackfriday.v2: 18

(Full list here.)

So it seems staying with github.com/russross/blackfriday/v2 would require fewer active projects to change their import path.

Trade-Offs Considered

The reason it has been so hard to make a decision and all the delays here is because neither solution is ideal—each one has some upsides and downsides—and the trade-offs also differ depending on whether you consider the short term, medium, or long-term outcomes.

• github.com/russross/blackfriday/v2 may be better in short term and probably better long term.

  This import path can only be used in module mode. Legacy GOPATH mode (which is on its way out, see cmd/go: default to GO111MODULE=on golang/go#41330) won't be supported (unless we merge the v2 branch into a v2 directory on master branch). It seems to be the more popular import path in newer projects that use blackfriday v2, and it's a more natural extension of the original github.com/russross/blackfriday import path of blackfriday v1.

• gopkg.in/russross/blackfriday.v2 may be better in the short and medium term, but maybe not long term.

  This can be used both in module mode and GOPATH mode. It has more known public importers in total according to imported by count at pkg.go.dev, 169 vs 140. But fewer new projects use it. It's a bigger change to go "back" to it after v2.0.1 (latest current release version) went with github.com/russross/blackfriday/v2, and it'll stop working if the external gopkg.in service ever goes away.

From my earlier discussion with @rtfb, the discussion in the comments above, and the data from walking the module index in 2020, it seems better to choose github.com/russross/blackfriday/v2 as the supported import path for blackfriday v2 going forward. (The existing released version at the gopkg.in import path may still work, but it won't be supported or endorsed.)

If there are many reports of github.com/russross/blackfriday/v2 not being supported in GOPATH mode causing trouble, there will still be a way for us to address it: by switching from major branch model (v2 version is on v2 branch) to major directory model (v2 version is in a v2 directory on master branch). This can be done later.

Next Steps

To that end, I suggest we take this sequence of steps to resolve this issue:

• Update README to be clear about the canonical import path for blackfriday v2.
  • I've sent two PRs to update the README on master (README: make a definitive decision on v2 import path #674) and v2 (v2/README: make a definitive decision on v2 import path #675) branches.
• Tag the latest commit on v2 branch as v2.1.0-pre.1 to make a pre-release version that people can try out.
• If no serious issues are reported with v2.1.0-pre.1 after a week or more, tag that same commit as version v2.1.0.

[depado]

Thanks a lot for that summary.

As a side note, the divergence between the two import styles also forces some decisions on developers. As an example, if a transitive dependency uses the github.com/russruss/blackfriday/v2 import path, it's impossible to use the gopkg.in one. It's the case if you're using spf13/cobra, which depends on cpuguy83/go-md2man/v2 that imports blackfriday using the github style import. And the error when trying to compile isn't clear at all, you need to take a deep look at your dependency graph to understand what's going on.

I also agree that using the github.com version is better suited for long term, it's also more intuitive for newcomers.

[dmitshur]

@russross @rtfb Any concerns if I proceed with the plan suggested in "Next Steps" of the comment above soon?

[russross]

Sounds good to me. - Russ

…

On Thu, Oct 22, 2020 at 10:07 PM Dmitri Shuralyov ***@***.***> wrote: @russross <https://github.com/russross> @rtfb <https://github.com/rtfb> Any objections if I proceed with the plan described above <#587 (comment)> soon? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#587 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAAP7FHB6CNRHZNGVN7JSALSMD6OJANCNFSM4LS5MTIA> .

[dmitshur]

Moving on to step 2, I've published a pre-release version v2.1.0-pre.1 at the tip of v2 branch:

$ go list -m -versions github.com/russross/blackfriday/v2
github.com/russross/blackfriday/v2 v2.0.0 v2.0.1 v2.1.0-pre.1

If you're following this issue and are able to, please test v2.1.0-pre.1 and comment (or report an issue) if you find a serious problem, before we promote the pre-release version to a final v2.1.0 release. You should be able to install it in your project with:

$ go get github.com/russross/blackfriday/[email protected]

Thanks.

[dmitshur]

There haven't been any reports of serious issues with the v2.1.0-pre.1 pre-release version (neither in this thread, nor in other issues), so I'm going to proceed with promoting it to v2.1.0. That will resolve this issue.

[dmitshur]

Version v2.1.0 is now released.

I've realized to resolve this fully, we also need to publish a new minor release for blackfriday v1, since its current latest release (v1.5.2) still has the old README. I'll follow a similar process:

• Tag the latest commit on master branch as v1.6.0-pre.1 to make a pre-release version that people can try out.
• If no serious issues are reported with v1.6.0-pre.1 after a week or more, tag that same commit as version v1.6.0.

Then we can close this issue.

[dmitshur]

There haven't been any reports of serious issues with the v1.6.0-pre.1 pre-release version (neither in this thread, nor in other issues), so I'm going to proceed with promoting it to v1.6.0.

Version v1.6.0 is now released.

I think this specific issue is finally resolved. Phew. 😓

[laher]

Great job @dmitshur - a big effort, well executed.
The result makes it easy for people to use blackfriday v2 in this Go modules world, which most of us live in now.
Many thanks