[RFC] Plots 2.0

[t-bltg]

Too many issues need stuff discussed but postponed to 2.0 because rightfully considered breaking.
Latest example at #4561, caused by a cycling bug identified more than two years ago in #2980.

Should we branch somehow and start working on 2.0 ?

I've closed around ~80 stale issues this we, we need to go through all the remaining opened issues, close the stale ones (or won't fix because of an upstream problem, or dependencies), apply labels to start focusing on important things (like scaling, fonts, backend selection, precompilation, consistency between backends, ...). We won't do this on our own, so please do join this attempt.

Labeled issues:

• [FR] change minorticks semantics #4532
• [FR] auto ylim calculation #4028
• [FR] Safe figure saving #3001
• [BUG] Cycling arguments #2980
• support scalars for hline! and vline! #2129
• Should layout be column major? #2047
• [FR] PythonPlot/PythonCall alternative to PyPlot/PyCall #4181
• [FR] Use Preferences.jl to configure default loaded backend #4513
• [FR] remove explicit GR dependency from Project.toml #4631
• [BUG] Inconsistency With Measurement Units In API #4350
• [FR] redesign backend dependencies #4567
• [FR] Harmonize how series data are used #4596

Labeled PRs:

• cleanup: drop deprecated pgfplots #4326
• GR margin 0-point set to allow no padding #3942
• Reduce number of exported names #3744
• Rework adding legends #3732
• WIP: Fonts #3473
• Remove StatsPlots warnings #3357
• Move @df macro to Plots #3351

[BioTurboNick]

Probably a good idea. Being able to rip out and rework the internal part of the pipeline would help tackle some of the performance issues.

Anything we could do to move to concretely typed data structures internally would likely pay dividends.

And the merged dictionaries setup may need reworking, because Julia doesn't like iterating over it.

I think one of those issues is mine suggesting a layout system that doesn't need handholding to fit everything in the window.

[t-bltg]

Thanks for the comments, don't hesitate to label stuff, and adjust the list here.

[BeastyBlacksmith]

I wanted to write this issue soonish anyway, so thanks for opening.

I think an important part is to define a common set of goals and non-goals we want to accomplish with this breaking release and only break things with a clear motivation.

For starters I am going to list my personal goals:

Towards a get-API and improving attribute management in general

I started this in #3069 #2854 and then in #4489.
Apart from a few functions like Plots.xlims users don't have reliable way to access plot attributes, especially in recipes.
This is a missing abstraction that people circumvent by reaching into internals which makes changing this breaking.
My plan was to add a struct like entry-point which will create attributes, aliases, documentation and getter-functions from one place in the source code.
Thanks to aliases that could have been non-breaking if people would never directly read from the plotattributes Dictionary in recipes ...
I also propose to have a dedicated syntax in RecipesBase for that (like e.g. <--).

Don't export symbols that are not part of the API

I don't think that needs explanation

Change defaults

I was for a long time gatekeeper of defaults considering this a (visually) breaking change. This would be the time to have that never ending discussion about what the best default color palette would be (but in a separate issue, please).

Explicit cycling

While cycling is useful in many cases, especially in combination with layers of recipes it causes some really hard to debug issues (thinking of JuliaPlots/GraphRecipes.jl#163 and others). I want that to be opt-in in Plots 2.0 rather than the default.

Getting rid of global mutable state

There are some low hanging fruit like the defaults and alias dictionaries, which don't have to be dictionaries.
The trickier part is how to keep track of current backend and last plot object without global mutables.

---

That's probably not all, but the first things that came to my head.

[EldarAgalarov]

• Need to properly re-implement plot_title, because it is completely broken: [BUG] plot_title affects subplot indexing because implemented as additional subplot which just emulating plot title #4612
• Attribute aliases should be deprecated/dropped (or leave max 1 alias per attribute): [BUG] Too many attribute aliase names in Plots API. They should be dropped or allow max 1 alias per attribute. #4613
• Measurement units should be unified: [BUG] Inconsistency With Measurement Units In API #4350

[isentropic]

Although I'm a bit less active these days as my grad school and julia work is over, I feel that issue is very important as we have learned a few things and annoyances of the current approach. I wanna just write off a few things that come to mind that should also be somehow considered:

1. Explicit GR dependency. Although it is good that Plots ships GR as the default, I remember mac M1 users complaining that such dependency renders Plots unusable at all. I somehow this tight dependency should be discussed. Or maybe the default backend should be changed to some backend that has no binary deps (like unicode plots? or plotly?). I'm not saying that backend specific code (backends/*.jl) should be stripped off, but rather Project.toml changed. Installation of GR is brittle for non-standard platforms and maybe we should avoid it gracefully. I see no clear best solution, but we should use 2.0 for this.
2. Global mutable defaults. We can leave the current global dict of defaults, but we should make it const. Any user overrides should still be passed on in plot args. For example I can have PLOTS_DEFAULTS = Dict(:palette => :seaborn_deep, :backend=>:pgfplotsx,) in startup.jl. Every call to plot should implicitly do plot(1:10, defaults=PLOTS_DEFAULTS) (somehow). If PLOTS_DEFAULTS variable is present in a user's namespace it should be passed on (somehow). This way we can keep the current dic, but have it completely immutable. Global state isn't bad, it's mutable global state that causes issues.
3. Thread safety. We should strive for thread safe plot if we can. Although this might depend on the backend, thread-safety would nice to have, for people who would like to leverage this. For example, composing animations could be much faster if everyframe is rendered in parallel. This might be too difficult/impractical but it is nice to have.

Ideally, we would need to discuss and assign priorities this endless backlog of things in a maintainers call. @BeastyBlacksmith could you let us know about the next one? Maybe we should ask other great julia figures about what they think is good. There are many great julia people who wish Plots was better, and it would also be beneficial to hear what they wanna say.
P.S.
Huge thanks to @BeastyBlacksmith @t-bltg and other contributors recently, not just for bringing this up but also for all your work. You guys are the best.

[BeastyBlacksmith]

I keep #3538 (comment) up to date. I had to shift this months meeting for a week to 17th of January.

[t-bltg]

Thanks @isentropic.

Regarding 1., see #4566 (comment).
It is now trivial to remove the explicit GR dependency in Project.toml in Plots 2.0 (I've opened #4631 to track this).

Regarding the issues mentioned by @EldarAgalarov:

• [BUG] plot_title affects subplot indexing because implemented as additional subplot which just emulating plot title #4612 can be fixed without breaking, so this does not have to be discussed here;
• it remains unclear whether [BUG] Too many attribute aliase names in Plots API. They should be dropped or allow max 1 alias per attribute. #4613 is an issue or not;
• [BUG] Inconsistency With Measurement Units In API #4350: reworking units is a goal for [email protected], added to the list.

[BeastyBlacksmith]

For starters I created a new orphan branch as playground to test simple implementations.
Currently it contains my vision of handling keywords, defaults and aliases. We can then test different ideas in PRs.

[isentropic]

Is it right to say that to move to 2.0, we'd need to for the most part accomplish only two things:

1. Use Preferences.jl to configure defaults
2. Use package extensions and then we could:
   1. Manage backends and remove GR as the dependency, as well as deprecate and stop testing for stale backends
   2. Possibly import many other recipes like @df as extensions

Other things are minor (but breaking nonetheless) and we can refine these later:

1. Tweak defaults
2. Change to col major layout
3. And other breaking niceties

@t-bltg @BeastyBlacksmith I have some time this week or couple and could work on this. What's the best strategy for this? Open a new branch and try to accomplish those two major things (Preferences.jl and packageext?)

[BeastyBlacksmith]

I played a bit with the Preferences stuff on the sandbox branch and found that it doesn't meet our requirements like being able to change defaults at runtime.

I started working on the Package Extension refactor here, but progress is slow and any help appreciated. I'm also open for having a call to walk you through the current state.

[isentropic]

I skimmed over the branch, man its a lot of change. Could I ask why did you opt for completely splitting the codebase into such submodules? Could you also make this fork available as a branch in the main repo, so that we can rebase and stuff easier? Are GR tests passing? Perhaps a maintainers call would be a good time

[isentropic]

As usual, thanks a lot for your amazing work

[BeastyBlacksmith]

Perhaps a maintainers call would be a good time

That would be tomorrow ;)

Could I ask why did you opt for completely splitting the codebase into such submodules?

Since I have to import stuff explicitly into the extension modules anyway I'd rather have to import a bunch of modules than each function individually

[lmiq]

Hello. I want to maybe create some reflection about not having a default backend, and the consequences for the usability and future of the Plots package.

I've gathered the data of the monthly downloads of Plots and all the backend from JuliaHub, and this is what I get:

Plots montlhy downloads: 35k

Backends:

GR: 36k
PyPlot: 5.2k
UnicodePlots: 7.6k
PGFPlotsX: 750
PlotlyJS: 5.2k
PythonPlot: 560
Gaston: 200
InspectDR: 81

I would say from that data that it is likely that Plots is used with the default backend the great majority of times. Even if all other backends were being used through it (certainly not), it would count for about 50% of the uses.

Part of the issue is just about naming "rights". Plots is a powerful name, and suggests to be the default essential plotting package for Juila (and it is). Exposing by default the backends will make it appear an accessory package and, perhaps, raise the question on why have it in the first place. A package that don't really provide the functionality could be called PlotsBase, PlotsInterfaces, PlotsCommon, or whatever.

I don't think that the default flexibility of switching backends, which is one of the original purposes of the package, is a appealing enough feature. In fact, I think Plots has grown up to be more than that (in some point of view), which is a plotting package with a very "Julian" and nice interface, mostly more natural and better than those of any of the backends.

Currently, I feel, the possibility of choosing the backend is a niche feature, that could be relegated to using PlotsBase, GR, etc, while keeping Plots as a much simpler frontend package for the default GR backend, as it is being used, apparently, by the majority of people.

It concerns that not having Plots (with that name!) with the simplest possible usability will become yet another first-Julia-trial bad first experience. Having options as environment variables or Preferences to set the default backend only makes the user experience hackier. Also, we must consider that that change will break thousands of scripts people have coded and distributed, including documentation of packages, which show some plots just taking for granted that using Plots provides a functional plot and similar essential functions.

One possibility is that these scripts will migrate to just using one of the backends directly, which I think would be bad for the future of the Plots package as a whole.

[mcabbott]

[Edit, this comment simultaneous with lmiq's just above!]

After briefly trying out the 2.0 branch, I believe the way you get 1.0's default behaviour is:

using Plots, GR
gr()

With just using Plots, the first plot tells you to do this, "E.g. import GR; gr() activates the GR backend.".

Questions:

1. Should there be a default backend always installed? UnicodePlots was mentioned on slack. At present @time_imports using UnicodePlots is surprisingly long... given that most users will (I assume) not end up using it.

2. How should the switch work?

Extensions could make import GR; plot(...) just work, but perhaps there would be surprises if you have loaded two backends. Or if some package you depend on has loaded PyPlot without you realising. I don't think extensions can tell whether the user added or loaded the package directly... maybe it can check the current environment, also a bit fragile?

The functions like gr() are a little weird, especially if exporting fewer names is a goal. I wonder whether the way to manually set a backend might be better done as a keyword accepting the module, like Plots.default(backend=GR). That would fit with per-plot setting plot(rand(30), backend=UnicodePlots), but no idea if that's easy or hard.

[BeastyBlacksmith]

Thank you for collecting these numbers!

The things I am currently trying to address with abandoning a default backend are the following:

• being lightweight: Currently its not possible to use any of the other backends without also installing GR
• improve stability by decoupling: this also ties in the former point. If for some reason installation of GR fails on your system you can't use Plots. While with v2 you can still use another backend and move on
• being explicit: I have the experience that many people don't recognize that Plots isn't a plotting library per se, rather its a unified interface for plotting libraries. While its kind of neat having such a good working abstraction that people don't notice. It creates some confusion when things go south and I go "thats a GR issue" and people answer "what is GR"?

That said it wouldn't be impossible to have the plotly backend working without additional dependencies. Like in the old days. But that comes with its own bag of Problems.
So in essence each backend comes with its own set of drawbacks and advantages and I don't want to defend why I chose one over the other. I actually want users to make a conscious decision.

I also think setting the first backend could be shorted to just using Plots; import GR but I am not clear yet whether that is something we should do.

[lmiq]

Thanks. I perfectly understand those goals. I'm just unsure about the meaning of the last one, for the package: "I have the experience that many people don't recognize that Plots isn't a plotting library per se". I also have that experience. I would say, nevertheless, that the great majority of people arrive to Plots searching for a plotting library per se, very few people choose Plots for it being a an abstraction over plotting libraries.

With this choice in 2.0 that would become explicit. But is that good? Or will it just move most people from here to just using one of the backends, decreasing the relevance of the unified interface?

I, personally, have a very specific concern about this: when I teach Julia courses, it is hard enough to get students going at the start, with all the Julia idiosyncrasies, to add one additional lower-level issue to think about. I think that there I would just choose one backend and move over, with the downside that Plots is just the best possible name for a standard plotting library.

My impression is that someone searching for the abstraction layer is easily willing to use PlotsInterfaces, or PlotsBase, etc, and understands what that means.

That said, I do not think I have anything else to contribute, so I'll not pollute further the thread, I'm sure the decision the team takes will be for the best. Thank you very much for your efforts.

[BioTurboNick]

I can imagine reasons why this might not be desired, but have you all considered making the bulk of Plots 2.0 PlotsCore or PlotsInterface and then have Plots just become a convenience package for PlotsInterface + GR?

That way people who are used to Plots experience no disruption, they can still switch backends if they like, and the backing lightweight interface package is available for those experienced users who want to use a different backend and avoid loading GR?

[BeastyBlacksmith]

I am open to that idea

[asinghvi17]

We do the same thing as is proposed here in Makie. To be specific, loading Makie does not load a backend, and people usually load as using CairoMakie or GLMakie.

What Makie does

There are three "API"s to control backends. The first is each backend's activate! function, which also allows the user to set backend specific configuration through keyword arguments.

This is most equivalent to the current gr(...) or pyplot(...) syntax.

Second, when saving a plot, the user can pass the backend and all screen configuration keyword arguments to save which will then save with that backend and configuration. For example, even if GLMakie is active, I can write save(filename, figure; backend= CairoMakie) and not mutate the global state

Third, the same keyword arguments for save are also applicable to display.

---

What Plots could do

Given that Plots' default behavior is to display a plot if you naively use using Plots; plot(...), I think that should continue to decrease breakage. Using the Plotly light backend seems like the best option to me.

There could be a Plots.activate! function which controls all backend state, to which the various gr(), plotly() functions could be shorthands, and potentially search for the backends in Base.loaded_modules and load them. Given this decoupling, it seems like indicating backend by module is a good idea, and Plots can define all rendering methods in extensions then.

These were my basic thoughts after skimming this thread, hope they help!

[isentropic]

I understand that we should prioritize the low-barrier to entry and other things for the purposes of education and just plain-ol ease of use. Currently with v2:

julia> using Plots # after ]add https://github.com/JuliaPlots/Plots.jl#v2 RecipesPipeline#v2

julia> plot(cumsum(randn(10,3); dims=1))
┌ Info: No backend activated yet. Load the backend library and call the activation function to do so.
└ E.g. `import GR; gr()` activates the GR backend.

This might be annoying yes, but how about we change such that gr backend activates implicitly after import GR and we could tweak the error message a little more like:

julia> using Plots # after ]add https://github.com/JuliaPlots/Plots.jl#v2

julia> plot(cumsum(randn(10,3); dims=1))
┌ Info: Plots.jl has updated to v2. You now probably want import GR as: `import GR` to enable graphical output. 
     For more info see [Plots.bac](https://docs.juliaplots.org/stable/backends/)
└ Now the new one liner is: `using Plots; import GR` (or other backend package of your choice)

Maybe we could even add colors and something to this message. I'm proposing that the initial backend activates itself implicitly in case there are no other backends loaded yet.

[isentropic]

For those who are in education, I strongly think having no default backend is even better because of how brittle GR at times could be. We can recommend unicodeplots backend in the Info message above in case their GR installation wasn't successful. We understand that this a breaking change, hence this a 2.0. I strongly think that this will be good long-term. It would also be good to hear what and why you'd like to change to lower the barrier and make Plots even more usable. @fonsp

We also wanna take this chance to introduce other changes like:

1. Make Plots column major by default (very breaking), consistent with Julia ecosystem and play well with others
2. Change the default colorscheme
3. Also see the number of issues above
   I feel like these are good long-term change albeit being breaking

[lmiq]

Just some additional data, on GitHUb code searches:

"using Plots" language:Julia

Results: 33.7k code entries

"using Plots" NOT ("pyplot()" OR "plotlyjs()" OR "unicodeplots() OR "pgfplotsx()" OR "pythonplot()") language:Julia

Results: 32.8k code entries.

"using Plots" "pyplot()" language:Julia

Results: 1.7k entries

"using PyPlot" language:Julia

Results: 8.9k entries