Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Roadmap -- JuliaPlots #222

Closed
13 of 23 tasks
tbreloff opened this issue May 5, 2016 · 25 comments
Closed
13 of 23 tasks

Roadmap -- JuliaPlots #222

tbreloff opened this issue May 5, 2016 · 25 comments

Comments

@tbreloff
Copy link
Member

tbreloff commented May 5, 2016

This issue continues the discussion from https://gitter.im/tbreloff/Plots.jl. Summarizing, there are a few goals that could be achieved through a repo restructuring into a new organization.

  • Keeping the Plots ecosystem within one account
  • Consolidation of Plots recipes across disciplines
  • Opportunities for collaboration and spreading of responsibilities
  • A first step towards becoming the "standard library" plotting package for Julia

There exist a couple possibilities of organizations to join, but I would prefer to maintain sole ownership over core Plots.jl for the time being. To that end, I propose forming the new organization JuliaPlots, of which I am the owner, and creating teams of collaborators for sub-systems of Plots:

  • Plots backends
  • Plots recipes: statistics, machine learning, graphs, and whatever else
  • Plots themes: ggplot2, solarized, etc
  • Plots examples/docs

My opinion on what should happen next:

  • Create RecipesBase.jl and move recipe macro there
  • Register RecipesBase with METADATA
  • Transfer tbreloff/Plots.jl to JuliaPlots/Plots.jl (maybe another time)
  • Update METADATA for Plots (what are the necessary steps??)
  • Create a batman logo for the org! (or... any other ideas?)
  • Transfer tbreloff/ExamplePlots.jl to JuliaPlots/PlotExamples.jl (or PlotDocs.jl?)
  • Update readthedocs site to point to JuliaPlots/PlotExamples.jl
  • Search for and replace hardcoded references to ExamplePlots
  • Create a team for docs maintenance (just me for now, unless anyone's interested)
  • Create JuliaPlots/PlotBackends.jl repo, and move much of the backend-specific code there. (over time, possibly move it into the individual backends?)
  • Create a team for maintaining the backends
  • Create JuliaPlots/PlotThemes.jl (@pkofod: or transfer from pkofod/PlotThemes.jl?)
  • Create a team for creating/maintaining themes
  • Create JuliaPlots/PlotRecipes.jl (@Evizero: or transfer JuliaML/MLPlots.jl as the starting point?)
  • Create a team for creating/maintaining recipes
  • Transfer tbreloff/VisualRegressionTests.jl into JuliaPlots
  • Update METADATA for VisualRegressionTests
  • Register the packages with METADATA
  • Submit PRs for http://julialang.org/community/, http://julialang.org/downloads/, and http://julialang.org/downloads/plotting.html
  • Write a blog post about the "how and why" of Plots

And there are some things which I think would be good for the long term, but don't need to happen soon:

  • Remove the JuliaPlot org (which was mainly for discussions last year), and include any pertinent info in the Plots docs or in an issue.
  • Pull out much of the code dealing with colors into a separate repo (@lobingera: maybe ColorMaps.jl?)
  • Move gadfly_shapes.jl into Gadfly

Everyone, please let me know your thoughts.

cc: @spencerlyon2 @diegozea @jheinen @Evizero @dlfivefifty @timholy @pkofod @SimonDanisch

@tbreloff
Copy link
Member Author

tbreloff commented May 5, 2016

Ref for changing repo URL: JuliaLang/METADATA.jl#649

@tbreloff
Copy link
Member Author

tbreloff commented May 5, 2016

By popular demand:

tmp

@Evizero
Copy link
Member

Evizero commented May 5, 2016

I went through the chat history to understand this better.

I see the motivation. I am a little confused about you proposing to move your biggest baby away from your personal account. Have you thought that through? You can always do this at some later point when you lose the time or interest to put so much effort into it.

Personally, I see the ability to move a package to an organization as a tool for different agendas

  • Increase the perceived legitimacy of the package (I do not think Plots needs that)
  • Perceived shared ownership: Make other people more interested in contributing as it appears to not have a sole owner anymore, but a group project (Again, I think Plots has a good momentum already)
  • Give up ownership since time or interest to maintain has faded (Also doesn't seem to apply to Plots)

On the contrary I also think having a package in ones personal account has the advantage of perceived sole ownership. For some reason one feels a lot more responsible for his/her own packages. Maybe because the github landing page feels a little like a portfolio and there is always this thought lingering that people might look you up and browse through your Popular repositories. so you want them to be nice and clean

Regardless. Moving a package isn't that difficult. the Metadata entry has a link that needs to be updated, so thats trivial. The most annoyances are travis, readthedocs, and coverall, but they aren't difficult, they are mainly just annoying and take some time to set up.

Concerning MLPlots: I think MLPlots should exist on its own as part of JuliaML and at least contain ML focused plots; especially for the packages in JuliaML. I do get the motivation for PlotRecipies and I also think it should exist. It does make more sense to me though that domain specific things should be closer to it's real domain. That seems a lot easier to maintain and also easier to stay interested in maintaining in the first place.

@tbreloff
Copy link
Member Author

tbreloff commented May 5, 2016

your biggest baby

My biggest baby is downstairs learning to crawl right now ;)

But seriously, I am with you on hesitations about the weakening the connection between @tbreloff and Plots.jl. I want that association to stay, but that reason alone isn't compelling enough. More important to me is not giving away commit access to the core of Plots. I'm just not ready for anyone else to make decisions regarding API/internals without my approval, which is why I think it makes more sense to start JuliaPlots (instead of moving to JuliaGraphics) where I retain veto power on the core.

The idea is that Plots.jl would still be my baby (for the time being) and that I would be the "owner" of JuliaPlots. Any non-core aspects of the ecosystem would be broken away from Plots.jl into packages and I would give commit access to those packages for collaborators. This way I don't need to approve all backend-specific code, or recipes, or themes, but I do have to approve anything that affects the core API and internals.

some later point when you lose the time or interest

I don't want to wait for this moment before making a move, as then I won't be able to transition it well.

Increase the perceived legitimacy of the package

Actually I think this is the main draw, and there's still a big gap between the people using/watching this repo vs the core devs approval/inclusion. Much of the organization problems could be solved if Plots was part of the julia "standard library" and package authors could be confident that most users would have it installed. Then package-specific visualizations (like those in MLPlots) could live inside the packages, not in a separate "recipes" package. The main thing preventing that is the core devs don't ever want packages to depend on graphics (even though pretty much everyone uses plotting software of some kind). I think moving the ecosystem away from my account increases the chances of making Plots a standard and officially-sanctioned package which many other packages depend on.

Concerning MLPlots

Like I said... in my ideal world MLPlots wouldn't really exist either because the plotting routines would live in LearnBase, OnlineAI, etc. But as a stop-gap measure I'm ok leaving the ML-specific routines there (and still having a PlotRecipes.jl for anything else).

@waldyrious
Copy link
Contributor

waldyrious commented May 5, 2016

some later point when you lose the time or interest

I don't want to wait for this moment before making a move, as then I won't be able to transition it well.

+:100:, I've seen this happen way too often, and the situation is pretty terrible because if no one is available to bless new maintainers, even forking the repo is not an attractive prospect as there's no guarantee of buy-in from the current users (not to mention it fragments the userbase).

Besides, having an organization makes it clear who its members are, whereas IIRC there's no way to see who has collaborator access to a repository, which is important so that contributors can know who to reach out to, ping, etc.

@Evizero
Copy link
Member

Evizero commented May 5, 2016

In the end it is your decision. I do get your argument, but ultimately the idea of yet-another-graphics-org seems counter productive on it's own. especially given the existence of JuliaGraphics where Immerse is.
Yet maybe such a compromise could achieve what you are aiming at.

The main thing preventing that is the core devs don't ever want packages to depend on graphics (even though pretty much everyone uses plotting software of some kind). I think moving the ecosystem away from my account increases the chances of making Plots a standard and officially-sanctioned package which many other packages depend on.

It is not that simple I'm afraid. Just think how minimalistic UnicodePlots is and yet even you were very hesitant to have it as a dependency in LearnBase for show methods. And UnicodePlots doesn't even have dependencies.

Bottom line: I do not know what the answer is but it needs a very very clever solution to achieve what you are after

@tbreloff
Copy link
Member Author

tbreloff commented May 5, 2016

Just think how minimalistic UnicodePlots is and yet even you were very hesitant to have it as a dependency in LearnBase for show methods.

You got me there. Plotting will always be contentious I suppose, and maybe it's wishful thinking that there will ever be a "standard library" plotting option that people can get behind. But... if people were forced to pick, it would be Plots, right?? ;)

@SimonDanisch
Copy link
Member

I think most packages shouldn't depend on a graphic package but should instead emit data that is easily visualizable by any graphic package. This doesn't only decrease dependencies, but it also makes everything more generic ;)

@Evizero
Copy link
Member

Evizero commented May 5, 2016

Plotting will always be contentious I suppose

Personally I think it's less about contention and more about the perfectionist (and minimalist) tendencies of people interested in writing optimized code. The Julia community seems to attract many such people and I am considering myself as one of them.

Furthermore, the design of Julia's package manager does promote the idea of having little islands of functionality that follow the "linux school of thought" of doing one thing only but one thing right. The package manager still has it's flaws but I did learn to like it

@Evizero
Copy link
Member

Evizero commented May 5, 2016

I wonder. How minimalistic can you make a package that defines everything needed to create custom _apply_recipe methods? Naively I would assume such a package needs only very very little code and no dependencies, no? If that is the case I could see things like learn base depend on it, since it would basically just be a package that defines a couple verbs

@tbreloff
Copy link
Member Author

tbreloff commented May 5, 2016

How minimalistic can you make a package that defines everything needed to create custom _apply_recipe methods?

💯 This is a really good idea! I want to think through the details of how this will work and add this to the restructuring plan.

Also I added a @recipe macro yesterday, getting even easier for package authors: http://plots.readthedocs.io/en/latest/pipeline/#step-3-_apply_recipe-callbacks

@diegozea
Copy link
Contributor

diegozea commented May 5, 2016

Plots is already minimal, but having a smaller option for package developers sounds good.

@Evizero
Copy link
Member

Evizero commented May 5, 2016

my personal tip: it should be as minimal as possible to promote adoption (every line counts). There should really be no dependency (not even colors or requires). If this can be done I am confident we can get the ball rolling with the state plots is in. I certainly would build it into the packages I am the maintainer of to help gain momentum

@tbreloff
Copy link
Member Author

tbreloff commented May 5, 2016

Agreed completely. This will be the first package that I add to JuliaPlots... absolutely minimal to allow recipe definitions for external packages. I'll post when I have something usable, and add it as a dependency to Plots.

@Evizero
Copy link
Member

Evizero commented May 6, 2016

One more thing I would consider important for wide spread adoption is a good way of testing the custom plots. The visual regression tests already take care of it for the most part but what I would still be unhappy with from a userstandpoint is the default huge size of the refimages. These more or less bloat the githistory. Is there a clean way to make the reference images by default really small and neatly compressed while still containing the relevant information?

@tbreloff
Copy link
Member Author

tbreloff commented May 6, 2016

Another great idea! Automatic compression/decompression as part of
VisualRegressionTests would be great to have.

On Friday, May 6, 2016, Christof Stocker [email protected] wrote:

One more thing I would consider important for wide spread adoption is a
good way of testing the custom plots. The visual regression tests already
take care of it for the most part but what I would still be unhappy with
from a userstandpoint is the default huge size of the refimages. These more
or bloat the githistory. Is a clean way to make the reference images by
default really small and neatly compressed while still containing the
relevant information?


You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
#222 (comment)

@Evizero
Copy link
Member

Evizero commented May 6, 2016

I am sure a lot could be achieved by a good deterministic way of downscaling the images

@lobingera
Copy link

to pick up from way above...

Yes, i could provide maintenance for a ColorMaps.jl if needed; i'd not claim that i'll be better than other choices, maybe i have a little bit more overview, as i did something similar for a research project once.

I cannot really see the need for another julia graphics/plots/plot organisation as https://github.com/JuliaGraphics/ and https://github.com/JuliaPlot/ already exist (and even that could be converged into 1 organisation only).
If you (@tbreloff) see problems of other developers interfering with your ideas, the simpler solution is to keep ownership on your account.

Still (especially now after the ShowOff + Gadfly + Compose incident) i'd rather like to see infrastructure packages (and Plots as mediator between common API and a list of backends is very clearly infrastructure) as shared development. Not only to provide diversity if an owner isn't available for accepting a PR. Diversity is always a source for new inspiration - but this can also be provided by enough github discussion.

@Evizero
Copy link
Member

Evizero commented May 6, 2016

I think the plan is to delete the old JuliaPlot.

Generally speaking I agree about JuliaGraphics, but practically speaking I do see a strong case for JuliaPlots existence. Bluntly put I see it like this: Tom continuously puts a lot of effort into improving Plots.jl for a long time now, and egoistic me would like it if that momentum continued for a while. Having a strong sense of visible authorship and control can have a powerful influence on ones motivation to spend significant time on something, so I think JuliaPlots is a decent compromise for a transition between sole authorship and community project. So, yes, in a simple world it would be better to just use JuliaGraphics for this, but the world is not that simple

@tbreloff
Copy link
Member Author

tbreloff commented May 6, 2016

I cannot really see the need for another julia graphics/plots/plot organisation

I really go back and forth on this one. Right now, my perspective is that JuliaGraphics is the better place for drawing-type package (like Compose, Colors, etc). A ColorMaps.jl should probably live there as well. I would like to see "data visualization" packages tend towards JuliaPlots, but that's probably wishful thinking in the short term. JuliaPlot should be removed... there's really nothing there but a few issues that contain some discussion from a year ago.

If you (@tbreloff) see problems of other developers interfering with your ideas, the simpler solution is to keep ownership on your account.

For now I haven't made a decision on what to do with core Plots... it'll stay at tbreloff for the time being. I do, however, want to pull out some pieces from within Plots and make them independent projects. The first of those is RecipesBase.

especially now after the ShowOff + Gadfly + Compose incident

This sort of scenario is very much on my mind, and I think eventually core Plots will be more of a community effort, but for now I'm too actively involved and I'm just not ready for that. You have my promise that I'll give others write access if I'm not able to spend time on the project.

@pkofod
Copy link
Contributor

pkofod commented May 6, 2016

Just trying to follow the discussion: what incident?

@tbreloff
Copy link
Member Author

tbreloff commented May 6, 2016

In a nutshell, Dan Jones went MIA, and he is the only one with commit access to ShowOff, a Gadfly dependency. As changes needed to be merged, everyone scrambled to fork it and point METADATA to the fork, and caused a lot of hemming and hawing about what to do.

@cormullion
Copy link

Talking of color maps, how many ways are there in Plots to specify a range of colors? While adding some examples for ColorSchemes I noticed c=... for contour and palette= for plots — isn't there also a cmap or colormap as well? (docsearch didn't work, sadly... :))

@tbreloff
Copy link
Member Author

tbreloff commented Jun 4, 2016

Those are all different. "c" is an alias for "seriescolor", "palette" is a
mechanism for setting the color list when automatically choosing series
colors, and I need to check on cmap/colormap... I don't remember those as
args.

On Saturday, June 4, 2016, cormullion [email protected] wrote:

Talking of color maps, how many ways are there in Plots to specify a range
of colors? While adding some examples for ColorSchemes
https://github.com/cormullion/ColorSchemes.jl I noticed c=... for
contour and palette= for plots — isn't there also a cmap or colormap as
well? (docsearch didn't work, sadly... :))


You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#222 (comment),
or mute the thread
https://github.com/notifications/unsubscribe/AA492icIMjViEKlk4JbM91TWe32hkD1tks5qIcC_gaJpZM4IXsDS
.

@mkborregaard
Copy link
Member

I think this is stale, and might be closed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

9 participants