package has a large filesystem footprint #264
Comments
|
I agree! Do you have any experience with this? Last time I tried to do On Wednesday, May 18, 2016, Tanmay Mohapatra [email protected]
|
|
Thanks @KristofferC I think this looks like a good option. I'm going to put it off a little longer until my dev work slows down. |
|
You realize that this will make all previous releases void since all commit SHA's will change. People will not be able to just run a Pkg.update. It definitely requires some thought how to do it best. |
|
Yes, appears difficult to do without some sort of support in the package manager. |
|
Yes... I assume I'll have to update METADATA with new commit info for each tag? I'm not ready to tackle that mess yet. |
|
As mentioned above, the suggestion was to use https://rtyley.github.io/bfg-repo-cleaner/. I think this will require updates to the commit shas in METADATA... do you foresee any other issues? |
|
I see 116MB total... 114MB is in the |
|
@StefanKarpinski says this is impossible. I guess the only real option is to wait for Pkg3. With libgit2, we can't even do shallow clones. |
|
I don't understand why this would be impossible. @StefanKarpinski maybe you can explain a little more? If the repo had a fresh commit history and we updated METADATA appropriately, why wouldn't this work? It might require people to manually delete their local download of Plots, I suppose? (which wouldn't make it impossible, just annoying) |
|
I think having people removing Plots.jl is ok to clean this up once, while the package is still in relatively early days and before it really explodes. |
|
We could just overwrite all the SHA1s for versions in METADATA, but that seems like a huge risk to me – it's basically indistinguishable from an attack on METADATA and I'm not sure how the package manager will react. We could do some testing first and see what happens? |
|
It would likely break updating for anyone who has an existing clone. Try tesing with single branch clones first, which wouldn't rewrite the existing tags completely. |
|
If we ever decide to do this, I tried locally and I think these are the non-METADATA steps: |
|
I have a local repo that seems to be small and in tact: |
|
The real question is how will Pkg interact with it. Is there any way to simulate or test that? |
|
Pkg3 was mentioned, but I haven't heard anything about it since JuliaCon... what's the latest? |
|
Pkg3 would only be relevant here if it's planning to move entirely away from using git repos and instead using non-repo tarball downloads of packages. Updating from an existing install to a repo where the history has been rewritten is not likely to work smoothly. Pkg.rm doesn't actually delete the package which will make this messier to deal with for users, and there's the separate .cache bare clone to worry about. |
|
Just for discussions sake, what would be the proper way to purge a repo from a user's system so that Pkg would download a fresh copy and start from scratch? (i.e. the most conservative way to make sure it's gone) |
|
How about creating Plots.jl as a new package with a new name? And then eventually renaming this package eventually to the new package so that we retain the issues and PRs. The downside is that we lose the nice Plots.jl name. |
|
You'd need to delete Plots from all the |
|
My understanding is that this needs a new repo, if we don't want to inconvenience users - and that there is no clean way otherwise. Shouldn't we do it sooner rather than later? Perhaps UnionOfPlots.jl. :-) I would hate to lose the issues and such, but perhaps github support can help us migrate them over. |
|
My understanding on Pkg3 from @StefanKarpinski is that it is expected in the 0.6 release timeframe. However, code readiness and migration to Pkg3 are completely different things, and perhaps even if Pkg3 is ready optimistically by around Jan 2017, it may take another 2-3 months to work out the kinks and migrate. |
|
The name is not changing. Users can be inconvenienced once if they need to On Sunday, October 16, 2016, Viral B. Shah [email protected] wrote:
|
|
I will only say that I wish this package was not 100MB. I am ok with whatever solution you choose to go with. |
|
I'm going to do this today, so prepare for the carnage. My plan:
|
|
If you break the existing tags, we'll be redirecting METADATA to point to a fork. Please don't break existing tags. Deleting them from METADATA won't be merged. |
|
Backed up: https://github.com/JuliaPlots/PlotsBackup.jl @tkelman would you prefer to change the url to point to the backup until the dust settles? |
|
Yup... and how many packages do that? |
|
Registered packages, none directly. JuliaBox does. Shipping products do.
Except users' analyses that they generated for a publication that needs revision a month later. |
|
Is is possible to use months-old versions with any of these at this point though? |
|
If I understand the implications of the output below, only the packages ExperimentalAnalysis and ImplicitEquations will possibly force older versions of Plots to be installed: |
|
Renaming Plots to PlotsBackup isn't the problem since GitHub handles redirection for you. The problem is then putting something in the place where Plots used to be but which is a completely unrelated git repo, which will confuse anyone's installation who has Plots installed. We went through this with the Stats/StatsBase renaming and it was rough – it seems like a bad experience to foist on Plots users. I'm not sure about the implications of changing tagged versions but I don't really like it. It seems likely to cause problems. @tbreloff, if you want to go that way, you should make a fork of METADATA and try it (and get some other people to try it as well with Plots previously installed). Otherwise, I think what @tkelman is proposing using single-branch clones is the best way to go, although I have some technological doubts there tbh. |
Only if you actually do a rename under the settings. If you just push a separate copy as a brand new repository, then there aren't any redirections. |
|
Well, the repo size doesn't really bother me. And I've spent about 10 hours too many on this issue. I'm just burnt out with Julia package management. If you guys care so much about the repo size, then let me break old tags. If you don't care enough, then I'll close the issue. |
|
It is quite likely that Plots.jl has fewer users than the stats packages did. I'd rather make a clean break personally and get a smaller package. Or just leave it as it is - since clearly there isn't a solution that satisfies all constraints. For my own use, I'll probably make my own clean copy of the Plots.jl repo, if the size bothers me too much going forward. Feel free to close it. |
|
It'd be great to have this done, keeping the name and infrastructure of Plots. With the current development of Pkg3, does it look like that will support a good solution? |
|
AFAIU Pkg3 will cause a large enough change that everything has to be "redone" and then SHA:s for old versions could be changed. |
|
This will be fixed by Pkg3, I've added |
|
Repo size is preventing me from using Plots on slower internet connections (which unfortunately I deal with a lot out in rural areas of USA). Anything that can be done to trim package sizes helps productivity for me and anyone leveraging what I'm doing. I'm currently waiting for the |
|
Pkg3 will use tarballs for standard packages which tend to be less than a megabyte. So this issue will be resolved when Pkg3 is merged into Base which will happen for 0.7 |
|
Close this as something that won't be fixed in Pkg2 ? Pkg3 will be out "soon" and this will be fixed automatically. Until then there is nothing we can do, so... |
|
Latest Plots archive is 192KB which is what will be downloaded in Pkg3. Could keep this open just to have something to close when Pkg3 lands ;) |
|
I think Pkg3 landing will leave you plenty of issues to close, but if you need something to do that afternoon we can keep this open :) |
|
As a teaser, this is installing Plots from scratch (in real time): |
|
Holy Moly, that's amazing. @pkofod and @KristofferC we could also close this issue ritually at some point where we're in a position to clink cold beer bottles together? |
|
Sounds like a plan! |
|
Just wait until the build steps are handled by BinDep2 and we spend some more time optimizing the resolver. Then this will really fly. |
|
I think it is time - and I will provide the cold beer bottles to clink together 🎉 ! So, when and where? :-) |
|
Could we do it tomorrow? :) |
|
Do it when you tag :) |
|
Missed the beer but I think this can be closed now anyway 🎉 |
|
I'll buy both of you beer when I get the chance. And anyone else in this thread :-) |
|
A Plots.jl with a small filesystem footprint... This is a whole new world! |
Plots.jl filesystem footprint is at around 110MB, mostly from its git history.
And they seem to be due to files deleted quite some time back.
It will be good to purge/reduce it in some way.
The text was updated successfully, but these errors were encountered: