Move DArrays from Base

[ViralBShah]

We need a ton of functionality to be developed on DArrays, and perhaps even experimentation with layouts and such. I feel it is best to move the DArray functionality out of Base into the JuliaParallel organization (https://github.com/JuliaParallel) and develop it there.

[jakebolewski]

👍, I can do this.

[ViralBShah]

Cc: @andreasnoack @amitmurthy @alanedelman

[andreasnoack]

I'm fine with this.

[jakebolewski]

What about SharedArrays?

[amitmurthy]

I am fine too.

I think SharedArrays should continue to be in Base, at least till we get threading support in.

Only concern - do we have any idea how many real users of DArrays's are there? I know 0.4 is all about breaking changes, but we should generally try and minimize stuff like name changes, package changes, etc in the user community.

[ViralBShah]

I think SharedArrays should stay. Even after threading, we will want them, but perhaps then they can be in a package.

There are probably a few users of DArrays, but they are likely to be cutting edge, since the capabilities do not compose very well. The current APIs could be a lot better too.

[ViralBShah]

@jakebolewski is fast:

https://github.com/JuliaParallel/DistributedArrays.jl

[tknopp]

great! It seems #5155 is slowly happening...

[mwil]

I was using DArrays for a few computations ... How do you actually use the packages in the new location? DistributedArrays is unknown and Pkg.add("JuliaParallel"), Pkg.add("DistributedArrays") or similar is not working.

[jiahao]

@mwil if you want to stay on 0.4, you can try Pkg.clone("https://github.com/JuliaParallel/DistributedArrays.jl.git"). But most users should be using 0.3 instead.

[timholy]

Why hasn't the package been registered?

[tkelman]

Can we please not leave users out in the cold, and actually do this correctly, making sure things are usable (registered, tested, working on both 0.3 and 0.4) before we take them out of base? This won't be the last time we do this.

[jiahao]

@tkelman Nobody knows how to migrate anything out of Base correctly. I doubt very much that we can maintain backwards compatibility. Nobody figured out how to do it for Base.Graphics, and then there was the whole StatsBase débâcle before that.

[tkelman]

Whatever happened with StatsBase was well over a year ago, I don't know what went wrong there. Now we're stumbling over wrong ways to do this by taking things out very hastily before replacements are working correctly, let's not keep repeating this pattern.

Are there any specific challenges to making DistributedArrays.jl work well on both 0.3 and 0.4?

Some good first steps would be broadly announcing on the mailing lists once there's consensus that some functionality should be migrated to a package, getting current users of that functionality to migrate to the package version before merging the PR that deletes it from base, and getting things to a working state with at least the obvious kinks worked out first.

[jiahao]

I agree that removing a module from Base in a dev version at the very least merits a mention on julia-dev. However, we need the flexibility of being able to break code in order to get things done. In this case the breakage is necessary because:

1. once a type foo is exported from a module and that module is loaded, you are stuck with having that type in the Main namespace and no other module can re-export foo. So you can't have two versions of the same module coexist without breakage, and you can't change the definition of foo without breaking backwards compatibility.
2. once a function bar is exported from a module A and another module B exports the same foo, any conflicting methods will get overwritten in B. But that means that you cannot change anything in B whilst guaranteeing the same behavior as A by re-exporting bar. So it would be impossible to make breaking changes in B, thus defeating the entire purpose of this use case.

The only solution is to rename everything, but then you have to deal with the pain of synchronizing the name change globally, which was the case in the Stats -> StatsBase rename.

[kmsquire]

Conditional loading in Compat? If the module in main exists, do nothing, else load the external module...

Are there problems with this?

[tkelman]

at the very least merits a mention on julia-dev

Probably julia-users at this point. It's been six months since 0.3.0, suggesting people stay away from 0.4 isn't going to be a viable response for much longer.

However, we need the flexibility of being able to break code in order to get things done.

When there's a clear immediate upside and good reason for the breakage, of course it needs to happen occasionally. Here code was just deleted, the upside is in modularity and development constraints of being inside vs outside of base but there's no urgency on those points. It's entirely possible to recognize those particular upsides before even removing any code or causing any breakage.

The only solution is to rename everything, but then you have to deal with the pain of synchronizing the name change globally, which was the case in the Stats -> StatsBase rename.

A migration is being forced anyway. Getting various other packages to use the new Graphics version was done quickly enough, but in retrospect it should not have used the same name as the base module.

Conditional loading in Compat? If the module in main exists, do nothing, else load the external module...

Sounds worth trying. Especially if it can be done using only 0.3 features, without needing one of #6195 #6884 or #7449.

[jiahao]

Here code was just deleted, the upside is in modularity and development constraints of being inside vs outside of base but there's no urgency on those points.

Unlike the Base.Graphics removal, at this point we do have a critical mass of applications waiting to be written (like parallel linear algebra) and the labor to do it (those cc'ed in the OP) and lots of potentially breaking changes (like user-customizable transports) to try out.

Getting various other packages to use the new Graphics version was done quickly enough, but in retrospect it should not have used the same name as the base module.

The problems with Graphics.jl are not so simple that they can be solved by renaming the module. Please read what I wrote above about importing conflicting definitions more carefully. The current situation is metastable and will collapse the moment a breaking change in Graphics.jl is made since all the newly transitioned code implicitly requires backward compatibility.

Conditional loading in Compat? If the module in main exists, do nothing, else load the external module...

I don't see how Compat.jl will solve the problem of breaking backward compatibility. At best it can only proffer the kind of feature freeze which can also be expressed by putting julia 0.4-dev+1234 in the REQUIRE file of the newly externalized package. Furthermore, it appears that Compat macros do not compose with macros or other applications requiring compile-time code generation. (I couldn't figure out how to use Compat in an expression like BinDeps.@install [:lib => :symbol]. BinDeps.@install @compat Dict(:lib => :symbol) does not work, but I haven't looked more closely at how to fix this.)

[tkelman]

at this point we do have a critical mass of applications waiting to be written (like parallel linear algebra) and the labor to do it (those cc'ed in the OP)

I find this hard to take seriously when the package is still not registered or passing its tests. better now, but this work really should've been done before removing code from base.

Please read what I wrote above about importing conflicting definitions more carefully. The current situation is metastable and will collapse the moment a breaking change in Graphics.jl is made since all the newly transitioned code implicitly requires backward compatibility.

Short-term solution, why not leave things unexported from the under-development replacement versions until they've been tested and evaluated as working? Should be easy enough to test things fully-qualified at first, or only import them into local modules to avoid the conflicting imports into the main namespace. Clearly we need a better long-term solution for this problem, and to be more careful about touching the global namespace.

Should we consider un-exporting some lesser-used modules within base to lessen this transition pain?

I couldn't figure out how to use Compat in an expression like BinDeps.@install [:lib => :symbol]

It sounds from JuliaIO/HDF5.jl#216 like @compat BinDeps.@install Dict(:lib => :symbol) might actually work for that one, but it does need more testing.

[jiahao]

Short-term solution, why not leave things unexported from the under-development replacement versions until they've been tested and evaluated as working? Should be easy enough to test things fully-qualified at first, or only import them into local modules to avoid the conflicting imports into the main namespace.

I think that's the only viable solution right now - keep previously exported identifiers untouched and develop new replacements in a different namespace or with different identifiers.

[tknopp]

I am with @tkelman that it would be good to first create and register the package before removing it. But other than that I cannot see whats the issue. Master is not(!) for users and I cannot see the point of discussing these things on the julia-users mailing list. A half year since the last release is not an argument, we still should not encourage using 0.4dev for users. This leads to the same situation as in 0.3 where we had a rolling release in the end.

Personally I do not fully understand the point of the Compat.jl package since IMHO packages should not developed against master. Better make more regular Julia releases instead if the features are really that important.

[tkelman]

Better make more regular Julia releases instead if the features are really that important.

This is the point. Features are important, staged functions and performance improvements and other changes are resulting in a group of users (and that big gray area between users and contributors) who do work against master. That's inherently risky of them, but letting them know things will be moving ahead of time is the best way to get their help.

More regular major releases is a good goal, but that's going to need feature definition, stabilization, and wide testing for the RC stage. But anyway we're way off topic.

[tknopp]

I think it needs destabilizing on master to get features in. And justifications on julia mailing lists while the primary development platform is github seems redundant.

Don't think that this is off-topic at all since you were not happy how this got merged.

[ViralBShah]

I believe the DistributedArrays.jl package has to still be added to METADATA. Any reason not to do it yet?

[jakebolewski]

The package is registered and tagged now that all the functionality is working again.

[timholy]

👍