bundle package flag --all-platforms has no effect

[tam7t]

When i run bundle package --all-platforms I would expect vendor/cache to be populated with .gem for all platforms, however I only get the current platform gem.

# bundle env
Environment

    Bundler   1.11.2
    Rubygems  2.5.1
    Ruby      2.3.0p0 (2015-12-25 revision 53290) [x86_64-linux]
    GEM_HOME  /usr/local/bundle
    GEM_PATH
    Git       2.1.4

Bundler settings

    path
      Set for the current user (/root/.bundle/config): "/usr/local/bundle"
    bin
      Set for the current user (/root/.bundle/config): "/usr/local/bundle/bin"
    silence_root_warning
      Set for the current user (/root/.bundle/config): "true"
    app_config
      Set via BUNDLE_APP_CONFIG: "/usr/local/bundle"

Gemfile

    source 'https://rubygems.org'

    gem 'google-protobuf', '3.0.0.alpha.5.0.3'

Gemfile.lock

    GEM
      remote: https://rubygems.org/
      specs:
        google-protobuf (3.0.0.alpha.5.0.3)

    PLATFORMS
      ruby

    DEPENDENCIES
      google-protobuf (= 3.0.0.alpha.5.0.3)

    BUNDLED WITH
       1.11.2

and

# bundle package --all-platforms --verbose
HTTP GET https://bundler.rubygems.org/api/v1/dependencies
HTTP 200 OK
Fetching gem metadata from https://rubygems.org/
Query List: ["google-protobuf"]
Query Gemcutter Dependency Endpoint API: google-protobuf
HTTP GET https://bundler.rubygems.org/api/v1/dependencies?gems=google-protobuf
HTTP 200 OK
Query List: []
Fetching version metadata from https://rubygems.org/
Query List: []
Installing google-protobuf 3.0.0.alpha.5.0.3
0:  google-protobuf (3.0.0.alpha.5.0.3) from /usr/local/bundle/specifications/google-protobuf-3.0.0.alpha.5.0.3-x86_64-linux.gemspec
Using bundler 1.11.2
0:  bundler (1.11.2) from /usr/local/bundle/specifications/bundler-1.11.2.gemspec
Bundle complete! 1 Gemfile dependency, 2 gems now installed.
Bundled gems are installed into /usr/local/bundle.
Updating files in vendor/cache
  * google-protobuf-3.0.0.alpha.5.0.3-x86_64-linux.gem

My ultimate desire would be to just get the ruby platform version and build from source, but I thought that --all-platforms would be the next best alternative. I couldn't find this reported elsewhere (but was mentioned in rubygems/bundler-features#4 (comment))

[segiddins]

That's because you only have the ruby platform in your lockfile, I believe

[tam7t]

There are multiple gem files that match the ruby platform depending on your processor architecture. The following is listed on rubygems. I would expect them all to be included in the cache with --all-platforms.

3.0.0.alpha.5.0.3 - February 11, 2016 x86-mingw32 (894 KB)
3.0.0.alpha.5.0.3 - February 11, 2016 x86-linux (743 KB)
3.0.0.alpha.5.0.3 - February 11, 2016 (206 KB)
3.0.0.alpha.5.0.3 - February 11, 2016 x64-mingw32 (949 KB)
3.0.0.alpha.5.0.3 - February 11, 2016 universal-darwin (742 KB)
3.0.0.alpha.5.0.3 - February 11, 2016 x86_64-linux (860 KB) 

[segiddins]

I believe --all-platforms just considers all platforms in the lockfile. if you want to include those gems, you'll need to bundle on those platforms so they're in the lock file.

[tam7t]

How would that work when the generated lock file on osx and on linux are the same, but the cached packages are incompatible?

[indirect]

@tam7t the cached packages are compatible. the cache contains the .gem files and nothing else.

[indirect]

hmm. that may only be partially true in this case. it is extremely rare for a gem to have prebuilt binary versions on either Linux or Darwin, but it seems that this gem does.

the gem that has no specific platform (which is the "ruby" platform) should contain the source code that needs to be compiled on either Linux or Darwin for the gem to function. it is the version that is cross-compatible. the precompiled versions will not be cross compatible (but they also don't have the platform "ruby", which seems to be the only one present in your Gemfile at the moment).

[tam7t]

@indirect are you suggesting that google is wrong to supply binary-builds in addition to their source-only gem? I would just expect that bundle package would prefer to source-only one or that bundle package --all-platforms would include all matching platforms and not just the one used by bundle install. The workaround we have is to manually include https://rubygems.org/downloads/google-protobuf-3.0.0.alpha.5.0.3.gem in the cache, but I just thought that bundler should handle this scenario more gracefully.

[indirect]

I'm suggesting that Google is alone in shipping binaries for Linux and Darwin among gem creators. Because that is a unique situation, Bundler does not have special handling for that case.

[tam7t]

Thanks! Out of curiosity I downloaded a copy of the rubygem database and found only < 100 gems that would be affected by this. Lucky me!

[segiddins]

I've made some progress on this -- it turned out to involve re-writing a lot of how the resolver and lockfile deal with platforms