[gettext] Split, fixes, faster build, checked-in config cache #30429

dg0yt · 2023-03-25T18:09:20Z

Major contributions:

Splits gettext-libintl out of gettext.
- Allows to simplify the portfiles (at the price of some redundancy e.g. in patches).
- Allows to clearly mark gettext-libintl as being under LGPL, not GPL.
- Allows to install gettext[tools] without reinstalling ports which only need gettext-libintl.
  To fully leverage the effect, downstream ports still need to switch dependencies.
Reverts use of the original makefiles, but adds new modifications to improve port build time.
In particlar, uses bash built-in evaluations instead of sub-shells, skips some subdirs, and enables concurrency in one large subdir.
(I won't try to upstream this. Upstream targets a larger set of platforms and tools and will refuse these changes. We have seen this for libunistring.)
~~Adds a script port which helps to maintain autoconf configuration cache files. This greatly reduces the time which is spent in the configure step on Windows.~~
Removed from PR
Adds a few more bits to what is quickly installed by gettext[core], removing some extra duties from port mchehab-zbar.
Adds more known/required configuration results, for speed and correctness.
Fixes [gettext] Build error (conflict with libxml2) #30579, libxml2 installation order problem.

.

Changes comply with the maintainer guide
SHA512s are updated for each updated download
The "supports" clause reflects platforms that may be fixed by this new version
Any fixed CI baseline entries are removed from that file.
Any patches that are no longer applied are deleted from the port's directory.
The version database is fixed by rerunning ./vcpkg x-add-version --all and committing the result.
Only one version is added to each modified port's versions file.

Thomas1664 · 2023-03-26T15:04:39Z

Maybe I should really start with the vcpkg-make port.

+1; I asked myself multiple times why cmake has its own port but make doesn't.

Coincidentally, I worked a few weeks ago on optimizing make builds as well. I thought that using native binaries instead of msys2 could speed up configure/ build but I was only able to compile make for Windows. Unfortunately, I was unable to get successful builds with it. Next I tried bash which also didn't work because it is unable to find getopt.h although I declared a dependency on getopt-win32 but I think this won't be the only missing header... I also thought about caching the results of configure but I didn't know how to implement this.

That being said, I highly appreciate your work.

dg0yt · 2023-03-26T16:16:49Z

MSYS2 exists because no better sufficiently complete environment exists for doing autotools builds on Windows. Probably it is reasonable to assume that they do a fairly good job with regard to the trade-off between compatibility and performance.
However, the autobuild buildsystems focus on portability at the price of performance. This implies using the smallest common denominator with regard to sh and make features, testing for toolchain properties, and applying gnulib portability polyfill. In addition, sometimes steps don't make good use of concurrency (cf. my libunistring patch).

For gettext[core]:x64-mingw-static, most native "build" time is actually spent in the configure step. Figures from my windows test machine, per build type (debug or release):

The build step takes app. 40 s.
The configure step takes more than 200 s.
- Reusing the full config cache reduces the this to app. 120 s.
"Bashifying" the frequent "x='`echo ... | sed ...`'" patterns (from AC_SUBST) saves another 15 s.
The ultimate improvement would be parallel configure. (And it shouldn't be too difficult to implement.)

(For the tracing of configure, I piped the output into ts from moreutils.)

At the moment I'm reviewing the differences of the windows (MSVC) config caches, as available from the CI failure logs. Some findings are surprising (at least for me):

Some values are "guessed" for non-native builds. It should be save to pre-seed this also for native builds.
e.g. -x64 (native) vs. x86 (cross):

--- collected/rel/config.cache-x64-windows-rel.log      2023-03-26 18:03:35.281517294 +0200
+++ collected/rel/config.cache-x86-windows-rel.log      2023-03-26 18:03:35.285517351 +0200
@@ -97 +97 @@
-ac_cv_func_malloc_0_nonnull=${ac_cv_func_malloc_0_nonnull=yes}
+ac_cv_func_malloc_0_nonnull=${ac_cv_func_malloc_0_nonnull='guessing yes'}

Some guessed values differ from detected values.
e.g. -x64 (native) vs. x86 (cross):

--- collected/rel/config.cache-x64-windows-rel.log      2023-03-26 18:03:35.281517294 +0200
+++ collected/rel/config.cache-x86-windows-rel.log      2023-03-26 18:03:35.285517351 +0200
@@ -469 +469 @@
-gl_cv_func_getcwd_null=${gl_cv_func_getcwd_null=yes}
+gl_cv_func_getcwd_null=${gl_cv_func_getcwd_null='guessing no'}

Even some detected values differ unexpectedly.
e.g. x64-windows (native) vs. x64-windows-static (cross):

--- collected/rel/config.cache-x64-windows-rel.log      2023-03-26 18:10:57.587822733 +0200
+++ collected/rel/config.cache-x64-windows-static-rel.log       2023-03-26 18:10:57.587822733 +0200
@@ -153 +153 @@
-ac_cv_func_snprintf=${ac_cv_func_snprintf=no}
+ac_cv_func_snprintf=${ac_cv_func_snprintf=yes}

dg0yt · 2023-03-27T06:40:07Z

ports/gettext/portfile.cmake

@@ -27,6 +27,27 @@ vcpkg_find_acquire_program(BISON)
 get_filename_component(BISON_PATH "${BISON}" DIRECTORY)
 vcpkg_add_to_path("${BISON_PATH}")

+if(NOT DEFINED VCPKG_AUTOTOOLS_CONFIG_CACHE)


Defining VCPKG_AUTOTOOLS_CONFIG_CACHE to a non-existing file (e.g. NONE) may be used to disable an existing cache in custom triplet files.

ports/gettext/portfile.cmake

dg0yt · 2023-03-27T06:42:06Z

ports/gettext/portfile.cmake

-        file(READ "${CURRENT_BUILDTREES_DIR}/config.cache-${TARGET_TRIPLET}" config_cache)
-        string(REGEX REPLACE "\nac_cv_env[^\n]*" "" config_cache "${config_cache}") # Eliminate build type flags
-        file(WRITE "${CURRENT_BUILDTREES_DIR}/config.cache-${TARGET_TRIPLET}" "${config_cache}")
+        vcpkg_config_cache_reuse()


Even in absence of an initial (checked-in) cache, this could generally speed up configuration for the second build type.

No longer used in these ports.

dg0yt · 2023-03-27T06:51:08Z

gettext[core,tools]:x64-windows: Down from 10.8 to 8.0 min.
gettext[core]:x64-uwp: Down from 4.4 min to 2.1 min.

dg0yt · 2023-04-10T12:53:37Z

With configuration caching, further tuning, libintl moved to a separate port and gettext-runtime tools moved to another feature:
gettext-libintl:x64-windows: Down to 1.7 min.
gettext[tools]:x64-windows: Down to 4.3 min.

In total, build time is down from to 10.7 min to 6 min. But there are other benefits:

Many ports only need to depend on gettext-libintl (at least if not using native language support). Installing tools later doesn't need to trigger a rebuild.
The custom Makefile is gone. Subdirs are built in the same order as intented upstream. (But still some subdirs are omitted.)
Splitting libintl out of gettext allows simpler portfiles, and a clear indication of license for each of the ports.

gettext[core] should now have everthing needed before running autoreconf. A particular use case: port mchehab-zbar.

Switching downstream ports from gettext to libintl should be done separately. There are two many of them. Merge conflicts and regression would block a merge for too long.

dg0yt · 2023-04-10T13:04:01Z

FTR the checked-in config.cache files were reviewed manually:

windows: Unified across CI configurations.
mingw: Unified x64 across MSYS2 and Ubuntu 18.04 host environments.

MonicaLiu0311 · 2023-05-05T08:10:45Z

gettext-libintl

The usage test passed (header files found):

gettext-libintl is compatible with built-in CMake targets:

    find_package(Intl REQUIRED)
    target_link_libraries(main PRIVATE Intl::Intl) # since CMake 3.20

gettext

All features are tested successfully in the following triplet:

x86-windows
x64-windows
x64-windows-static

mchehab-zbar

All features are tested successfully in the following triplet:

x86-windows
x64-windows
x64-windows-static
arm64-windows

JavierMatosD

Overall, I like the changes in this PR. That said, there are some things I would like to discuss with the team before going forward. Namely, the maintaining of autoconf configuration cache files and the "bashifying" configure.

JavierMatosD · 2023-05-18T21:17:56Z

Hi @dg0yt, I'd appreciate your opinion on the matter. Do you think it's necessary to include the vcpkg-autoconf-cache port for the splitting out of gettext-libintl? If not, I suggest creating a separate pull request specifically for the cache configuration management. Although the vcpkg-autoconf-cache port seems promising, I believe it should be thoroughly spec'd out before merging.

Regarding the "bashifying" of the configure script, I think it's acceptable.

So, unless you strongly believe that both the splitting out of gettext-libintl and vcpkg-autoconf-cache should be merged together, I kindly request you to separate the pull request. :)

Looking forward to your feedback.

dg0yt · 2023-05-19T18:57:38Z

Do you think it's necessary to include the vcpkg-autoconf-cache port for the splitting out of gettext-libintl? If not, I suggest creating a separate pull request specifically for the cache configuration management. Although the vcpkg-autoconf-cache port seems promising, I believe it should be thoroughly spec'd out before merging.

There is no problem with that. It was an important of forming this PR, but in the end it the PR is designed to allow make such cuts feasible.

But there can be some some extra total build time in doing less config caching than before.

dg0yt · 2023-05-20T09:37:11Z

With configuration caching, further tuning, libintl moved to a separate port and gettext-runtime tools moved to another feature: gettext-libintl:x64-windows: Down to 1.7 min. gettext[tools]:x64-windows: Down to 4.3 min.

In total, build time is down from to 10.7 min to 6 min. But there are other benefits:

Back to key benefits but without checked-in config cache.

gettext-libintl:x64-windows: 3.5 min.
gettext[tools]:x64-windows: 6.5 min.
Total: 10 min, slightly less than current master gettext[tools]:x64-windows (11 min), despite one additional pass of configuration.

dg0yt added 6 commits March 25, 2023 11:39

[gettext] Preset more MSVC checks

4fc8aa0

Modify configure to speed up windows builds

537b93b

Support checked-in config cache

cd618d4

Collect config [skip actions]

429b3a6

Add x64 config

195d830

Collect config [skip actions]

ad943e5

MonicaLiu0311 self-assigned this Mar 27, 2023

Re-use most of windows config.cache [skip actions]

03a77d9

dg0yt force-pushed the gettext branch from 0260dce to 03a77d9 Compare March 27, 2023 06:36

dg0yt commented Mar 27, 2023

View reviewed changes

MonicaLiu0311 added the category:port-bug The issue is with a library, which is something the port should already support label Mar 27, 2023

dg0yt mentioned this pull request Mar 31, 2023

[gettext] Build error (conflict with libxml2) #30579

Closed

WIP [skip actions]

6f103cd

dg0yt force-pushed the gettext branch from 10ae584 to 6f103cd Compare April 9, 2023 15:14

dg0yt added 5 commits April 9, 2023 17:44

WIP [skip actions]

a9b600d

WIP [skip actions]

3bbcb2a

WIP

3c53c30

[mchehab-zbar] Update gettext dependencies

8bc0626

Add missing patch

8bdcec3

Monica and others added 2 commits April 11, 2023 13:52

Merge branch 'master' of https://github.com/microsoft/vcpkg into gettext

255c3e1

Don't test find_package(Gettext)

1ed8da8

dg0yt marked this pull request as ready for review April 12, 2023 03:35

dg0yt marked this pull request as draft April 12, 2023 03:38

dg0yt force-pushed the gettext branch 2 times, most recently from 4cb4046 to 6b74ee8 Compare May 1, 2023 07:55

dg0yt force-pushed the gettext branch from 47ea0d4 to 230078e Compare May 1, 2023 14:09

[vcpkg-autotools-cache] Script port for config caching

87bad75

dg0yt force-pushed the gettext branch from 230078e to 87bad75 Compare May 1, 2023 14:23

dg0yt changed the title ~~[gettext] Fixes, faster build~~ [gettext] Split, fixes, faster build, checked-in config cache May 1, 2023

dg0yt marked this pull request as ready for review May 1, 2023 15:15

Add missing file

972672e

MonicaLiu0311 previously approved these changes May 5, 2023

View reviewed changes

MonicaLiu0311 added category:new-port The issue is requesting a new library to be added; consider making a PR! info:reviewed Pull Request changes follow basic guidelines labels May 5, 2023

JavierMatosD reviewed May 16, 2023

View reviewed changes

JavierMatosD added the requires:vcpkg-team-review This PR or issue requires someone on the vcpkg team to take a further look. label May 16, 2023

dg0yt mentioned this pull request May 18, 2023

[gettext] split up the port #25912

Closed

JavierMatosD removed the requires:vcpkg-team-review This PR or issue requires someone on the vcpkg team to take a further look. label May 18, 2023

JavierMatosD marked this pull request as draft May 18, 2023 21:18

dg0yt dismissed MonicaLiu0311’s stale review via ffff02e May 20, 2023 06:35

Remove vcpkg-autoconf-cache

622831a

dg0yt force-pushed the gettext branch from ffff02e to 622831a Compare May 20, 2023 08:05

dg0yt marked this pull request as ready for review May 20, 2023 16:00

JavierMatosD merged commit 5b744ed into microsoft:master May 22, 2023

dg0yt deleted the gettext branch May 22, 2023 03:10

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[gettext] Split, fixes, faster build, checked-in config cache #30429

[gettext] Split, fixes, faster build, checked-in config cache #30429

dg0yt commented Mar 25, 2023 •

edited

Loading

Thomas1664 commented Mar 26, 2023

dg0yt commented Mar 26, 2023

dg0yt Mar 27, 2023

dg0yt Mar 27, 2023

dg0yt Apr 12, 2023

dg0yt commented Mar 27, 2023

dg0yt commented Apr 10, 2023 •

edited

Loading

dg0yt commented Apr 10, 2023

MonicaLiu0311 commented May 5, 2023

JavierMatosD left a comment

JavierMatosD commented May 18, 2023

dg0yt commented May 19, 2023

dg0yt commented May 20, 2023 •

edited

Loading

[gettext] Split, fixes, faster build, checked-in config cache #30429

[gettext] Split, fixes, faster build, checked-in config cache #30429

Conversation

dg0yt commented Mar 25, 2023 • edited Loading

Thomas1664 commented Mar 26, 2023

dg0yt commented Mar 26, 2023

dg0yt Mar 27, 2023

Choose a reason for hiding this comment

dg0yt Mar 27, 2023

Choose a reason for hiding this comment

dg0yt Apr 12, 2023

Choose a reason for hiding this comment

dg0yt commented Mar 27, 2023

dg0yt commented Apr 10, 2023 • edited Loading

dg0yt commented Apr 10, 2023

MonicaLiu0311 commented May 5, 2023

gettext-libintl

gettext

mchehab-zbar

JavierMatosD left a comment

Choose a reason for hiding this comment

JavierMatosD commented May 18, 2023

dg0yt commented May 19, 2023

dg0yt commented May 20, 2023 • edited Loading

dg0yt commented Mar 25, 2023 •

edited

Loading

dg0yt commented Apr 10, 2023 •

edited

Loading

dg0yt commented May 20, 2023 •

edited

Loading