Enable all platforms again #976

mathisloge · 2021-07-02T15:31:25Z

So this PR adds a way to build for all platforms that are supported by mapnik (including windows)
Currently the whole dependency setup is done via vcpkg. I recommend this for windows, because, as #848 stated, it would take a lot of effort to pull in all the dependencies with a script.
On linux and osx the system libraries should be used. This splitting is generally easy. On windows the submodule would be cloned or, as it is also possibile, added via FetchContent. Otherwise mapnik would be found via pkg-config or cmakes find_package.

~~However since this will use cmake as a build system i had to remove node-gyp. I think it will need discussion if you want to pursude this path.~~
This will use cmake-js as the build system and node-pre-gyp for packaging and publishing.

Depends on:

CMake build support mapnik#4191
[mapbox-wagyu] add new port microsoft/vcpkg#18771
[Mapnik] new port microsoft/vcpkg#18849
~~currently there is a known issue that happens when installing mapnik in manifest mode. The proj port will inject a invalid include path which get carried into mapniktargets.cmake.~~ RESOLVED

if someone wants to try this PR, the file build\vcpkg_installed\<your_triplet>\share\mapnik\MapnikTargets.cmake needs a patch at line 57 with [...]vcpkg_installed/include to [...]vcpkg_installed/<your_triplet>/include. You will need to build manually with cmake-js build--CDCMAKE_TOOLCHAIN_FILE="./vcpkg/scripts/buildsystems/vcpkg.cmake" --target node-mapnik and after the first build file, apply the described workaround. RESOLVED

Todo:

create tarball with plugins etc.
auto generate mapnik_settings.json with the correct values
as soon as all PRs are merged (vcpkg related) switch vcpkg submodule to microsoft/vcpkg
~~maybe there is a way to use node-pre-gyp in conjunction with cmake-js( build with cmake, publish and package with node-pre-gyp)~~ so building with cmake-js and package & publish with node-pre-gyp works pretty well
ogr and gdal currently don't work. Need to investigate why (only tested on windows). There might be a missing hidden dependency. ~~could narrow down the problem: somehow the libpq could not be imported.~~ was just a missing PATH entry
windows unicode need some reworks. (think this should be fixed when the CI changes are getting merged into mapnik@master)

prebuild binaries

For development and testing purposes, there is a devbuild npm package:
https://www.npmjs.com/package/@mathisloge/mapnik

Currently there should be published packages for windows and linux.

integration tests

Simple integration test with windows and linux each with node versions 12, 14 and 16
https://github.com/mathisloge/node-mapnik-test

mathisloge · 2021-07-02T15:34:10Z

I will wait for some feedback before putting more effort into it.

mathisloge · 2021-07-02T18:39:31Z

mainly seeking some feedback from you @artemp :D

This reverts commit e29ac46.

mathisloge · 2022-01-22T10:32:16Z

@GimpMaster So I've added github publishing for the build process.
There should be windows and linux prebuild packages for https://www.npmjs.com/package/@mathisloge/mapnik

3nprob · 2022-06-03T02:04:36Z

Dockerfile

@@ -0,0 +1,9 @@
+FROM ghcr.io/mathisloge/mapnik:docker


Is there a Dockerfile for this base image?

Looks like it's build from this branch mapnik/mapnik#4267

3nprob · 2022-06-03T02:19:32Z

This image does not build for me (linux, aarch64, built base image from mapnik/mapnik@7631f93):

Step 6/7 : RUN npm install && rm -rf build*
 ---> Running in 700a0d4f20f4
npm WARN deprecated [email protected]: The querystring API is considered Legacy. new code should use the URLSearchParams API instead.
npm WARN deprecated [email protected]: Please upgrade  to version 7 or higher.  Older versions may use Math.random() in certain circumstances, which is known to be problematic.  See https://v8.dev/blog/math-random for details.

> @mathisloge/[email protected] install
> node-pre-gyp install --fallback-to-build

node-pre-gyp info it worked if it ends with ok
node-pre-gyp info using [email protected]
node-pre-gyp info using [email protected] | linux | arm64
node-pre-gyp info check checked for "/nodemapnik/lib/binding/napi-v3/mapnik.node" (not found)
node-pre-gyp http GET https://github.com/mathisloge/node-mapnik/releases/download/v5.0.0-devbuild10/linux-arm64-napi-v3.tar.gz
node-pre-gyp ERR! install response status 404 Not Found on https://github.com/mathisloge/node-mapnik/releases/download/v5.0.0-devbuild10/linux-arm64-napi-v3.tar.gz
node-pre-gyp WARN Pre-built binaries not installable for @mathisloge/[email protected] and [email protected] (node-v72 ABI, glibc) (falling back to source compile with node-gyp)
node-pre-gyp WARN Hit error response status 404 Not Found on https://github.com/mathisloge/node-mapnik/releases/download/v5.0.0-devbuild10/linux-arm64-napi-v3.tar.gz
gyp info it worked if it ends with ok
gyp info using [email protected]
gyp info using [email protected] | linux | arm64
gyp info ok
gyp info it worked if it ends with ok
gyp info using [email protected]
gyp info using [email protected] | linux | arm64
gyp info find Python using Python version 3.9.5 found at "/usr/bin/python3"
gyp info spawn /usr/bin/python3
gyp info spawn args [
gyp info spawn args   '/usr/share/nodejs/node-gyp/gyp/gyp_main.py',
gyp info spawn args   'binding.gyp',
gyp info spawn args   '-f',
gyp info spawn args   'make',
gyp info spawn args   '-I',
gyp info spawn args   '/nodemapnik/build/config.gypi',
gyp info spawn args   '-I',
gyp info spawn args   '/usr/share/nodejs/node-gyp/addon.gypi',
gyp info spawn args   '-I',
gyp info spawn args   '/usr/include/nodejs/common.gypi',
gyp info spawn args   '-Dlibrary=shared_library',
gyp info spawn args   '-Dvisibility=default',
gyp info spawn args   '-Dnode_root_dir=/usr/include/nodejs',
gyp info spawn args   '-Dnode_gyp_dir=/usr/share/nodejs/node-gyp',
gyp info spawn args   '-Dnode_lib_file=/usr/include/nodejs/<(target_arch)/node.lib',
gyp info spawn args   '-Dmodule_root_dir=/nodemapnik',
gyp info spawn args   '-Dnode_engine=v8',
gyp info spawn args   '--depth=.',
gyp info spawn args   '--no-parallel',
gyp info spawn args   '--generator-output',
gyp info spawn args   'build',
gyp info spawn args   '-Goutput_dir=.'
gyp info spawn args ]
gyp info ok
gyp info it worked if it ends with ok
gyp info using [email protected]
gyp info using [email protected] | linux | arm64
gyp info spawn make
gyp info spawn args [ 'BUILDTYPE=Release', '-C', 'build' ]
make: Entering directory '/nodemapnik/build'
  ACTION configuring node-mapnik... build/CMakeCache.txt
CMake Error at /usr/share/cmake-3.23/Modules/CMakeDetermineSystem.cmake:130 (message):
  Could not find toolchain file:
  /nodemapnik/vcpkg/scripts/buildsystems/vcpkg.cmake
Call Stack (most recent call first):
  CMakeLists.txt:3 (project)


CMake Error: CMake was unable to find a build program corresponding to "Unix Makefiles".  CMAKE_MAKE_PROGRAM is not set.  You probably need to select a different build tool.
CMake Error: CMAKE_C_COMPILER not set, after EnableLanguage
CMake Error: CMAKE_CXX_COMPILER not set, after EnableLanguage
-- Configuring incomplete, errors occurred!
make: *** [build-node-mapnik.target.mk:13: build/CMakeCache.txt] Error 1
make: Leaving directory '/nodemapnik/build'
gyp ERR! build error
gyp ERR! stack Error: `make` failed with exit code: 2
gyp ERR! stack     at ChildProcess.onExit (/usr/share/nodejs/node-gyp/lib/build.js:194:23)
gyp ERR! stack     at ChildProcess.emit (events.js:314:20)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:276:12)
gyp ERR! System Linux 5.18.0-4-MANJARO-ARM
gyp ERR! command "/usr/bin/node" "/usr/share/nodejs/node-gyp/bin/node-gyp.js" "build" "--fallback-to-build" "--module=/nodemapnik/lib/binding/napi-v3/mapnik.node" "--module_name=mapnik" "--module_path=/nodemapnik/lib/binding/napi-v3" "--napi_version=7" "--
node_abi_napi=napi" "--napi_build_version=3" "--node_napi_label=napi-v3"
gyp ERR! cwd /nodemapnik
gyp ERR! node -v v12.21.0
gyp ERR! node-gyp -v v7.1.2
gyp ERR! not ok
node-pre-gyp ERR! build error
node-pre-gyp ERR! stack Error: Failed to execute '/usr/bin/node /usr/share/nodejs/node-gyp/bin/node-gyp.js build --fallback-to-build --module=/nodemapnik/lib/binding/napi-v3/mapnik.node --module_name=mapnik --module_path=/nodemapnik/lib/binding/napi-v3 --n
api_version=7 --node_abi_napi=napi --napi_build_version=3 --node_napi_label=napi-v3' (1)
node-pre-gyp ERR! stack     at ChildProcess.<anonymous> (/nodemapnik/node_modules/@mapbox/node-pre-gyp/lib/util/compile.js:89:23)
node-pre-gyp ERR! stack     at ChildProcess.emit (events.js:314:20)
node-pre-gyp ERR! stack     at maybeClose (internal/child_process.js:1022:16)
node-pre-gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:287:5)
node-pre-gyp ERR! System Linux 5.18.0-4-MANJARO-ARM
node-pre-gyp ERR! command "/usr/bin/node" "/nodemapnik/node_modules/.bin/node-pre-gyp" "install" "--fallback-to-build"
node-pre-gyp ERR! cwd /nodemapnik
node-pre-gyp ERR! node -v v12.21.0
node-pre-gyp ERR! node-pre-gyp -v v1.0.9
node-pre-gyp ERR! not ok
Failed to execute '/usr/bin/node /usr/share/nodejs/node-gyp/bin/node-gyp.js build --fallback-to-build --module=/nodemapnik/lib/binding/napi-v3/mapnik.node --module_name=mapnik --module_path=/nodemapnik/lib/binding/napi-v3 --napi_version=7 --node_abi_napi=n
api --napi_build_version=3 --node_napi_label=napi-v3' (1)
npm notice
npm notice New major version of npm available! 7.5.2 -> 8.12.1
npm notice Changelog: <https://github.com/npm/cli/releases/tag/v8.12.1>
npm notice Run `npm install -g [email protected]` to update!
npm notice
npm ERR! code 1
npm ERR! path /nodemapnik
npm ERR! command failed
npm ERR! command sh -c node-pre-gyp install --fallback-to-build

npm ERR! A complete log of this run can be found in:
npm ERR!     /root/.npm/_logs/2022-06-03T02_14_10_285Z-debug.log

Pre-built binaries should not be required when building from source.

3nprob · 2022-06-03T02:51:09Z

Trying to manually patch it up by adding '-DCMAKE_C_COMPILER=gcc', '-DCMAKE_CXX_COMPILER=g++' to the actions in binding.gyp and removing the CMAKE_TOOLCHAIN_FILE declaration, we get some further progress but it stops soon after at:

-- Performing Test CMAKE_HAVE_LIBC_PTHREAD - Failed                                                                                                                                                                                                   [109/1905]
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- Found PkgConfig: /usr/bin/pkg-config (found version "0.29.2")
-- Checking for module 'cairo'
--   Found cairo, version 1.16.0
-- Checking for module 'proj'
--   Found proj, version 7.2.1
-- Found Protobuf: /usr/lib/aarch64-linux-gnu/libprotobuf.so;-lpthread (found version "3.12.4")
-- Found the following ICU libraries:
--   uc (required)
--   i18n (required)
--   data (required)
-- Found ICU: /usr/include (found version "67.1")
-- Found Boost: /usr/lib/aarch64-linux-gnu/cmake/Boost-1.74.0/BoostConfig.cmake (found suitable version "1.74.0", minimum required is "1.73") found components: filesystem system regex
-- Found Freetype: /usr/lib/aarch64-linux-gnu/libfreetype.so (found version "2.10.4")
-- Found LibXml2: /usr/lib/aarch64-linux-gnu/libxml2.so (found version "2.9.10")
-- Found ZLIB: /usr/lib/aarch64-linux-gnu/libz.so (found version "1.2.11")
-- Found PNG: /usr/lib/aarch64-linux-gnu/libpng.so (found version "1.6.37")
-- Found JPEG: /usr/lib/aarch64-linux-gnu/libjpeg.so (found version "80")
-- Found TIFF: /usr/lib/aarch64-linux-gnu/libtiff.so (found version "4.2.0")
-- Found WebP: /usr/lib/aarch64-linux-gnu/libwebp.so (found version "0.6.1")
-- Found the following ICU libraries:
--   uc (required)
--   i18n (required)
--   data (required)
-- Found ICU: /usr/include (found version "67.1")
CMake Error at /usr/lib/aarch64-linux-gnu/cmake/harfbuzz/harfbuzz-config.cmake:49 (add_library):
  add_library cannot create imported target "harfbuzz::harfbuzz" because
  another target with the same name already exists.
Call Stack (most recent call first):
  /usr/share/cmake-3.23/Modules/CMakeFindDependencyMacro.cmake:47 (find_package)
  /usr/local/lib/cmake/mapnik/mapnikConfig.cmake:28 (find_dependency)
  build/_deps/mapnik_vector_tile-src/CMakeLists.txt:24 (find_package)


CMake Error at /usr/lib/aarch64-linux-gnu/cmake/harfbuzz/harfbuzz-config.cmake:54 (add_library):
  add_library cannot create imported target "harfbuzz::icu" because another
  target with the same name already exists.
Call Stack (most recent call first):
  /usr/share/cmake-3.23/Modules/CMakeFindDependencyMacro.cmake:47 (find_package)
  /usr/local/lib/cmake/mapnik/mapnikConfig.cmake:28 (find_dependency)
  build/_deps/mapnik_vector_tile-src/CMakeLists.txt:24 (find_package)


CMake Error at /usr/lib/aarch64-linux-gnu/cmake/harfbuzz/harfbuzz-config.cmake:60 (add_library):
  add_library cannot create imported target "harfbuzz::subset" because
  another target with the same name already exists.
Call Stack (most recent call first):
  /usr/share/cmake-3.23/Modules/CMakeFindDependencyMacro.cmake:47 (find_package)
  /usr/local/lib/cmake/mapnik/mapnikConfig.cmake:28 (find_dependency)
  build/_deps/mapnik_vector_tile-src/CMakeLists.txt:24 (find_package)


CMake Error at /usr/lib/aarch64-linux-gnu/cmake/harfbuzz/harfbuzz-config.cmake:69 (add_library):
  add_library cannot create imported target "harfbuzz::gobject" because
  another target with the same name already exists.
Call Stack (most recent call first):
  /usr/share/cmake-3.23/Modules/CMakeFindDependencyMacro.cmake:47 (find_package)
  /usr/local/lib/cmake/mapnik/mapnikConfig.cmake:28 (find_dependency)
  build/_deps/mapnik_vector_tile-src/CMakeLists.txt:24 (find_package)

Same result even if I remove the one mention of harfbuzz in this branch in scripts/install_deps.sh

I'm not a cmake pro and not sure at all how to continue from here. Currently attempting from 3nprob@6ba144f (https://github.com/mathisloge/node-mapnik/compare/master...3nprob:linux-docker-build?expand=1) and I can't see where harfbuzz would enter in the first,let alone second, place.

mathisloge · 2022-06-03T21:02:47Z

Hey, I'm currently not on my PC so I can't help you that much.

You can try to set -DCMAKE_DISABLE_FIND_PACKAGE_harfbuzz=OFF to force pkgconf to find harfbuzz when configuring mapnik. Unfortunately harfbuzz's own provided cmake config files are seriously broken on some distributions.

3nprob · 2022-06-04T03:24:02Z

You can try to set -DCMAKE_DISABLE_FIND_PACKAGE_harfbuzz=OFF to force pkgconf to find harfbuzz when configuring mapnik. Unfortunately harfbuzz's own provided cmake config files are seriously broken on some distributions.

Hmm, I tried setting that both here and here with same outcome. Would it go somewhere else?

wxitguy · 2022-06-22T18:27:33Z

Hello,

I was able to successfully build the latest devbuild10 Mapnik on my Windows 10 system.

My issue is that I am trying to build Mapnik to use in an Electron app for Windows. When I get to the point that I can successfully build Mapnik from source, I then run it through electron-rebuild it does appear to rebuild successfully but when starting my program I get "a dynamic link library (DLL) initialization routine failed" errors every time.

My electron version is 14.2.9 but I've tried several different versions of electron up to 18 with the same result.

I'm not sure what is missing or needs to be done at this point, any thoughts/ideas?

mathisloge · 2022-06-22T20:16:31Z

@3nprob oh sorry. You actually need to set DCMAKE_DISABLE_FIND_PACKAGE_harfbuzz to ON my fault.

@wxitguy glad to hear that you were successful compiling from source. IIRC you need to compile a different napi lib for electron.
I'm currently using https://github.com/mathisloge/cmake-napi but it handles only the real node api bindings and not the one for electron. As I'm currently fairly busy with other things in my live, I won't be able to look into it before august. But I would be really happy, if you could report any progress on your site.

artemp and others added 10 commits February 10, 2021 11:49

proj6 - WIP

7e93421

Merge branch 'master' into proj6

f308203

Update test data to use proj6 format

0f0dcbc

add vcpkg

12b2adc

getting started

3d685ee

enable compilation

0e7bcc1

first successful build

cce4830

remove deps replaced by vcpkg

4041d9d

remove lock

4c3124e

remove wrong var

9a84454

mathisloge mentioned this pull request Jul 2, 2021

[MAPNIK] new port added microsoft/vcpkg#11656

Closed

mathisloge added 17 commits July 2, 2021 23:31

update deps

31bd956

add some deployment

1855a94

generate mapnik_settings.js

27ae60b

correct settings paths

6ba2b19

fix prebuild

9bf9199

add publish binary

88b0a2a

add some debug output

13feaa8

add proj copy

869fdf6

copy dlls on windows

76c9062

copy dlls from executables

0875547

remove not needed script

cae1ae2

update vcpkg

ca07cf6

fix cmake

1a2ecdf

some fixes

c055b4e

only use vcpkg on windows

f2d7231

update vcpkg

c510aed

update cmake-js

fda4399

mathisloge added 10 commits January 21, 2022 11:38

collect logs

ca23f42

remove duplicate input-geojson in manifest

029a626

readd vcpkg for development

e29ac46

Revert "readd vcpkg for development"

815cbd9

This reverts commit e29ac46.

test vcpkg gdal fix

f360272

run only windows for debugging

ff62aeb

windows has not enough disk space. disable

4cec3f3

bump version

9762ed5

bump version

6b8b28b

fix publish script

a937665

mathisloge added 9 commits January 22, 2022 11:41

fix publish

c444664

remove publish

ba62427

unix improvements

d1100be

windows fixes

f5fa7a2

wrong toolchain

e6a8021

use vcpkg

8e0300b

remove rpath

9366943

ubuntu install sys deps

641f560

install make

2e63b33

mathisloge mentioned this pull request Mar 14, 2022

Road to v4 mapnik/mapnik#4299

Open

6 tasks

3nprob reviewed Jun 3, 2022

View reviewed changes

prusswan mentioned this pull request Jun 25, 2022

In search of the OG windows exe version! tilemill-project/tilemill#2755

Closed

mathisloge mentioned this pull request Aug 22, 2022

Using cmake-js instead of node-gyp #985

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enable all platforms again #976

Enable all platforms again #976

mathisloge commented Jul 2, 2021 •

edited

Loading

mathisloge commented Jul 2, 2021

mathisloge commented Jul 2, 2021

mathisloge commented Jan 22, 2022

3nprob Jun 3, 2022

3nprob Jun 3, 2022

3nprob commented Jun 3, 2022 •

edited

Loading

3nprob commented Jun 3, 2022 •

edited

Loading

mathisloge commented Jun 3, 2022 •

edited

Loading

3nprob commented Jun 4, 2022

wxitguy commented Jun 22, 2022

mathisloge commented Jun 22, 2022

Enable all platforms again #976

Are you sure you want to change the base?

Enable all platforms again #976

Conversation

mathisloge commented Jul 2, 2021 • edited Loading

Todo:

prebuild binaries

integration tests

mathisloge commented Jul 2, 2021

mathisloge commented Jul 2, 2021

mathisloge commented Jan 22, 2022

3nprob Jun 3, 2022

Choose a reason for hiding this comment

3nprob Jun 3, 2022

Choose a reason for hiding this comment

3nprob commented Jun 3, 2022 • edited Loading

3nprob commented Jun 3, 2022 • edited Loading

mathisloge commented Jun 3, 2022 • edited Loading

3nprob commented Jun 4, 2022

wxitguy commented Jun 22, 2022

mathisloge commented Jun 22, 2022

mathisloge commented Jul 2, 2021 •

edited

Loading

3nprob commented Jun 3, 2022 •

edited

Loading

3nprob commented Jun 3, 2022 •

edited

Loading

mathisloge commented Jun 3, 2022 •

edited

Loading