macOS Catalina and Mojave fails due to hardened rules

[klmz]

Since the last release, 9.6.17, i'm no longer able to use embedded-postgres. I get an IllegalStateException saying initDb failed:

27.03.2020 11:37:39.309 [main] WARN  o.s.w.c.s.GenericWebApplicationContext - Exception encountered during context initialization - cancelling refresh attempt: 
org.springframework.beans.factory.BeanCreationException: 
Error creating bean with name 'liquibase' defined in class path resource [org/springframework/boot/autoconfigure/liquibase/LiquibaseAutoConfiguration$LiquibaseConfiguration.class]: 
Bean instantiation via factory method failed; nested exception is org.springframework.beans.BeanInstantiationException: 
Failed to instantiate [liquibase.integration.spring.SpringLiquibase]: 
Factory method 'liquibase' threw exception; nested exception is org.springframework.beans.factory.BeanCreationException: 
Error creating bean with name 'dataSource': 
FactoryBean threw exception on object creation; nested exception is com.google.common.util.concurrent.UncheckedExecutionException: 
java.lang.IllegalStateException: 
Process [/var/folders/3n/7mpl5l6j62v11xttxsjsk4wcfmyktc/T/embedded-pg/PG-5001232d5f8eb9a35a50839624b9a760/bin/initdb, -A, trust, -U, postgres, -D, /var/folders/3n/7mpl5l6j62v11xttxsjsk4wcfmyktc/T/epg17954755376373960693, -E, UTF-8] failed

Some more investigation reveals that initDb is indeed unable to run:

❯ /var/folders/3n/7mpl5l6j62v11xttxsjsk4wcfmyktc/T/embedded-pg/PG-5001232d5f8eb9a35a50839624b9a760/bin/initdb -A trust -U postgres -D /var/folders/3n/7mpl5l6j62v11xttxsjsk4wcfmyktc/T/epg17954755376373960693 -E UTF-8
dyld: Library not loaded: @loader_path/../lib/libz.1.dylib
  Referenced from: /private/var/folders/3n/7mpl5l6j62v11xttxsjsk4wcfmyktc/T/embedded-pg/PG-5001232d5f8eb9a35a50839624b9a760/lib/libxml2.2.dylib
  Reason: no suitable image found.  Did find:
	file system relative paths not allowed in hardened programs
no data was returned by command ""/private/var/folders/3n/7mpl5l6j62v11xttxsjsk4wcfmyktc/T/embedded-pg/PG-5001232d5f8eb9a35a50839624b9a760/bin/postgres" -V"
The program "postgres" is needed by initdb but was not found in the
same directory as "/private/var/folders/3n/7mpl5l6j62v11xttxsjsk4wcfmyktc/T/embedded-pg/PG-5001232d5f8eb9a35a50839624b9a760/bin/initdb".
Check your installation.

It seems like this issue: #17 seems to cause this problem since going back to 9.6.16 fixes the issue.

I'm running on MacOS 10.15.2 and java 11:

openjdk version "11.0.2" 2019-01-15
OpenJDK Runtime Environment 18.9 (build 11.0.2+9)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.2+9, mixed mode)

[tomix26]

Ok, thanks for the report, I'll take a look at it.

[tomix26]

Ok, I already know what is causing the problem.

The problem is in EnterpriseDB binaries. Because the MacOS bundle contains exactly these binaries but only in reduced and repacked form. The previous binaries were not notarized and was running in legacy mode, so it was working fine. Whereas the new version of binaries are running in the hardened runtime which was introduced in macOS 10.14 Mojave. Apps running in the hardened runtime have some restrictons. One of them relates to using dynamically linked libraries. For using them must be enabled runtime exception to allow DYLD environment variables and maybe some other exceptions.

Unfortunately, at this moment the EnterpriseDB binaries do not have this exception enabled, so it does not work with the latest MacOS versions (Mojave and Catalina). We can only hope that people from EnterpriseDB will fix this issue and the next versions will be working again.

[chrisjenx]

Arg, so we can't run these on OSx 😢

[chrisjenx]

Downgrading to 12.1 fixed this for me thankfully 😥

[tomix26]

@chrisjenx Thanks for the confirmation.

[phiros]

Any progress on this?

[chrisjenx]

I dropped embedded postgres db, we now run a local test db and our ci (GitHub Actions) runs a postgres server. Not only is this much much faster ~2s per test down to ~200ms (including dropping the pragma and recreating it for each test). It's a little closer to how your application would probably be talking db, we found some interesting connection closed issues by moving to a single instance which was masked by dropping and creating a new db on each test.

[tomix26]

@phiros Nope, the latest versions have the same problem.

[tomix26]

@chrisjenx Good for you. However, database cloning should not have such an impact on performance. Creating a new database on my machine takes only a few milliseconds, depending on the size of the database.

[phiros]

BTW for people thinking they are affected by exactly this bug:
Make sure you checked your locale settings as well! This does not seem important but it is (postgres is quite picky about some environment settings). See for more details: https://stackoverflow.com/questions/7165108/in-os-x-lion-lang-is-not-set-to-utf-8-how-to-fix-it/8161863#8161863

[chrisjenx]

@chrisjenx Good for you. However, database cloning should not have such an impact on performance. Creating a new database on my machine takes only a few milliseconds, depending on the size of the database.

Yeah not sure, it would of thought so too, probably an osx issue? (My unix machine runs the same stack about 2-3x faster, so probably some crappy bottleneck in osx)

[gaddesai]

BTW for people thinking they are affected by exactly this bug:
Make sure you checked your locale settings as well! This does not seem important but it is (postgres is quite picky about some environment settings). See for more details: https://stackoverflow.com/questions/7165108/in-os-x-lion-lang-is-not-set-to-utf-8-how-to-fix-it/8161863#8161863

This fixed for me

[trecloux]

@tomix26 Do you think that the repackaging changes you applied in #30 might have fix this issue ?

[tomix26]

@trecloux Nope, still the same problem.

[tomix26]

I did some more digging and it seems that the primary problem was a missing libz library. I included the library into the bundle and now everything works fine.

[SButterfly]

Hi @tomix26
I reproduced the problem with dyld: Library not loaded: @loader_path/../lib/libz.1.dylib in 12.2.0 version.
In what version did you fix it?

[tomix26]

The fixed versions were not officially released yet, I've released them right now. But since already published versions cannot be overridden in maven central repository, I had to release special versions with -1 suffix. So fixed versions are 9.5.24-1, 9.6.20-1, 10.15.0-1, 11.10.0-1, 12.5.0-1, 13.1.0-1. See the releases page for more information.

[ilatypov]

Ok, I already know what is causing the problem.

The problem is in EnterpriseDB binaries. Because the MacOS bundle contains exactly these binaries but only in reduced and repacked form. The previous binaries were not notarized and was running in legacy mode, so it was working fine. Whereas the new version of binaries are running in the hardened runtime which was introduced in macOS 10.14 Mojave. Apps running in the hardened runtime have some restrictons. One of them relates to using dynamically linked libraries. For using them must be enabled runtime exception to allow DYLD environment variables and maybe some other exceptions.

Unfortunately, at this moment the EnterpriseDB binaries do not have this exception enabled, so it does not work with the latest MacOS versions (Mojave and Catalina). We can only hope that people from EnterpriseDB will fix this issue and the next versions will be working again.

I figured a similar but probably unrelated problem causes my initdb execution failure. I am using docker/toolbox that communicates with an in-memory Linux and Docker daemon running in a VirtualBox VM. The Linux kernel mounts my laptop's /Users filesystem, and that mounting is limited to providing "a minimal file system to allow exchange of files between the host and guest",

https://forums.virtualbox.org/viewtopic.php?f=3&t=91596

The following permission error precedes my initdb execution failure (after using strace -F -s 1024).

[pid   129] openat(AT_FDCWD, "pg_wal/xlogtemp.129", O_RDWR) = 5
[pid   129] close(5)                    = 0
[pid   129] link("pg_wal/xlogtemp.129", "pg_wal/000000010000000000000001") = -1 EPERM (Operation not permitted)
[pid   129] write(2, "2021-09-27 14:30:13.835 UTC [129] LOG:  could not link file \"pg_wal/xlogtemp.129\" to \"pg_wal/000000010000000000000001\": Ope
) = 144
[pid   129] unlink("pg_wal/xlogtemp.129") = 0
[pid   129] openat(AT_FDCWD, "pg_wal/000000010000000000000001", O_RDWR) = -1 ENOENT (No such file or directory)
[pid   129] write(2, "2021-09-27 14:30:13.841 UTC [129] FATAL:  could not open file \"pg_wal/000000010000000000000001\": No such file or directory\n"
) = 123

The output of my failed initdb invokation is as follows (running as user gradle),

$ /Users/FIRST.LAST/api/temp/embedded-pg/PG-e87e3425862187f1dd1df32927495e17/bin/initdb -A trust -U postgres -D /Users/FIRST.LAST/api/temp/epg5944749101546369145 -E UTF-8
The files belonging to this database system will be owned by user "gradle".
This user must also own the server process.

The database cluster will be initialized with locale "C".
The default text search configuration will be set to "english".

Data page checksums are disabled.

fixing permissions on existing directory /Users/FIRST.LAST/api/temp/epg5944749101546369145 ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... UTC
creating configuration files ... ok
running bootstrap script ... 2021-09-27 14:36:50.465 UTC [138] LOG:  could not link file "pg_wal/xlogtemp.138" to "pg_wal/000000010000000000000001": Operation not permitted
2021-09-27 14:36:50.467 UTC [138] FATAL:  could not open file "pg_wal/000000010000000000000001": No such file or directory
child process exited with exit code 1
initdb: removing contents of data directory "/Users/FIRST.LAST/api/temp/epg5944749101546369145"

The cause is different from the hardened runtime issue uncovered by @tomix26 .

As a work-around, I used an unnamed volume in /tmp for temporary files. It has to be /tmp as Postgres ignores ${TMPDIR} creating a socket in /tmp. The volume appears as a Linux native filesystem to containers. I did not use --tmpfs /tmp:exec because that would take half of the memory.

ENV TMPDIR=/tmp
ENV JAVA_TOOL_OPTIONS=-Djava.io.tmpdir=/tmp
VOLUME /tmp

[smarvasti-bc]

All versions of zonky embedded postgres still fail for me. MacOS 12.2 M1 Pro

[thedevd]

For me ru.yandex.qatools.embed.postgresql started failing after I upgraded Mac os from 11 to 12.

[kovacshuni]

Still a problem. I'm using testcontainers-scala instead (for some testcontainers-java may be more helpful). Maybe it's even better like this. Still the problem exists.

[ssoleimanii]

Same problem here with MacOS Monetary (12.3.1) M1!

[tainguyen210]

Same problem here with MacOS Monterey (12.3) M1

[quintencls]

Same problem here with MacOS 12.5 on a MacBook Pro 16-inch 2019

[quintencls]

FYI Using these dependencies is working for me:

testImplementation('io.zonky.test:embedded-database-spring-test:2.1.1')
testImplementation('io.zonky.test:embedded-postgres:2.0.0')

I was using only embedded-database-spring-test 1.6.3 before:

testImplementation('io.zonky.test:embedded-database-spring-test:1.6.3')

After upgrading I just needed to set the provider and type for the embedded db:

@AutoConfigureEmbeddedDatabase(
        provider = AutoConfigureEmbeddedDatabase.DatabaseProvider.ZONKY,
        type = AutoConfigureEmbeddedDatabase.DatabaseType.POSTGRES
)