MariaDB container - sudden reliability issues?

[rnorth]

I’ve been noticing an increase in ‘random’ test failures on CI and locally and have been doing some digging: I think for some reason there’s been a sudden drop in reliability of the MariaDB container (or part of our tests around it). I think this has started in roughly the last month, but I'm not sure.

Running the SimpleMariaDBTest class on a loop, I’m seeing approx 50% of tests failing with the following exception:

com.zaxxer.hikari.pool.PoolInitializationException: Exception during pool initialization

	at com.zaxxer.hikari.pool.BaseHikariPool.initializeConnections(BaseHikariPool.java:544)
	at com.zaxxer.hikari.pool.BaseHikariPool.<init>(BaseHikariPool.java:171)
	at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:60)
	at com.zaxxer.hikari.pool.HikariPool.<init>(HikariPool.java:48)
	at com.zaxxer.hikari.HikariDataSource.<init>(HikariDataSource.java:80)
	at org.testcontainers.junit.SimpleMariaDBTest.performQuery(SimpleMariaDBTest.java:79)
	at org.testcontainers.junit.SimpleMariaDBTest.testMariaDBWithCustomIniFile(SimpleMariaDBTest.java:63)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
	at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
	at com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:58)
	at com.intellij.rt.execution.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:242)
	at com.intellij.rt.execution.junit.JUnitStarter.main(JUnitStarter.java:70)
Caused by: java.sql.SQLNonTransientConnectionException: Could not connect to 192.168.99.100:32774: unexpected end of stream, read 0 bytes from 4
	at org.mariadb.jdbc.internal.util.ExceptionMapper.get(ExceptionMapper.java:123)
	at org.mariadb.jdbc.internal.util.ExceptionMapper.throwException(ExceptionMapper.java:69)
	at org.mariadb.jdbc.Driver.connect(Driver.java:110)
	at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:92)
	at com.zaxxer.hikari.util.DriverDataSource.getConnection(DriverDataSource.java:98)
	at com.zaxxer.hikari.pool.BaseHikariPool.addConnection(BaseHikariPool.java:438)
	at com.zaxxer.hikari.pool.BaseHikariPool.initializeConnections(BaseHikariPool.java:542)
	... 28 more

This happens after testcontainers has successfully opened a connection and run a test query through it.

This seems to be isolated to the MariaDB container/tests though - for example, MySQL should be the closest comparison but is getting 100% success.

This is happening with Docker for Mac, Docker Machine and on Travis. I've also tried checking and tweaking wait timeouts on the DB via the connection string, but it makes no difference (the defaults are long timeouts or no timeouts as far as I can see).

I have no idea what's causing this right now, so I'm curious if anybody else is seeing this or has ideas (or a solution!).

Right now I'm afraid I'm considering pulling MariaDB out of the main testcontainers repo/release as a high priority - I can't release a new version containing this in good faith :(

Any thoughts?

[rnorth]

I've extracted MariaDB into its own repo so we can maintain it independently. Closing ticket.