roachtest: hibernate failed #40538

cockroach-teamcity · 2019-09-06T10:47:40Z

SHA: https://github.com/cockroachdb/cockroach/commits/201b400c5a59d42d436f417a284a129cff3ed7b3

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=hibernate PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1471316&tab=buildLog

The test failed on branch=release-19.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20190906-1471316/hibernate/run_1
	hibernate.go:321,hibernate.go:332,test_runner.go:688: 
		Tests run on Cockroach v19.1.4-14-g201b400
		Tests run against Hibernate 5.4.4
		9527 Total Tests Run
		9175 tests passed
		352 tests failed
		51 tests passed unexpectedly
		0 tests failed unexpectedly
		35 tests expected failed, but not run
		For a full summary look at the hibernate artifacts 
		
		An updated blacklist (hibernateBlackList19_1) is available in the artifacts' hibernate log

The text was updated successfully, but these errors were encountered:

jordanlewis · 2019-09-06T14:21:42Z

  51 tests passed unexpectedly

?

cockroach-teamcity · 2019-09-07T01:08:33Z

SHA: https://github.com/cockroachdb/cockroach/commits/4784fe3c51545db5fb5d411937ec1db2ef2b9761

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=hibernate PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1472753&tab=buildLog

The test failed on branch=provisional_201909060000_v19.2.0-beta.20190910, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20190906-1472753/hibernate/run_1
	hibernate.go:321,hibernate.go:332,test_runner.go:688: 
		Tests run on Cockroach v19.2.0-alpha.20190606-2082-g4784fe3
		Tests run against Hibernate 5.4.4
		9310 Total Tests Run
		8892 tests passed
		418 tests failed
		0 tests passed unexpectedly
		157 tests failed unexpectedly
		26 tests expected failed, but not run
		For a full summary look at the hibernate artifacts 
		
		An updated blacklist (hibernateBlackList19_2) is available in the artifacts' hibernate log

cockroach-teamcity · 2019-09-07T13:05:33Z

SHA: https://github.com/cockroachdb/cockroach/commits/4e1deb21a3e28387c45e6a87a3cefe6bb49f39ac

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=hibernate PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1473326&tab=buildLog

The test failed on branch=provisional_201909070026_v19.2.0-beta.20190911, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20190907-1473326/hibernate/run_1
	hibernate.go:321,hibernate.go:332,test_runner.go:688: 
		Tests run on Cockroach v19.2.0-alpha.20190606-2110-g4e1deb2
		Tests run against Hibernate 5.4.4
		9166 Total Tests Run
		8719 tests passed
		447 tests failed
		0 tests passed unexpectedly
		194 tests failed unexpectedly
		34 tests expected failed, but not run
		For a full summary look at the hibernate artifacts 
		
		An updated blacklist (hibernateBlackList19_2) is available in the artifacts' hibernate log

rafiss · 2019-09-09T15:02:56Z

51 tests passed unexpectedly

I guess they flaked last time, which was the run I used to create the current blacklist. I will continue refining this! But if these keep passing/failing unreliably then the noise won't stop.

cockroach-teamcity · 2019-09-09T16:06:14Z

SHA: https://github.com/cockroachdb/cockroach/commits/47bb2a58c87fc1259291ec9dde78de3e54bd8a3d

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=hibernate PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1475396&tab=buildLog

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20190909-1475396/hibernate/run_1
	hibernate.go:321,hibernate.go:332,test_runner.go:688: 
		Tests run on Cockroach v19.2.0-alpha.20190606-2139-g47bb2a5
		Tests run against Hibernate 5.4.4
		8929 Total Tests Run
		8229 tests passed
		700 tests failed
		0 tests passed unexpectedly
		461 tests failed unexpectedly
		48 tests expected failed, but not run
		For a full summary look at the hibernate artifacts 
		
		An updated blacklist (hibernateBlackList19_2) is available in the artifacts' hibernate log

rafiss · 2019-09-09T19:19:16Z

Regarding the 19.2 failures posted here, it seems like ALL of them are caused by the test sometimes being unable to connect to the database, during test setup. See the stack trace below. When I rerun these tests locally, they pass.

org.hibernate.service.spi.ServiceException: Unable to create requested service [org.hibernate.engine.jdbc.env.spi.JdbcEnvironment]
	at org.hibernate.service.internal.AbstractServiceRegistryImpl.createService(AbstractServiceRegistryImpl.java:275)
	at org.hibernate.service.internal.AbstractServiceRegistryImpl.initializeService(AbstractServiceRegistryImpl.java:237)
	at org.hibernate.service.internal.AbstractServiceRegistryImpl.getService(AbstractServiceRegistryImpl.java:214)
	at org.hibernate.id.factory.internal.DefaultIdentifierGeneratorFactory.injectServices(DefaultIdentifierGeneratorFactory.java:152)
	at org.hibernate.service.internal.AbstractServiceRegistryImpl.injectDependencies(AbstractServiceRegistryImpl.java:286)
	at org.hibernate.service.internal.AbstractServiceRegistryImpl.initializeService(AbstractServiceRegistryImpl.java:243)
	at org.hibernate.service.internal.AbstractServiceRegistryImpl.getService(AbstractServiceRegistryImpl.java:214)
	at org.hibernate.boot.internal.InFlightMetadataCollectorImpl.<init>(InFlightMetadataCollectorImpl.java:175)
	at org.hibernate.boot.model.process.spi.MetadataBuildingProcess.complete(MetadataBuildingProcess.java:118)
	at org.hibernate.jpa.boot.internal.EntityManagerFactoryBuilderImpl.metadata(EntityManagerFactoryBuilderImpl.java:900)
	at org.hibernate.jpa.boot.internal.EntityManagerFactoryBuilderImpl.build(EntityManagerFactoryBuilderImpl.java:931)
	at org.hibernate.jpa.test.BaseEntityManagerFunctionalTestCase.buildEntityManagerFactory(BaseEntityManagerFunctionalTestCase.java:75)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
	at org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
	at org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.lang.Thread.run(Thread.java:748)
Caused by: org.hibernate.exception.JDBCConnectionException: Error calling Driver#connect
	at org.hibernate.exception.internal.SQLStateConversionDelegate.convert(SQLStateConversionDelegate.java:112)
	at org.hibernate.exception.internal.StandardSQLExceptionConverter.convert(StandardSQLExceptionConverter.java:42)
	at org.hibernate.engine.jdbc.spi.SqlExceptionHelper.convert(SqlExceptionHelper.java:113)
	at org.hibernate.engine.jdbc.connections.internal.BasicConnectionCreator.convertSqlException(BasicConnectionCreator.java:118)
	at org.hibernate.engine.jdbc.connections.internal.DriverConnectionCreator.makeConnection(DriverConnectionCreator.java:41)
	at org.hibernate.engine.jdbc.connections.internal.BasicConnectionCreator.createConnection(BasicConnectionCreator.java:58)
	at org.hibernate.engine.jdbc.connections.internal.DriverManagerConnectionProviderImpl$PooledConnections.addConnections(DriverManagerConnectionProviderImpl.java:321)
	at org.hibernate.engine.jdbc.connections.internal.DriverManagerConnectionProviderImpl$PooledConnections.<init>(DriverManagerConnectionProviderImpl.java:240)
	at org.hibernate.engine.jdbc.connections.internal.DriverManagerConnectionProviderImpl$PooledConnections.<init>(DriverManagerConnectionProviderImpl.java:218)
	at org.hibernate.engine.jdbc.connections.internal.DriverManagerConnectionProviderImpl$PooledConnections$Builder.build(DriverManagerConnectionProviderImpl.java:359)
	at org.hibernate.engine.jdbc.connections.internal.DriverManagerConnectionProviderImpl.buildPool(DriverManagerConnectionProviderImpl.java:98)
	at org.hibernate.engine.jdbc.connections.internal.DriverManagerConnectionProviderImpl.configure(DriverManagerConnectionProviderImpl.java:73)
	at org.hibernate.boot.registry.internal.StandardServiceRegistryImpl.configureService(StandardServiceRegistryImpl.java:107)
	at org.hibernate.service.internal.AbstractServiceRegistryImpl.initializeService(AbstractServiceRegistryImpl.java:246)
	at org.hibernate.service.internal.AbstractServiceRegistryImpl.getService(AbstractServiceRegistryImpl.java:214)
	at org.hibernate.engine.jdbc.env.internal.JdbcEnvironmentInitiator.buildJdbcConnectionAccess(JdbcEnvironmentInitiator.java:145)
	at org.hibernate.engine.jdbc.env.internal.JdbcEnvironmentInitiator.initiateService(JdbcEnvironmentInitiator.java:66)
	at org.hibernate.engine.jdbc.env.internal.JdbcEnvironmentInitiator.initiateService(JdbcEnvironmentInitiator.java:35)
	at org.hibernate.boot.registry.internal.StandardServiceRegistryImpl.initiateService(StandardServiceRegistryImpl.java:101)
	at org.hibernate.service.internal.AbstractServiceRegistryImpl.createService(AbstractServiceRegistryImpl.java:263)
	... 24 more
Caused by: org.postgresql.util.PSQLException: Connection to localhost:26257 refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
	at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:245)
	at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:49)
	at org.postgresql.jdbc.PgConnection.<init>(PgConnection.java:195)
	at org.postgresql.Driver.makeConnection(Driver.java:452)
	at org.postgresql.Driver.connect(Driver.java:254)
	at org.hibernate.engine.jdbc.connections.internal.DriverConnectionCreator.makeConnection(DriverConnectionCreator.java:38)
	... 39 more
Caused by: java.net.ConnectException: Connection refused (Connection refused)
	at java.net.PlainSocketImpl.socketConnect(Native Method)
	at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
	at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
	at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
	at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
	at java.net.Socket.connect(Socket.java:589)
	at org.postgresql.core.PGStream.<init>(PGStream.java:69)
	at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:156)
	... 44 more

The blacklist had too many expected failures. Most of them were related to earlier issues with the test being unable to connect to the cluster. touches cockroachdb#40538 Release note: None

jordanlewis · 2019-09-10T12:59:48Z

@rafiss sometimes this means there are crashes - can you make sure the server logs also look good?

40609: roachtest: update 19.1 hibernate blacklist r=rafiss a=rafiss The blacklist had too many expected failures. Most of them were related to earlier issues with the test being unable to connect to the cluster. Also, remove the `--info` option from the test runner command, since it makes too much output. touches #40538 Release note: None 40617: opt: fix scalar building error handling r=RaduBerinde a=RaduBerinde We are incorrectly returning `nil` error in an error case. This leads to an assertion error instead of a "could not decorrelate subquery" error. Fixes #40590. Release note: None 40619: cliccl: Change license acquisition URL r=rohany a=rohany Change the cockroach demo license acquisition URLs to point to a new URL. Release note: None Co-authored-by: Rafi Shamim <[email protected]> Co-authored-by: Radu Berinde <[email protected]> Co-authored-by: Rohan Yadav <[email protected]>

cockroach-teamcity · 2019-09-11T15:45:24Z

SHA: https://github.com/cockroachdb/cockroach/commits/bdf41f7d03f0dafeeaf3bc6aac40f502ab069a6a

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=hibernate PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1478770&tab=buildLog

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20190911-1478770/hibernate/run_1
	hibernate.go:321,hibernate.go:332,test_runner.go:688: 
		Tests run on Cockroach v19.2.0-alpha.20190606-2197-gbdf41f7
		Tests run against Hibernate 5.4.4
		9518 Total Tests Run
		9032 tests passed
		486 tests failed
		0 tests passed unexpectedly
		199 tests failed unexpectedly
		0 tests expected failed, but not run
		For a full summary look at the hibernate artifacts 
		
		An updated blacklist (hibernateBlackList19_2) is available in the artifacts' hibernate log

cockroach-teamcity · 2019-09-12T11:27:33Z

SHA: https://github.com/cockroachdb/cockroach/commits/991282eacbbe1315fde694be9785ad8f6fa929d3

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=hibernate PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1481778&tab=buildLog

The test failed on branch=release-2.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20190912-1481778/hibernate/run_1
	hibernate.go:321,hibernate.go:332,test_runner.go:688: 
		Tests run on Cockroach v2.1.8-14-g991282e
		Tests run against Hibernate 5.4.4
		9527 Total Tests Run
		9164 tests passed
		363 tests failed
		0 tests passed unexpectedly
		29 tests failed unexpectedly
		0 tests expected failed, but not run
		For a full summary look at the hibernate artifacts 
		
		An updated blacklist (hibernateBlackList2_1) is available in the artifacts' hibernate log

cockroach-teamcity · 2019-09-12T19:48:42Z

SHA: https://github.com/cockroachdb/cockroach/commits/99bed2718a9b5d1c6b94ce0d65dc40d1e746de16

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=hibernate PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1481740&tab=buildLog

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20190912-1481740/hibernate/run_1
	test_runner.go:703: test timed out (10h0m0s)

cockroach-teamcity · 2019-09-13T09:33:11Z

SHA: https://github.com/cockroachdb/cockroach/commits/404cbf9085a55c9d05455ac3dd2ada1719833150

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=hibernate PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1484632&tab=buildLog

The test failed on branch=provisional_201909101743_v19.2.0-beta.20190913, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20190912-1484632/hibernate/run_1
	hibernate.go:321,hibernate.go:332,test_runner.go:688: 
		Tests run on Cockroach v19.2.0-alpha.20190606-2185-g404cbf9
		Tests run against Hibernate 5.4.4
		9516 Total Tests Run
		9012 tests passed
		504 tests failed
		0 tests passed unexpectedly
		217 tests failed unexpectedly
		0 tests expected failed, but not run
		For a full summary look at the hibernate artifacts 
		
		An updated blacklist (hibernateBlackList19_2) is available in the artifacts' hibernate log

cockroach-teamcity · 2019-09-13T15:51:57Z

SHA: https://github.com/cockroachdb/cockroach/commits/e8faca611a902766154ed82581d6d3a7483ad231

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=hibernate PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1484919&tab=buildLog

The test failed on branch=provisional_201908291837_v19.2.0-beta.20190903, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20190913-1484919/hibernate/run_1
	test_runner.go:703: test timed out (10h0m0s)

cockroach-teamcity · 2019-09-13T18:15:10Z

SHA: https://github.com/cockroachdb/cockroach/commits/404cbf9085a55c9d05455ac3dd2ada1719833150

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=hibernate PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1484994&tab=buildLog

The test failed on branch=provisional_201909101743_v19.2.0-beta.20190913, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20190913-1484994/hibernate/run_1
	test_runner.go:703: test timed out (10h0m0s)

cockroach-teamcity · 2019-09-14T11:31:04Z

SHA: https://github.com/cockroachdb/cockroach/commits/42d307e191ff6787a45e058be164fa452c47f368

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=hibernate PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1487519&tab=buildLog

The test failed on branch=release-2.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20190914-1487519/hibernate/run_1
	hibernate.go:321,hibernate.go:332,test_runner.go:688: 
		Tests run on Cockroach v2.1.8-18-g42d307e
		Tests run against Hibernate 5.4.4
		9527 Total Tests Run
		9164 tests passed
		363 tests failed
		0 tests passed unexpectedly
		29 tests failed unexpectedly
		0 tests expected failed, but not run
		For a full summary look at the hibernate artifacts 
		
		An updated blacklist (hibernateBlackList2_1) is available in the artifacts' hibernate log

cockroach-teamcity · 2019-09-15T06:21:23Z

SHA: https://github.com/cockroachdb/cockroach/commits/62b1678f652461bbc1aaf6bc2c0dd03105ce0ebe

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=hibernate PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1488785&tab=buildLog

The test failed on branch=40765, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20190914-1488785/hibernate/run_1
	hibernate.go:321,hibernate.go:332,test_runner.go:688: 
		Tests run on Cockroach v19.2.0-alpha.20190606-2261-g62b1678
		Tests run against Hibernate 5.4.4
		9484 Total Tests Run
		9040 tests passed
		444 tests failed
		0 tests passed unexpectedly
		158 tests failed unexpectedly
		1 test expected failed, but not run
		For a full summary look at the hibernate artifacts 
		
		An updated blacklist (hibernateBlackList19_2) is available in the artifacts' hibernate log

cockroach-teamcity · 2019-09-16T04:51:52Z

SHA: https://github.com/cockroachdb/cockroach/commits/62b1678f652461bbc1aaf6bc2c0dd03105ce0ebe

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=hibernate PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1489712&tab=buildLog

The test failed on branch=40765, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20190915-1489712/hibernate/run_1
	hibernate.go:321,hibernate.go:332,test_runner.go:688: 
		Tests run on Cockroach v19.2.0-alpha.20190606-2261-g62b1678
		Tests run against Hibernate 5.4.4
		9518 Total Tests Run
		9056 tests passed
		462 tests failed
		0 tests passed unexpectedly
		175 tests failed unexpectedly
		0 tests expected failed, but not run
		For a full summary look at the hibernate artifacts 
		
		An updated blacklist (hibernateBlackList19_2) is available in the artifacts' hibernate log

rafiss · 2019-09-30T19:51:24Z

The past few runs of the Hibernate nightlies against 19.1 have been stable (passing the expected passes and failing the expected fails). So it seems like this instability might be specific to 19.2, which is probably concerning, as @nvanbenschoten had asked last week.

cockroach-teamcity · 2019-10-01T03:33:52Z

SHA: https://github.com/cockroachdb/cockroach/commits/250f4c36de2b88eff443cf9be9cd5d2759312c88

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=hibernate PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1514702&tab=artifacts#/hibernate

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20190930-1514702/hibernate/run_1
	hibernate.go:322,hibernate.go:333,test_runner.go:689: 
		Tests run on Cockroach v19.2.0-alpha.20190606-2584-g250f4c3
		Tests run against Hibernate 5.4.6
		8389 Total Tests Run
		7795 tests passed
		594 tests failed
		0 tests passed unexpectedly
		373 tests failed unexpectedly
		18 tests expected failed, but not run
		For a full summary look at the hibernate artifacts 
		
		An updated blacklist (hibernateBlackList19_2) is available in the artifacts' hibernate log

cockroach-teamcity · 2019-10-01T14:09:52Z

SHA: https://github.com/cockroachdb/cockroach/commits/09d51e9f6265ed70caf49385be905606ebf722c7

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=hibernate PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1515124&tab=artifacts#/hibernate

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191001-1515124/hibernate/run_1
	hibernate.go:322,hibernate.go:333,test_runner.go:689: 
		Tests run on Cockroach v19.2.0-alpha.20190606-2603-g09d51e9
		Tests run against Hibernate 5.4.6
		8016 Total Tests Run
		7173 tests passed
		843 tests failed
		0 tests passed unexpectedly
		617 tests failed unexpectedly
		13 tests expected failed, but not run
		For a full summary look at the hibernate artifacts 
		
		An updated blacklist (hibernateBlackList19_2) is available in the artifacts' hibernate log

rafiss · 2019-10-01T15:54:09Z

Copying comment from the beta release issue: These Hibernate issues are not a beta blocker. The tests are concerningly flaky though, and we will continue investigating this during the rest of the release period. The issues stem from the Hibernate tests occasionally being unable to connect to the DB.

cockroach-teamcity · 2019-10-02T11:13:46Z

SHA: https://github.com/cockroachdb/cockroach/commits/4ae9892838aaf1df1f38bb8dd6bb8849adfaf16a

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=hibernate PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1516578&tab=artifacts#/hibernate

The test failed on branch=release-19.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191002-1516578/hibernate/run_1
	hibernate.go:322,hibernate.go:333,test_runner.go:689: 
		Tests run on Cockroach v19.1.5-14-g4ae9892
		Tests run against Hibernate 5.4.6
		9570 Total Tests Run
		9230 tests passed
		340 tests failed
		0 tests passed unexpectedly
		1 test failed unexpectedly
		0 tests expected failed, but not run
		For a full summary look at the hibernate artifacts 
		
		An updated blacklist (hibernateBlackList19_1) is available in the artifacts' hibernate log

cockroach-teamcity · 2019-10-02T14:11:30Z

SHA: https://github.com/cockroachdb/cockroach/commits/4dcfd7d8899d90dd1816153bf59b5df647c9bbd3

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=hibernate PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1516560&tab=artifacts#/hibernate

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191002-1516560/hibernate/run_1
	hibernate.go:322,hibernate.go:333,test_runner.go:689: 
		Tests run on Cockroach v19.2.0-alpha.20190606-2626-g4dcfd7d
		Tests run against Hibernate 5.4.6
		9570 Total Tests Run
		9314 tests passed
		256 tests failed
		0 tests passed unexpectedly
		17 tests failed unexpectedly
		0 tests expected failed, but not run
		For a full summary look at the hibernate artifacts 
		
		An updated blacklist (hibernateBlackList19_2) is available in the artifacts' hibernate log

cockroach-teamcity · 2019-10-03T10:35:48Z

SHA: https://github.com/cockroachdb/cockroach/commits/39112effea3100d1c678e88319199d2d017f930e

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=hibernate PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1518434&tab=artifacts#/hibernate

The test failed on branch=release-19.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191003-1518434/hibernate/run_1
	hibernate.go:322,hibernate.go:333,test_runner.go:689: 
		Tests run on Cockroach v19.1.5-16-g39112ef
		Tests run against Hibernate 5.4.6
		9570 Total Tests Run
		9230 tests passed
		340 tests failed
		0 tests passed unexpectedly
		1 test failed unexpectedly
		0 tests expected failed, but not run
		For a full summary look at the hibernate artifacts 
		
		An updated blacklist (hibernateBlackList19_1) is available in the artifacts' hibernate log

cockroach-teamcity · 2019-10-03T14:06:42Z

SHA: https://github.com/cockroachdb/cockroach/commits/1ed03c9811f01fef31950a5cb75a7b591af6fc26

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=hibernate PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1518416&tab=artifacts#/hibernate

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191003-1518416/hibernate/run_1
	hibernate.go:322,hibernate.go:333,test_runner.go:689: 
		Tests run on Cockroach v19.2.0-beta.20190930-50-g1ed03c9
		Tests run against Hibernate 5.4.6
		9362 Total Tests Run
		9005 tests passed
		357 tests failed
		0 tests passed unexpectedly
		120 tests failed unexpectedly
		2 tests expected failed, but not run
		For a full summary look at the hibernate artifacts 
		
		An updated blacklist (hibernateBlackList19_2) is available in the artifacts' hibernate log

cockroach-teamcity · 2019-10-04T10:30:34Z

SHA: https://github.com/cockroachdb/cockroach/commits/948a00e7b418b82d5421ad6aa0d651d7e9eeec91

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=hibernate PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1520710&tab=artifacts#/hibernate

The test failed on branch=release-19.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191004-1520710/hibernate/run_1
	hibernate.go:322,hibernate.go:333,test_runner.go:689: 
		Tests run on Cockroach v19.1.5-18-g948a00e
		Tests run against Hibernate 5.4.6
		9570 Total Tests Run
		9230 tests passed
		340 tests failed
		0 tests passed unexpectedly
		1 test failed unexpectedly
		0 tests expected failed, but not run
		For a full summary look at the hibernate artifacts 
		
		An updated blacklist (hibernateBlackList19_1) is available in the artifacts' hibernate log

cockroach-teamcity · 2019-10-04T14:37:53Z

SHA: https://github.com/cockroachdb/cockroach/commits/445ec411db8cde8cc090c7ffcd7ac70be9b21954

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=hibernate PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1520692&tab=artifacts#/hibernate

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191004-1520692/hibernate/run_1
	hibernate.go:322,hibernate.go:333,test_runner.go:689: 
		Tests run on Cockroach v19.2.0-beta.20190930-68-gca32e6f
		Tests run against Hibernate 5.4.6
		9341 Total Tests Run
		8970 tests passed
		371 tests failed
		0 tests passed unexpectedly
		134 tests failed unexpectedly
		2 tests expected failed, but not run
		For a full summary look at the hibernate artifacts 
		
		An updated blacklist (hibernateBlackList19_2) is available in the artifacts' hibernate log

cockroach-teamcity · 2019-10-06T08:15:55Z

SHA: https://github.com/cockroachdb/cockroach/commits/1c99165c39c3714f1ce9986bff75ce517f977630

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=hibernate PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1523814&tab=artifacts#/hibernate

The test failed on branch=provisional_201910051220_v19.2.0-beta.20191014, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191006-1523814/hibernate/run_1
	hibernate.go:322,hibernate.go:333,test_runner.go:689: 
		Tests run on Cockroach v19.2.0-beta.20190930-99-g1c99165
		Tests run against Hibernate 5.4.6
		6411 Total Tests Run
		5508 tests passed
		903 tests failed
		0 tests passed unexpectedly
		669 tests failed unexpectedly
		13 tests expected failed, but not run
		For a full summary look at the hibernate artifacts 
		
		An updated blacklist (hibernateBlackList19_2) is available in the artifacts' hibernate log

cockroach-teamcity · 2019-10-07T13:37:44Z

SHA: https://github.com/cockroachdb/cockroach/commits/8d3bdd75d0080cdb089b65d55f4e5fcf4a3ea905

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=hibernate PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1524120&tab=artifacts#/hibernate

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191007-1524120/hibernate/run_1
	hibernate.go:322,hibernate.go:333,test_runner.go:689: 
		Tests run on Cockroach v19.2.0-beta.20190930-105-g0a352e4
		Tests run against Hibernate 5.4.6
		7774 Total Tests Run
		7435 tests passed
		339 tests failed
		0 tests passed unexpectedly
		93 tests failed unexpectedly
		1 test expected failed, but not run
		For a full summary look at the hibernate artifacts 
		
		An updated blacklist (hibernateBlackList19_2) is available in the artifacts' hibernate log

cockroach-teamcity · 2019-10-08T13:23:13Z

SHA: https://github.com/cockroachdb/cockroach/commits/c0cab6082c2383e77f0a806c10a1506b6aa7b6bf

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=hibernate PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1525981&tab=artifacts#/hibernate

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191008-1525981/hibernate/run_1
	hibernate.go:322,hibernate.go:333,test_runner.go:689: 
		Tests run on Cockroach v19.2.0-beta.20190930-136-gc0cab60
		Tests run against Hibernate 5.4.6
		7879 Total Tests Run
		7621 tests passed
		258 tests failed
		0 tests passed unexpectedly
		11 tests failed unexpectedly
		0 tests expected failed, but not run
		For a full summary look at the hibernate artifacts 
		
		An updated blacklist (hibernateBlackList19_2) is available in the artifacts' hibernate log

rafiss · 2019-10-08T15:47:58Z

It looks like a full test run happened without crashing and/or connection timeouts. That's an interesting data point.

cockroach-teamcity · 2019-10-08T21:18:28Z

SHA: https://github.com/cockroachdb/cockroach/commits/27ea46b456d6b3dabac2c0f6c5c120ed316fffce

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=hibernate PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1526784&tab=artifacts#/hibernate

The test failed on branch=provisional_201910072154_v19.2.0-beta.20191014, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191008-1526784/hibernate/run_1
	hibernate.go:322,hibernate.go:333,test_runner.go:689: 
		Tests run on Cockroach v19.2.0-beta.20190930-116-g27ea46b
		Tests run against Hibernate 5.4.6
		7423 Total Tests Run
		6981 tests passed
		442 tests failed
		0 tests passed unexpectedly
		224 tests failed unexpectedly
		29 tests expected failed, but not run
		For a full summary look at the hibernate artifacts 
		
		An updated blacklist (hibernateBlackList19_2) is available in the artifacts' hibernate log

cockroach-teamcity · 2019-10-09T06:19:57Z

SHA: https://github.com/cockroachdb/cockroach/commits/51957cb7aec516e8ce333f85acbbd68fc7415a32

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=hibernate PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1528314&tab=artifacts#/hibernate

The test failed on branch=release-2.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191009-1528314/hibernate/run_1
	hibernate.go:48,hibernate.go:364,test_runner.go:689: all attempts failed for change database replication and GC time due to error: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1570597897-28-n1cpu4:1 -- ./cockroach sql --insecure -e
		        'ALTER RANGE default CONFIGURE ZONE USING num_replicas = 1, gc.ttlseconds = 120;' returned:
		stderr:
		
		stdout:
		ple times.
		       
		      --url postgresql://[user[:passwd]@]host[:port]/[db][?parameters...]   
		        Connection URL, e.g. "postgresql://myuser@localhost:26257/mydb". If
		        left empty, the connection flags are used (host, port, user, database,
		        insecure, certs-dir).
		        Environment variable: COCKROACH_URL
		       
		  -u, --user string                                                         
		        Database user name.
		        Environment variable: COCKROACH_USER
		
		Global Flags:
		      --logtostderr Severity[=DEFAULT]   logs at or above this threshold go to stderr (default NONE)
		      --no-color                         disable standard error log colorization
		      --vmodule moduleSpec               comma-separated list of pattern=N settings for file-filtered logging (significantly hurts performance)
		
		Error: flag needs an argument: 'e' in -e
		Failed running "sql"
		bash: line 1: ALTER RANGE default CONFIGURE ZONE USING num_replicas = 1, gc.ttlseconds = 120;: command not found
		Error:  exit status 127
		: exit status 1

cockroach-teamcity · 2019-10-09T06:28:50Z

SHA: https://github.com/cockroachdb/cockroach/commits/06026bab5cdb3695ddeba209c94175cd9460b170

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=hibernate PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1528278&tab=artifacts#/hibernate

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191009-1528278/hibernate/run_1
	hibernate.go:48,hibernate.go:364,test_runner.go:689: all attempts failed for change database replication and GC time due to error: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1570597867-25-n1cpu4:1 -- ./cockroach sql --insecure -e
		        'ALTER RANGE default CONFIGURE ZONE USING num_replicas = 1, gc.ttlseconds = 120;' returned:
		stderr:
		
		stdout:
		    --watch duration                                                      
		                                                                                    Repeat the SQL statement(s) specified with --execute with the specified
		                                                                                    period. The client will stop watching if an execution of the SQL
		                                                                                    statement(s) fail.
		
		Global Flags:
		      --logtostderr Severity[=DEFAULT]   logs at or above this threshold go to stderr (default NONE)
		      --no-color                         disable standard error log colorization
		      --vmodule moduleSpec               comma-separated list of pattern=N settings for file-filtered logging (significantly hurts performance)
		
		Error: flag needs an argument: 'e' in -e
		Failed running "sql"
		bash: line 1: ALTER RANGE default CONFIGURE ZONE USING num_replicas = 1, gc.ttlseconds = 120;: command not found
		Error:  exit status 127
		: exit status 1

cockroach-teamcity · 2019-10-09T06:45:44Z

SHA: https://github.com/cockroachdb/cockroach/commits/5b223f9b2ebcbaeb9dd7738fc947b36d217b9f22

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=hibernate PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1528296&tab=artifacts#/hibernate

The test failed on branch=release-19.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191009-1528296/hibernate/run_1
	hibernate.go:48,hibernate.go:364,test_runner.go:689: all attempts failed for change database replication and GC time due to error: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1570597957-25-n1cpu4:1 -- ./cockroach sql --insecure -e
		        'ALTER RANGE default CONFIGURE ZONE USING num_replicas = 1, gc.ttlseconds = 120;' returned:
		stderr:
		
		stdout:
		                                                             Environment variable: COCKROACH_URL
		                                                                                   
		  -u, --user string                                                         
		                                                                                    Database user name.
		                                                                                    Environment variable: COCKROACH_USER
		
		Global Flags:
		      --logtostderr Severity[=DEFAULT]   logs at or above this threshold go to stderr (default NONE)
		      --no-color                         disable standard error log colorization
		      --vmodule moduleSpec               comma-separated list of pattern=N settings for file-filtered logging (significantly hurts performance)
		
		Error: flag needs an argument: 'e' in -e
		Failed running "sql"
		bash: line 1: ALTER RANGE default CONFIGURE ZONE USING num_replicas = 1, gc.ttlseconds = 120;: command not found
		Error:  exit status 127
		: exit status 1

cockroach-teamcity · 2019-10-10T06:18:40Z

SHA: https://github.com/cockroachdb/cockroach/commits/d4c59df1dd71fb97ccf356d4eda49f0c70dcda36

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=hibernate PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1531454&tab=artifacts#/hibernate

The test failed on branch=release-2.1, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191010-1531454/hibernate/run_1
	hibernate.go:66,hibernate.go:218,test_runner.go:689: all attempts failed for change jobs retention time due to error: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod run teamcity-1570684277-27-n1cpu4:1 -- ./cockroach sql --insecure -e "SET CLUSTER SETTING jobs.retention_time = '180s';" returned:
		stderr:
		
		stdout:
		Error: pq: unknown cluster setting 'jobs.retention_time'
		Failed running "sql"
		Error:  exit status 1
		: exit status 1

rafiss · 2019-10-10T16:20:25Z

The hibernate test suite finally passed flake-free against version 19.2 last night.

The above error for 2.1 is because it is using a cluster setting that didn't exist in 2.1. It will be fixed by #41504

cockroach-teamcity · 2019-10-11T06:11:47Z

SHA: https://github.com/cockroachdb/cockroach/commits/41bac2fd99a91d92cb3ff6426789f7e64dd6b14a

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=hibernate PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1534498&tab=artifacts#/hibernate

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191011-1534498/hibernate/run_1
	cluster.go:1695,hibernate.go:36,hibernate.go:192,test_runner.go:689: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod start teamcity-1570770659-35-n1cpu4:1 returned:
		stderr:
		
		stdout:
		 CLUSTER SETTING server.remote_debugging.mode = 'any';
		SET CLUSTER SETTING cluster.organization = 'Cockroach Labs - Production Testing';
		SET CLUSTER SETTING enterprise.license = 'crl-0-EJL04ukFGAEiI0NvY2tyb2FjaCBMYWJzIC0gUHJvZHVjdGlvbiBUZXN0aW5n';" &&
					touch /mnt/data1/cockroach/settings-initialized
		fi
		
		Error: cannot dial server.
		Is the server running?
		If the server is running, check --host client-side and --advertise server-side.
		
		dial tcp 127.0.0.1:26257: connect: connection refused
		Failed running "sql"
		
		github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.Cockroach.Start.func2
			/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cockroach.go:297
		github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.(*SyncedCluster).Parallel.func1.1
			/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cluster_synced.go:1547
		runtime.goexit
			/usr/local/go/src/runtime/asm_amd64.s:1337: 
		I191011 06:11:35.667172 1 cluster_synced.go:1629  command failed
		: exit status 1

cockroach-teamcity · 2019-10-12T06:15:36Z

SHA: https://github.com/cockroachdb/cockroach/commits/fad7d8c705cdef3c274a8a3bbcb9d08d7386e1ed

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=hibernate PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1535757&tab=artifacts#/hibernate

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191012-1535757/hibernate/run_1
	cluster.go:1695,hibernate.go:36,hibernate.go:192,test_runner.go:689: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/bin/roachprod start teamcity-1570857008-34-n1cpu4:1 returned:
		stderr:
		
		stdout:
		 CLUSTER SETTING server.remote_debugging.mode = 'any';
		SET CLUSTER SETTING cluster.organization = 'Cockroach Labs - Production Testing';
		SET CLUSTER SETTING enterprise.license = 'crl-0-EJL04ukFGAEiI0NvY2tyb2FjaCBMYWJzIC0gUHJvZHVjdGlvbiBUZXN0aW5n';" &&
					touch /mnt/data1/cockroach/settings-initialized
		fi
		
		Error: cannot dial server.
		Is the server running?
		If the server is running, check --host client-side and --advertise server-side.
		
		dial tcp 127.0.0.1:26257: connect: connection refused
		Failed running "sql"
		
		github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.Cockroach.Start.func2
			/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cockroach.go:297
		github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install.(*SyncedCluster).Parallel.func1.1
			/home/agent/work/.go/src/github.com/cockroachdb/cockroach/pkg/cmd/roachprod/install/cluster_synced.go:1547
		runtime.goexit
			/usr/local/go/src/runtime/asm_amd64.s:1337: 
		I191012 06:15:25.688588 1 cluster_synced.go:1629  command failed
		: exit status 1

rafiss · 2019-10-14T15:26:08Z

Not sure why this is failing. It looks like it fails during roachtest setup when setting the enterprise license, before Hibernate is involved.

When I run roachtest from my computer, it works fine.

rafiss · 2019-10-14T15:35:56Z

E191012 06:15:24.918452 1 cli/error.go:233 cockroach server exited with error: could not query store capacity: open /mnt/data1/cockroach/real/cockroach-temp718633906: permission denied

rafiss · 2019-10-14T15:51:24Z

Should be fixed by #41552 and probably was initially caused by #41498

cockroach-teamcity · 2019-10-15T09:30:22Z

SHA: https://github.com/cockroachdb/cockroach/commits/a40cefbb5bd5e9d34f5db930334a385d0934d2d0

Parameters:

To repro, try:

# Don't forget to check out a clean suitable branch and experiment with the
# stress invocation until the desired results present themselves. For example,
# using stress instead of stressrace and passing the '-p' stressflag which
# controls concurrency.
./scripts/gceworker.sh start && ./scripts/gceworker.sh mosh
cd ~/go/src/github.com/cockroachdb/cockroach && \
stdbuf -oL -eL \
make stressrace TESTS=hibernate PKG=roachtest TESTTIMEOUT=5m STRESSFLAGS='-maxtime 20m -timeout 10m' 2>&1 | tee /tmp/stress.log

Failed test: https://teamcity.cockroachdb.com/viewLog.html?buildId=1538863&tab=artifacts#/hibernate

The test failed on branch=master, cloud=gce:
test artifacts and logs in: /home/agent/work/.go/src/github.com/cockroachdb/cockroach/artifacts/20191015-1538863/hibernate/run_1
	orm_helpers.go:174,orm_helpers.go:111,java_helpers.go:199,hibernate.go:182,hibernate.go:192,test_runner.go:689: 
		Tests run on Cockroach v19.2.0-beta.20190930-313-ga40cefb
		Tests run against hibernate 5.4.6
		7879 Total Tests Run
		7633 tests passed
		246 tests failed
		0 tests skipped
		0 tests ignored
		1 test passed unexpectedly
		0 tests failed unexpectedly
		0 tests expected failed but skipped
		0 tests expected failed but not run
		For a full summary look at the hibernate artifacts 
		
		An updated blacklist (hibernateBlackList19_2) is available in the artifacts' hibernate log

rafiss · 2019-10-16T15:02:00Z

After optimizing some pg_catalog queries (#41281) and changing cluster settings used during the test run (#41339), this no longer seems to be flaky. Closing.

cockroach-teamcity added C-test-failure Broken test (automatically or manually discovered). O-roachtest O-robot Originated from a bot. labels Sep 6, 2019

cockroach-teamcity added this to the 19.2 milestone Sep 6, 2019

cockroach-teamcity assigned andreimatei Sep 6, 2019

jordanlewis assigned rafiss and unassigned andreimatei Sep 6, 2019

rafiss mentioned this issue Sep 9, 2019

roachtest: update 19.1 hibernate blacklist #40609

Merged

This was referenced Oct 1, 2019

release: v19.2.0-beta.20190930 #41128

Closed

roachtest: stop making "unexpected passes" in nightly ORM test create a Github issue #41226

Closed

rafiss closed this as completed Oct 16, 2019

roachtest: hibernate failed #40538

roachtest: hibernate failed #40538

Comments

cockroach-teamcity commented Sep 6, 2019

jordanlewis commented Sep 6, 2019

cockroach-teamcity commented Sep 7, 2019

cockroach-teamcity commented Sep 7, 2019

rafiss commented Sep 9, 2019 • edited Loading

cockroach-teamcity commented Sep 9, 2019

rafiss commented Sep 9, 2019

jordanlewis commented Sep 10, 2019

cockroach-teamcity commented Sep 11, 2019

cockroach-teamcity commented Sep 12, 2019

cockroach-teamcity commented Sep 12, 2019

cockroach-teamcity commented Sep 13, 2019

cockroach-teamcity commented Sep 13, 2019

cockroach-teamcity commented Sep 13, 2019

cockroach-teamcity commented Sep 14, 2019

cockroach-teamcity commented Sep 15, 2019

cockroach-teamcity commented Sep 16, 2019

rafiss commented Sep 30, 2019

cockroach-teamcity commented Oct 1, 2019

cockroach-teamcity commented Oct 1, 2019

rafiss commented Oct 1, 2019

cockroach-teamcity commented Oct 2, 2019

cockroach-teamcity commented Oct 2, 2019

cockroach-teamcity commented Oct 3, 2019

cockroach-teamcity commented Oct 3, 2019

cockroach-teamcity commented Oct 4, 2019

cockroach-teamcity commented Oct 4, 2019

cockroach-teamcity commented Oct 6, 2019

cockroach-teamcity commented Oct 7, 2019

cockroach-teamcity commented Oct 8, 2019

rafiss commented Oct 8, 2019

cockroach-teamcity commented Oct 8, 2019

cockroach-teamcity commented Oct 9, 2019

cockroach-teamcity commented Oct 9, 2019

cockroach-teamcity commented Oct 9, 2019

cockroach-teamcity commented Oct 10, 2019

rafiss commented Oct 10, 2019

cockroach-teamcity commented Oct 11, 2019

cockroach-teamcity commented Oct 12, 2019

rafiss commented Oct 14, 2019 • edited Loading

rafiss commented Oct 14, 2019

rafiss commented Oct 14, 2019 • edited Loading

cockroach-teamcity commented Oct 15, 2019

rafiss commented Oct 16, 2019

rafiss commented Sep 9, 2019 •

edited

Loading

rafiss commented Oct 14, 2019 •

edited

Loading

rafiss commented Oct 14, 2019 •

edited

Loading