Still get problems on test #716

BowenXiao1999 · 2020-07-03T14:29:29Z

Question

What I did

make test failed, but I change nothing on local. Then I build up bin for cdc:

But I still get the same error stack partially:

FAIL    github.com/pingcap/ticdc/cdc/kv 0.761s
ok      github.com/pingcap/ticdc/cdc/model      0.606s  coverage: 26.4% of statements

----------------------------------------------------------------------
FAIL: buffer_test.go:61: bufferSuite.TestWaitsCanBeCanceled

buffer_test.go:86:
    c.Fatal("AddEntry doesn't stop in time.")
... Error: AddEntry doesn't stop in time.

OOPS: 4 passed, 1 FAILED
--- FAIL: TestSuite (31.38s)
FAIL
coverage: 17.9% of statements
FAIL    github.com/pingcap/ticdc/cdc/puller     32.623s
ok      github.com/pingcap/ticdc/cdc/puller/frontier    117.606s        coverage: 85.5% of statements

...
FAIL    github.com/pingcap/ticdc/pkg/etcd       0.594s
ok      github.com/pingcap/ticdc/pkg/filter     0.597s  coverage: 92.3% of statements
ok      github.com/pingcap/ticdc/pkg/flags      0.501s  coverage: 91.8% of statements
ok      github.com/pingcap/ticdc/pkg/notify     7.646s  coverage: 77.3% of statements
?       github.com/pingcap/ticdc/pkg/quotes     [no test files]
ok      github.com/pingcap/ticdc/pkg/regionspan 0.763s  coverage: 85.2% of statements
ok      github.com/pingcap/ticdc/pkg/retry      3.894s  coverage: 100.0% of statements
ok      github.com/pingcap/ticdc/pkg/scheduler  0.508s  coverage: 79.6% of statements
ok      github.com/pingcap/ticdc/pkg/types      0.509s  coverage: 91.7% of statements
ok      github.com/pingcap/ticdc/pkg/util       0.759s  coverage: 75.5% of statements
FAIL
make: *** [unit_test] Error 1

What I expect

I want to test my code, but I get test error. So I stash my commit and try again, still same error. So I'm confused at how the test work.

The text was updated successfully, but these errors were encountered:

amyangfei · 2020-07-04T03:08:55Z

The cause of failure is wait time is not enough. We can increase the wait time here.

Besides, the unit test can be executed by single case. ref https://github.com/pingcap/ticdc/tree/master/tests#unit-test

BowenXiao1999 · 2020-07-04T04:29:44Z

Do we have bin for Mac? The links are all about Linux.

amyangfei · 2020-07-04T05:33:31Z

Do we have bin for Mac? The links are all about Linux.

https://tiup-mirrors.pingcap.com/tidb-v4.0.2-darwin-amd64.tar.gz

Can replace tidb with other components.

BowenXiao1999 · 2020-07-04T06:14:53Z

[2020/07/04 14:12:33.650 +08:00] [ERROR] [main.go:213] ["The current temporary storage dir has been occupied by another instance, check tmp-storage-path config and make sure they are different."] [TempStoragePath="/tmp/0_tidb/MC4wLjAuMDo0MDAwLzAuMC4wLjA6MTAwODA=/tmp-storage"] [error="fslock: lock is held"] [stack="github.com/pingcap/log.Error\n\t/Users/pingcap/gopkg/pkg/mod/github.com/pingcap/[email protected]/global.go:42\nmain.initializeTempDir\n\t/Users/pingcap/workspace/build-darwin-amd64-4.0/go/src/github.com/pingcap/tidb/tidb-server/main.go:213\nmain.main\n\t/Users/pingcap/workspace/build-darwin-amd64-4.0/go/src/github.com/pingcap/tidb/tidb-server/main.go:171\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:203"]
/Users/bytedance/code/ticdc/tests/_utils/start_tidb_cluster: line 159: mysql: command not found
/Users/bytedance/code/ticdc/tests/_utils/start_tidb_cluster: line 159: mysql: command not found
/Users/bytedance/code/ticdc/tests/_utils/start_tidb_cluster: line 159: mysql: command not found
/Users/bytedance/code/ticdc/tests/_utils/start_tidb_cluster: line 159: mysql: command not found
/Users/bytedance/code/ticdc/tests/_utils/start_tidb_cluster: line 159: mysql: command not found
/Users/bytedance/code/ticdc/tests/_utils/start_tidb_cluster: line 159: mysql: command not found
/Users/bytedance/code/ticdc/tests/_utils/start_tidb_cluster: line 159: mysql: command not found
/Users/bytedance/code/ticdc/tests/_utils/start_tidb_cluster: line 159: mysql: command not found
/Users/bytedance/code/ticdc/tests/_utils/start_tidb_cluster: line 159: mysql: command not found
/Users/bytedance/code/ticdc/tests/_utils/start_tidb_cluster: line 159: mysql: command not found
/Users/bytedance/code/ticdc/tests/_utils/start_tidb_cluster: line 159: mysql: command not found
/Users/bytedance/code/ticdc/tests/_utils/start_tidb_cluster: line 159: mysql: command not found

I get test error like this. I think I'm able to write to /tmp/tidb_cdc_test cuz I have ran sudo su.

BTW, where can I download sync_diff_inspector for mac? I got all binaries ok except this one.

I deleted /tmp/tidb_cdc_test, now there is no log info but only /Users/bytedance/code/ticdc/tests/_utils/start_tidb_cluster: line 159: mysql: command not found

amyangfei · 2020-07-04T06:29:53Z

line 159: mysql: command not found
means mysql client is needed.

sync_diff_inspector doesn't have a mac release, try to compile it. https://github.com/pingcap/tidb-tools

BowenXiao1999 · 2020-07-04T12:03:30Z

Thanks. But I still get problems.

I ran sudo make integration_test CASE=partition_table.

There are some invalid signals, I consider it as some noise.

killall: unknown signal w; valid signals:
HUP INT QUIT ILL TRAP ABRT EMT FPE KILL BUS SEGV SYS PIPE ALRM TERM URG STOP 
TSTP CONT CHLD TTIN TTOU IO XCPU XFSZ VTALRM PROF WINCH INFO USR1 USR2 
killall: unknown signal w; valid signals:
......

Then, it seems to work, but I get MySQL connection error and some error logs. The script keeps retrying so the failed message keeps logging.

start tidb cluster in /tmp/tidb_cdc_test/partition_table
Starting Upstream PD...
Starting Downstream PD...
Starting Upstream TiKV...
Starting Downstream TiKV...
Starting Upstream TiDB...
Starting Downstream TiDB...
Verifying Upstream TiDB is started...
ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (61)
[2020/07/04 19:22:00.445 +08:00] [ERROR] [main.go:213] ["The current temporary storage dir has been occupied by another instance, check tmp-storage-path config and make sure they are different."] [TempStoragePath="/tmp/0_tidb/MC4wLjAuMDo0MDAwLzAuMC4wLjA6MTAwODA=/tmp-storage"] [error="fslock: lock is held"] [stack="github.com/pingcap/log.Error\n\t/Users/pingcap/gopkg/pkg/mod/github.com/pingcap/[email protected]/global.go:42\nmain.initializeTempDir\n\t/Users/pingcap/workspace/build-darwin-amd64-4.0/go/src/github.com/pingcap/tidb/tidb-server/main.go:213\nmain.main\n\t/Users/pingcap/workspace/build-darwin-amd64-4.0/go/src/github.com/pingcap/tidb/tidb-server/main.go:171\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:203"]
[2020/07/04 19:22:00.445 +08:00] [ERROR] [main.go:213] ["The current temporary storage dir has been occupied by another instance, check tmp-storage-path config and make sure they are different."] [TempStoragePath="/tmp/0_tidb/MC4wLjAuMDo0MDAxLzAuMC4wLjA6MTAwODE=/tmp-storage"] [error="fslock: lock is held"] [stack="github.com/pingcap/log.Error\n\t/Users/pingcap/gopkg/pkg/mod/github.com/pingcap/[email protected]/global.go:42\nmain.initializeTempDir\n\t/Users/pingcap/workspace/build-darwin-amd64-4.0/go/src/github.com/pingcap/tidb/tidb-server/main.go:213\nmain.main\n\t/Users/pingcap/workspace/build-darwin-amd64-4.0/go/src/github.com/pingcap/tidb/tidb-server/main.go:171\nruntime.main\n\t/usr/local/go/src/runtime/proc.go:203"]
ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (61)
ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (61)
ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (61)
ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (61)
ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (61)
ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (61)
ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (61)
ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (61)
ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (61)
ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (61)
ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (61)
ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (61)
ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (61)
ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (61)
ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (61)
ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (61)
ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (61)
ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (61)
ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (61)
ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (61)
ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (61)
ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (61)
ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (61)
ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (61)
ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (61)
ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (61)
ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (61)
ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (61)
ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (61)

I have MySQL client installed and launched. I can enter MySQL by:

C02YV0JFLVCL:ticdc bytedance$ mysql
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 9
Server version: 8.0.20 MySQL Community Server - GPL

Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> show variables like 'PORT';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| port          | 3306  |
+---------------+-------+
1 row in set (0.00 sec)

mysql>

I have tried google some suggestion. Some config like bind address seems not work for me.

amyangfei · 2020-07-04T12:08:21Z

pay attention to this error, it is upstream database that can't be connected

Verifying Upstream TiDB is started...
ERROR 2003 (HY000): Can't connect to MySQL server on '127.0.0.1' (61)
[2020/07/04 19:22:00.445 +08:00] [ERROR] [main.go:213] ["The current temporary storage dir has been occupied by another instance, check tmp-storage-path config and make sure they are different."]

BowenXiao1999 · 2020-07-04T14:45:14Z

Now I can connect to finally.

But it logs

Verifying Downstream TiDB is started...
ERROR 1146 (42S02) at line 1: Table 'mysql.tidb' doesn't exist
ERROR 1146 (42S02) at line 1: Table 'mysql.tidb' doesn't exist
ERROR 1146 (42S02) at line 1: Table 'mysql.tidb' doesn't exist

After I create a tidb table in MySQL, I get

ERROR 1054 (42S22) at line 1: Unknown column 'variable_name' in 'where clause'

Do I need to do something in this mysql server?

amyangfei · 2020-07-05T00:18:24Z

Now I can connect to finally.

But it logs

Verifying Downstream TiDB is started...
ERROR 1146 (42S02) at line 1: Table 'mysql.tidb' doesn't exist
ERROR 1146 (42S02) at line 1: Table 'mysql.tidb' doesn't exist
ERROR 1146 (42S02) at line 1: Table 'mysql.tidb' doesn't exist

After I create a tidb table in MySQL, I get

ERROR 1054 (42S22) at line 1: Unknown column 'variable_name' in 'where clause'

Do I need to do something in this mysql server?

The downstream database is also a TiDB, https://github.com/pingcap/ticdc/blob/37072effd0869904ebf22b37a1016fb80361dadd/tests/_utils/start_tidb_cluster#L148-L155

Have you changed the startup script manually? You don't need to start any databases before the test, the test script will start both upstream tidb cluster and downstream cluster automatically.

amyangfei · 2020-07-05T00:25:52Z

Now I can connect to finally.

But it logs

Verifying Downstream TiDB is started...
ERROR 1146 (42S02) at line 1: Table 'mysql.tidb' doesn't exist
ERROR 1146 (42S02) at line 1: Table 'mysql.tidb' doesn't exist
ERROR 1146 (42S02) at line 1: Table 'mysql.tidb' doesn't exist

After I create a tidb table in MySQL, I get

ERROR 1054 (42S22) at line 1: Unknown column 'variable_name' in 'where clause'

Do I need to do something in this mysql server?

mysql.tidb is a special system table in TiDB only.

BowenXiao1999 · 2020-07-05T03:27:10Z

Now I can connect to finally.
But it logs
Verifying Downstream TiDB is started...
ERROR 1146 (42S02) at line 1: Table 'mysql.tidb' doesn't exist
ERROR 1146 (42S02) at line 1: Table 'mysql.tidb' doesn't exist
ERROR 1146 (42S02) at line 1: Table 'mysql.tidb' doesn't exist
After I create a tidb table in MySQL, I get
ERROR 1054 (42S22) at line 1: Unknown column 'variable_name' in 'where clause'
Do I need to do something in this mysql server?
mysql.tidb is a special system table in TiDB only.

Yes.. my local MySQL was still running. I shut it down and now it logs that I should install a TiFlash..

/Users/bytedance/code/ticdc/tests/_utils/start_tidb_cluster: line 273: tiflash: command not found

I download tiflash from https://tiup-mirrors.pingcap.com/tiflash-v4.0.2-darwin-amd64.tar.gz. Is there a document to install tiflash? Or just add the binary to environment variable.

amyangfei · 2020-07-06T05:25:59Z

Now I can connect to finally.
But it logs
Verifying Downstream TiDB is started...
ERROR 1146 (42S02) at line 1: Table 'mysql.tidb' doesn't exist
ERROR 1146 (42S02) at line 1: Table 'mysql.tidb' doesn't exist
ERROR 1146 (42S02) at line 1: Table 'mysql.tidb' doesn't exist
After I create a tidb table in MySQL, I get
ERROR 1054 (42S22) at line 1: Unknown column 'variable_name' in 'where clause'
Do I need to do something in this mysql server?
mysql.tidb is a special system table in TiDB only.
Yes.. my local MySQL was still running. I shut it down and now it logs that I should install a TiFlash..
/Users/bytedance/code/ticdc/tests/_utils/start_tidb_cluster: line 273: tiflash: command not found
I download tiflash from https://tiup-mirrors.pingcap.com/tiflash-v4.0.2-darwin-amd64.tar.gz. Is there a document to install tiflash? Or just add the binary to environment variable.

Sorry for the delayed response, the TiFlash is used for testing whether TiCDC can work normally with a TiFlash in cluster, you can comment the startup scripts if you don't want to start it.
Besides, if you want to follow the standard procedure integration test, the tiflash related binaries should be prepared, including flash_cluster_manager, libtiflash_proxy.so and tiflash binary, all of these files can be found in tiflash package.

https://github.com/pingcap/ticdc/blob/47510f5a51491c93a5bedeca39b602bca98ef096/scripts/jenkins_ci/integration_test_common.groovy#L52-L53

BowenXiao1999 · 2020-07-06T12:17:30Z

@amyangfei update:
I'm able to run integration test locally.

So, recap for testing and make things correct, I should check these for debug:

run make unit_test and go test for unit test in your changed directory and other directory. (https://github.com/pingcap/ticdc/tree/master/tests#unit-test says test each function seperatelly but i dk which test function will be involved).
run make integration_test CASE=partition_table and ddl_reentrant for integration test.

Is it correct?
For files in /tmp/tidb_cdc_test, which file should I looked into if it's helpful for debug?

amyangfei · 2020-07-06T15:52:26Z

For files in /tmp/tidb_cdc_test, which file should I looked into if it's helpful for debug?

No existing tests test this DDL
More tests should be added to test the new DDL can be executed as expected. The ddl_reentrant test is a special test for DDL reentrant verification. You can take this test as an example

amyangfei · 2020-07-06T15:53:20Z

Debug log for integration test can be found in /tmp/tidb_cdc_test/<test_name>/cdc.log

BowenXiao1999 · 2020-07-06T16:11:48Z

For files in /tmp/tidb_cdc_test, which file should I looked into if it's helpful for debug?

No existing tests test this DDL

More tests should be added to test the new DDL can be executed as expected. The ddl_reentrant test is a special test for DDL reentrant verification. You can take this test as an example

Yes, I should begin to add some unit test first and handle the complicate case.

amyangfei · 2020-08-14T09:12:31Z

Close this issue now, feel free to re open it if you have any problems.

BowenXiao1999 added the question Further information is requested. label Jul 3, 2020

BowenXiao1999 mentioned this issue Jul 4, 2020

Support exchange partition in TiCDC #639

Closed

1 task

BowenXiao1999 mentioned this issue Jul 5, 2020

feat: basic implementation of exchange partition #718

Closed

amyangfei closed this as completed Aug 14, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Still get problems on test #716

Still get problems on test #716

BowenXiao1999 commented Jul 3, 2020

amyangfei commented Jul 4, 2020

BowenXiao1999 commented Jul 4, 2020

amyangfei commented Jul 4, 2020

BowenXiao1999 commented Jul 4, 2020 •

edited

Loading

amyangfei commented Jul 4, 2020

BowenXiao1999 commented Jul 4, 2020

amyangfei commented Jul 4, 2020

BowenXiao1999 commented Jul 4, 2020

amyangfei commented Jul 5, 2020

amyangfei commented Jul 5, 2020

BowenXiao1999 commented Jul 5, 2020

amyangfei commented Jul 6, 2020

BowenXiao1999 commented Jul 6, 2020

amyangfei commented Jul 6, 2020

amyangfei commented Jul 6, 2020

BowenXiao1999 commented Jul 6, 2020

amyangfei commented Aug 14, 2020

Still get problems on test #716

Still get problems on test #716

Comments

BowenXiao1999 commented Jul 3, 2020

Question

What I did

What I expect

amyangfei commented Jul 4, 2020

BowenXiao1999 commented Jul 4, 2020

amyangfei commented Jul 4, 2020

BowenXiao1999 commented Jul 4, 2020 • edited Loading

amyangfei commented Jul 4, 2020

BowenXiao1999 commented Jul 4, 2020

amyangfei commented Jul 4, 2020

BowenXiao1999 commented Jul 4, 2020

amyangfei commented Jul 5, 2020

amyangfei commented Jul 5, 2020

BowenXiao1999 commented Jul 5, 2020

amyangfei commented Jul 6, 2020

BowenXiao1999 commented Jul 6, 2020

amyangfei commented Jul 6, 2020

amyangfei commented Jul 6, 2020

BowenXiao1999 commented Jul 6, 2020

amyangfei commented Aug 14, 2020

BowenXiao1999 commented Jul 4, 2020 •

edited

Loading