local backend panic when build with CGO_ENABLED=0

[sleepymole]

Bug Report

Please answer these questions before submitting your issue. Thanks!

1. What did you do? If possible, provide a recipe for reproducing the error.

   I build tidb-lightning with CGO_ENABLED=0 on windows. Then I use local backend to restore about 1G csv files.

2. What did you expect to see?

   All csv files are restored successfully.

3. What did you see instead?

   tidb-lightning panic in pebble's source code.

Output

Verbose debug logs will be written to C:/Users/mrxia/Projects/tidb-lightning.log
[2020/12/24 16:34:17.020 +08:00] [INFO] [client.go:193] ["[pd] create pd client with endpoints"] [pd-address="[192.168.232.2:2379]"]

[2020/12/24 16:34:17.024 +08:00] [INFO] [base_client.go:296] ["[pd] update member urls"] [old-urls="[http://192.168.232.2:2379]"] [new-urls="[http://localhost:2379]"]

[2020/12/24 16:34:17.024 +08:00] [INFO] [base_client.go:308] ["[pd] switch leader"] [new-leader=http://localhost:2379] [old-leader=]

[2020/12/24 16:34:17.024 +08:00] [INFO] [base_client.go:112] ["[pd] init cluster id"] [cluster-id=6909738232910158800]

[2020/12/24 16:34:17.436 +08:00] [INFO] [client.go:193] ["[pd] create pd client with endpoints"] [pd-address="[192.168.232.2:2379]"]

[2020/12/24 16:34:17.439 +08:00] [INFO] [base_client.go:296] ["[pd] update member urls"] [old-urls="[http://192.168.232.2:2379]"] [new-urls="[http://localhost:2379]"]

[2020/12/24 16:34:17.439 +08:00] [INFO] [base_client.go:308] ["[pd] switch leader"] [new-leader=http://localhost:2379] [old-leader=]

[2020/12/24 16:34:17.439 +08:00] [INFO] [base_client.go:112] ["[pd] init cluster id"] [cluster-id=6909738232910158800]

[2020/12/24 16:34:17.448 +08:00] [INFO] [pd.go:362] ["pause scheduler successful at beginning"] [name="[balance-region-scheduler,balance-leader-scheduler,balance-hot-region-scheduler]"]

[2020/12/24 16:34:17.450 +08:00] [INFO] [pd.go:370] ["pause configs successful at beginning"] [cfg="{"enable-location-replacement":"false","leader-schedule-limit":4,"max-merge-region-keys":0,"max-merge-region-size":0,"max-pending-peer-count":2147483647,"max-snapshot-count":3,"region-schedule-limit":40}"]

[2020/12/24 16:34:17.451 +08:00] [INFO] [client.go:193] ["[pd] create pd client with endpoints"] [pd-address="[192.168.232.2:2379]"]

[2020/12/24 16:34:17.454 +08:00] [INFO] [base_client.go:296] ["[pd] update member urls"] [old-urls="[http://192.168.232.2:2379]"] [new-urls="[http://localhost:2379]"]

[2020/12/24 16:34:17.454 +08:00] [INFO] [base_client.go:308] ["[pd] switch leader"] [new-leader=http://localhost:2379] [old-leader=]

[2020/12/24 16:34:17.454 +08:00] [INFO] [base_client.go:112] ["[pd] init cluster id"] [cluster-id=6909738232910158800]

[2020/12/24 16:34:17.454 +08:00] [INFO] [client.go:193] ["[pd] create pd client with endpoints"] [pd-address="[192.168.232.2:2379]"]

[2020/12/24 16:34:17.457 +08:00] [INFO] [base_client.go:296] ["[pd] update member urls"] [old-urls="[http://192.168.232.2:2379]"] [new-urls="[http://localhost:2379]"]

[2020/12/24 16:34:17.457 +08:00] [INFO] [base_client.go:308] ["[pd] switch leader"] [new-leader=http://localhost:2379] [old-leader=]

[2020/12/24 16:34:17.457 +08:00] [INFO] [base_client.go:112] ["[pd] init cluster id"] [cluster-id=6909738232910158800]

unexpected fault address 0xffffffffffffffff

fatal error: fault

[signal 0xc0000005 code=0x0 addr=0xffffffffffffffff pc=0x21f2581]
goroutine 840 [running]:

runtime.throw(0x2bc14a3, 0x5)

C:/Program Files/Go/src/runtime/panic.go:1116 +0x79 fp=0xc0033169c8 sp=0xc003316998 pc=0xa9b939

runtime.sigpanic()

C:/Program Files/Go/src/runtime/signal_windows.go:249 +0x24f fp=0xc0033169f8 sp=0xc0033169c8 pc=0xab06af

encoding/binary.littleEndian.Uint32(...)

C:/Program Files/Go/src/encoding/binary/binary.go:65

github.com/cockroachdb/pebble/sstable.(*blockIter).init(0xc000c48670, 0x2cff2a0, 0x6d2f73726573552f, 0x6f72502f61697872, 0x65742f737463656a, 0x0, 0x220e7f4, 0x2b05ee0)

C:/Users/mrxia/go/pkg/mod/github.com/cockroachdb/[email protected]/sstable/block.go:259 +0x61 fp=0xc003316a40 sp=0xc0033169f8 pc=0x21f2581

github.com/cockroachdb/pebble/sstable.(*blockIter).initHandle(0xc000c48670, 0x2cff2a0, 0xc000f76700, 0x0, 0x0, 0xc000f76700)

C:/Users/mrxia/go/pkg/mod/github.com/cockroachdb/[email protected]/sstable/block.go:286 +0xd5 fp=0xc003316a90 sp=0xc003316a40 pc=0x21f2855

github.com/cockroachdb/pebble/sstable.(*singleLevelIterator).init(0xc000c48600, 0xc000f56500, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)

C:/Users/mrxia/go/pkg/mod/github.com/cockroachdb/[email protected]/sstable/reader.go:222 +0x125 fp=0xc003316ae0 sp=0xc003316a90 pc=0x21fa0e5

github.com/cockroachdb/pebble/sstable.(*Reader).NewIter(0xc000f56500, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)

C:/Users/mrxia/go/pkg/mod/github.com/cockroachdb/[email protected]/sstable/reader.go:1726 +0x1b8 fp=0xc003316b48 sp=0xc003316ae0 pc=0x22010f8

github.com/cockroachdb/pebble.(*tableCacheShard).newIters(0xc000699100, 0xc00065c000, 0xc0000e1320, 0xc0000e1320, 0x0, 0xc00107b5d0, 0x0, 0x0, 0x0, 0x0, ...)

C:/Users/mrxia/go/pkg/mod/github.com/cockroachdb/[email protected]/table_cache.go:208 +0x25d fp=0xc003316bf0 sp=0xc003316b48 pc=0x2270abd

github.com/cockroachdb/pebble.(*tableCache).newIters(0xc000201cc0, 0xc00065c000, 0xc0000e1320, 0xc0000e1320, 0x0, 0xc00107b5d0, 0x0, 0x0, 0x0, 0x0, ...)

C:/Users/mrxia/go/pkg/mod/github.com/cockroachdb/[email protected]/table_cache.go:56 +0xb8 fp=0xc003316cd8 sp=0xc003316bf0 pc=0x226ff78

github.com/cockroachdb/pebble.(*tableCache).newIters-fm(0xc00065c000, 0xc0000e1320, 0xc0000e1320, 0x0, 0xc00107b5d0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)

C:/Users/mrxia/go/pkg/mod/github.com/cockroachdb/[email protected]/table_cache.go:53 +0x85 fp=0xc003316dc0 sp=0xc003316cd8 pc=0x228f385

github.com/cockroachdb/pebble.(*levelIter).loadFile(0xc0024f6908, 0xc00065c000, 0x1, 0x30)

C:/Users/mrxia/go/pkg/mod/github.com/cockroachdb/[email protected]/level_iter.go:324 +0x2ec fp=0xc003316ff8 sp=0xc003316dc0 pc=0x225cdec

github.com/cockroachdb/pebble.(*levelIter).SeekGE(0xc0024f6908, 0xc001716810, 0x25, 0x30, 0xda94, 0xc006a7ba74, 0x0, 0x0)

C:/Users/mrxia/go/pkg/mod/github.com/cockroachdb/[email protected]/level_iter.go:375 +0xa8 fp=0xc003317048 sp=0xc003316ff8 pc=0x225d128

github.com/cockroachdb/pebble.(*mergingIter).seekGE(0xc0024f6168, 0xc001716810, 0x25, 0x30, 0x0)

C:/Users/mrxia/go/pkg/mod/github.com/cockroachdb/[email protected]/merging_iter.go:808 +0x576 fp=0xc003317180 sp=0xc003317048 pc=0x2263656

github.com/cockroachdb/pebble.(*mergingIter).SeekGE(0xc0024f6168, 0xc001716810, 0x25, 0x30, 0xc0024f6000, 0x0, 0x0, 0x0)

C:/Users/mrxia/go/pkg/mod/github.com/cockroachdb/[email protected]/merging_iter.go:863 +0x96 fp=0xc0033171b8 sp=0xc003317180 pc=0x22637b6

github.com/cockroachdb/pebble.(*Iterator).First(0xc0024f6000, 0x0)

C:/Users/mrxia/go/pkg/mod/github.com/cockroachdb/[email protected]/iterator.go:422 +0xaa fp=0xc003317208 sp=0xc0033171b8 pc=0x225502a

github.com/pingcap/tidb-lightning/lightning/backend.(*local).writeAndIngestByRange(0xc0001ab5c0, 0x2fd3560, 0xc000d5f000, 0xc000f5e300, 0xc001716810, 0x25, 0x30, 0xc001716870, 0x26, 0x26, ...)

C:/Users/mrxia/Projects/tidb-lightning/lightning/backend/local.go:879 +0x15d fp=0xc003317de8 sp=0xc003317208 pc=0x260843d

github.com/pingcap/tidb-lightning/lightning/backend.(*local).writeAndIngestByRanges.func1(0xc0001ab5c0, 0xc000c86c40, 0xc000f5e300, 0xc001716810, 0x25, 0x30, 0xc001716870, 0x26, 0x26, 0xc000383860, ...)

C:/Users/mrxia/Projects/tidb-lightning/lightning/backend/local.go:1057 +0x199 fp=0xc003317f80 sp=0xc003317de8 pc=0x2625279

runtime.goexit()

C:/Program Files/Go/src/runtime/asm_amd64.s:1374 +0x1 fp=0xc003317f88 sp=0xc003317f80 pc=0xad2b81

created by github.com/pingcap/tidb-lightning/lightning/backend.(*local).writeAndIngestByRanges

C:/Users/mrxia/Projects/tidb-lightning/lightning/backend/local.go:1053 +0x469

4. Versions of the cluster
   master.

This problem is very likely caused by memory corruption in pebble. See cockroachdb/pebble#1011. I test with the latest code of pebble. It seems this bug is fixed.