terminate called without an active exception

[vderic]

Environment

Delta-rs version: 0.15.1

Binding: python

Environment:

• Cloud provider: local machine
• OS: ubuntu 18.1
• Other:

---

Bug

What happened:
write_deltalake() return and program exit but the thread does not finish yet.

It will have segfault "terminate called without an active exception" or no result found right after write_deltalake()

What you expected to happen:

write_deltalake() shall wait for all threads done before return.

How to reproduce it:
from deltalake import DeltaTable, write_deltalake
from deltalake.exceptions import DeltaError, DeltaProtocolError, TableNotFoundError, CommitFailedError
import pyarrow as pa
import sys

if name == "main":

try:
    table_uri = "kvdb"
    ids = pa.array([1,2,3,4], pa.int64())
    vectors = pa.array([[1.0,3,4,2],[1.0,3,4,5], [3.0,5,6,7], [4.0,5,68,6]], pa.list_(pa.float32()))
    names = ["id", "vector"]
    data = pa.Table.from_arrays([ids, vectors], names = names)
    write_deltalake(table_uri, data, mode='append')
    dt = DeltaTable(table_uri)
    print(dt.to_pandas())

except DeltaError as e:
    print(e)

More details:

multiple call of the db_bug.py will have segfault. Just like below:

$ python3 db_bug.py
id vector
0 1 [1.0, 3.0, 4.0, 2.0]
1 2 [1.0, 3.0, 4.0, 5.0]
2 3 [3.0, 5.0, 6.0, 7.0]
3 4 [4.0, 5.0, 68.0, 6.0]
4 1 [1.0, 3.0, 4.0, 2.0]
5 2 [1.0, 3.0, 4.0, 5.0]
6 3 [3.0, 5.0, 6.0, 7.0]
7 4 [4.0, 5.0, 68.0, 6.0]
$ python3 db_bug.py
id vector
0 1 [1.0, 3.0, 4.0, 2.0]
1 2 [1.0, 3.0, 4.0, 5.0]
2 3 [3.0, 5.0, 6.0, 7.0]
3 4 [4.0, 5.0, 68.0, 6.0]
4 1 [1.0, 3.0, 4.0, 2.0]
5 2 [1.0, 3.0, 4.0, 5.0]
6 3 [3.0, 5.0, 6.0, 7.0]
7 4 [4.0, 5.0, 68.0, 6.0]
8 1 [1.0, 3.0, 4.0, 2.0]
9 2 [1.0, 3.0, 4.0, 5.0]
10 3 [3.0, 5.0, 6.0, 7.0]
11 4 [4.0, 5.0, 68.0, 6.0]
$ python3 db_bug.py
id vector
0 1 [1.0, 3.0, 4.0, 2.0]
1 2 [1.0, 3.0, 4.0, 5.0]
2 3 [3.0, 5.0, 6.0, 7.0]
3 4 [4.0, 5.0, 68.0, 6.0]
4 1 [1.0, 3.0, 4.0, 2.0]
5 2 [1.0, 3.0, 4.0, 5.0]
6 3 [3.0, 5.0, 6.0, 7.0]
7 4 [4.0, 5.0, 68.0, 6.0]
8 1 [1.0, 3.0, 4.0, 2.0]
9 2 [1.0, 3.0, 4.0, 5.0]
10 3 [3.0, 5.0, 6.0, 7.0]
11 4 [4.0, 5.0, 68.0, 6.0]
12 1 [1.0, 3.0, 4.0, 2.0]
13 2 [1.0, 3.0, 4.0, 5.0]
14 3 [3.0, 5.0, 6.0, 7.0]
15 4 [4.0, 5.0, 68.0, 6.0]
$ python3 db_bug.py
id vector
0 1 [1.0, 3.0, 4.0, 2.0]
1 2 [1.0, 3.0, 4.0, 5.0]
2 3 [3.0, 5.0, 6.0, 7.0]
3 4 [4.0, 5.0, 68.0, 6.0]
4 1 [1.0, 3.0, 4.0, 2.0]
5 2 [1.0, 3.0, 4.0, 5.0]
6 3 [3.0, 5.0, 6.0, 7.0]
7 4 [4.0, 5.0, 68.0, 6.0]
8 1 [1.0, 3.0, 4.0, 2.0]
9 2 [1.0, 3.0, 4.0, 5.0]
10 3 [3.0, 5.0, 6.0, 7.0]
11 4 [4.0, 5.0, 68.0, 6.0]
12 1 [1.0, 3.0, 4.0, 2.0]
13 2 [1.0, 3.0, 4.0, 5.0]
14 3 [3.0, 5.0, 6.0, 7.0]
15 4 [4.0, 5.0, 68.0, 6.0]
16 1 [1.0, 3.0, 4.0, 2.0]
17 2 [1.0, 3.0, 4.0, 5.0]
18 3 [3.0, 5.0, 6.0, 7.0]
19 4 [4.0, 5.0, 68.0, 6.0]
$ python3 db_bug.py
id vector
0 1 [1.0, 3.0, 4.0, 2.0]
1 2 [1.0, 3.0, 4.0, 5.0]
2 3 [3.0, 5.0, 6.0, 7.0]
3 4 [4.0, 5.0, 68.0, 6.0]
4 1 [1.0, 3.0, 4.0, 2.0]
5 2 [1.0, 3.0, 4.0, 5.0]
6 3 [3.0, 5.0, 6.0, 7.0]
7 4 [4.0, 5.0, 68.0, 6.0]
8 1 [1.0, 3.0, 4.0, 2.0]
9 2 [1.0, 3.0, 4.0, 5.0]
10 3 [3.0, 5.0, 6.0, 7.0]
11 4 [4.0, 5.0, 68.0, 6.0]
12 1 [1.0, 3.0, 4.0, 2.0]
13 2 [1.0, 3.0, 4.0, 5.0]
14 3 [3.0, 5.0, 6.0, 7.0]
15 4 [4.0, 5.0, 68.0, 6.0]
16 1 [1.0, 3.0, 4.0, 2.0]
17 2 [1.0, 3.0, 4.0, 5.0]
18 3 [3.0, 5.0, 6.0, 7.0]
19 4 [4.0, 5.0, 68.0, 6.0]
20 1 [1.0, 3.0, 4.0, 2.0]
21 2 [1.0, 3.0, 4.0, 5.0]
22 3 [3.0, 5.0, 6.0, 7.0]
23 4 [4.0, 5.0, 68.0, 6.0]
$ python3 db_bug.py
id vector
0 1 [1.0, 3.0, 4.0, 2.0]
1 2 [1.0, 3.0, 4.0, 5.0]
2 3 [3.0, 5.0, 6.0, 7.0]
3 4 [4.0, 5.0, 68.0, 6.0]
4 1 [1.0, 3.0, 4.0, 2.0]
5 2 [1.0, 3.0, 4.0, 5.0]
6 3 [3.0, 5.0, 6.0, 7.0]
7 4 [4.0, 5.0, 68.0, 6.0]
8 1 [1.0, 3.0, 4.0, 2.0]
9 2 [1.0, 3.0, 4.0, 5.0]
10 3 [3.0, 5.0, 6.0, 7.0]
11 4 [4.0, 5.0, 68.0, 6.0]
12 1 [1.0, 3.0, 4.0, 2.0]
13 2 [1.0, 3.0, 4.0, 5.0]
14 3 [3.0, 5.0, 6.0, 7.0]
15 4 [4.0, 5.0, 68.0, 6.0]
16 1 [1.0, 3.0, 4.0, 2.0]
17 2 [1.0, 3.0, 4.0, 5.0]
18 3 [3.0, 5.0, 6.0, 7.0]
19 4 [4.0, 5.0, 68.0, 6.0]
20 1 [1.0, 3.0, 4.0, 2.0]
21 2 [1.0, 3.0, 4.0, 5.0]
22 3 [3.0, 5.0, 6.0, 7.0]
23 4 [4.0, 5.0, 68.0, 6.0]
24 1 [1.0, 3.0, 4.0, 2.0]
25 2 [1.0, 3.0, 4.0, 5.0]
26 3 [3.0, 5.0, 6.0, 7.0]
27 4 [4.0, 5.0, 68.0, 6.0]
$ python3 db_bug.py
id vector
0 1 [1.0, 3.0, 4.0, 2.0]
1 2 [1.0, 3.0, 4.0, 5.0]
2 3 [3.0, 5.0, 6.0, 7.0]
3 4 [4.0, 5.0, 68.0, 6.0]
4 1 [1.0, 3.0, 4.0, 2.0]
5 2 [1.0, 3.0, 4.0, 5.0]
6 3 [3.0, 5.0, 6.0, 7.0]
7 4 [4.0, 5.0, 68.0, 6.0]
8 1 [1.0, 3.0, 4.0, 2.0]
9 2 [1.0, 3.0, 4.0, 5.0]
10 3 [3.0, 5.0, 6.0, 7.0]
11 4 [4.0, 5.0, 68.0, 6.0]
12 1 [1.0, 3.0, 4.0, 2.0]
13 2 [1.0, 3.0, 4.0, 5.0]
14 3 [3.0, 5.0, 6.0, 7.0]
15 4 [4.0, 5.0, 68.0, 6.0]
16 1 [1.0, 3.0, 4.0, 2.0]
17 2 [1.0, 3.0, 4.0, 5.0]
18 3 [3.0, 5.0, 6.0, 7.0]
19 4 [4.0, 5.0, 68.0, 6.0]
20 1 [1.0, 3.0, 4.0, 2.0]
21 2 [1.0, 3.0, 4.0, 5.0]
22 3 [3.0, 5.0, 6.0, 7.0]
23 4 [4.0, 5.0, 68.0, 6.0]
24 1 [1.0, 3.0, 4.0, 2.0]
25 2 [1.0, 3.0, 4.0, 5.0]
26 3 [3.0, 5.0, 6.0, 7.0]
27 4 [4.0, 5.0, 68.0, 6.0]
28 1 [1.0, 3.0, 4.0, 2.0]
29 2 [1.0, 3.0, 4.0, 5.0]
30 3 [3.0, 5.0, 6.0, 7.0]
31 4 [4.0, 5.0, 68.0, 6.0]
terminate called without an active exception
Aborted (core dumped)

[vderic]

with environment variable PYTHONFAULTHANDLER=1,

I got the message below:

terminate called without an active exception
Fatal Python error: Aborted

Thread 0x00007f2cce309680 (most recent call first):

Extension modules: numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, pyarrow.lib, pyarrow._hdfsio, pyarrow._compute, pyarrow._acero, pyarrow._fs, pyarrow._csv, pyarrow._json, pandas._libs.tslibs.np_datetime, pandas._libs.tslibs.dtypes, pandas._libs.tslibs.base, pandas._libs.tslibs.nattype, pandas._libs.tslibs.timezones, pandas._libs.tslibs.ccalendar, pandas._libs.tslibs.fields, pandas._libs.tslibs.timedeltas, pandas._libs.tslibs.tzconversion, pandas._libs.tslibs.timestamps, pandas._libs.properties, pandas._libs.tslibs.offsets, pandas._libs.tslibs.strptime, pandas._libs.tslibs.parsing, pandas._libs.tslibs.conversion, pandas._libs.tslibs.period, pandas._libs.tslibs.vectorized, pandas._libs.ops_dispatch, pandas._libs.missing, pandas._libs.hashtable, pandas._libs.algos, pandas._libs.interval, pandas._libs.lib, pandas._libs.ops, pandas._libs.arrays, pandas._libs.tslib, pandas._libs.sparse, pandas._libs.indexing, pandas._libs.index, pandas._libs.internals, pandas._libs.join, pandas._libs.writers, pandas._libs.window.aggregations, pandas._libs.window.indexers, pandas._libs.reshape, pandas._libs.groupby, pandas._libs.json, pandas._libs.parsers, pandas._libs.testing, pyarrow._dataset, pyarrow._dataset_orc, pyarrow._parquet, pyarrow._dataset_parquet, pyarrow._hdfs, pyarrow._gcsfs, pyarrow._s3fs (total: 66)
Aborted (core dumped)

[ion-elgreco]

Likely related to pandas<->pyarrow and not deltalake, so closing this issue. I also cannot reproduce it locally