Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix issue when row contains an empty nested array #161

Merged
merged 1 commit into from
Aug 14, 2020

Conversation

dourvaris
Copy link
Contributor

@dourvaris dourvaris commented Aug 12, 2020

Hi, I'd appreciate a review on this as I'm not certain correct approach was used to solve it.

This PR fixes an issue where clickhouse driver is raising a NotImplemented error when all rows in result set are empty nested arrays causing the following exception:

    175         data = nested_column._read_data(
    176             nested_column_size, buf,
--> 177             nulls_map=nulls_map
    178         )
    179

/usr/local/lib/python3.7/site-packages/clickhouse_driver/columns/base.py in _read_data(self, n_items, buf, nulls_map)
     93
     94     def _read_data(self, n_items, buf, nulls_map=None):
---> 95         items = self.read_items(n_items, buf)
     96
     97         if self.after_read_items:

/usr/local/lib/python3.7/site-packages/clickhouse_driver/columns/base.py in read_items(self, n_items, buf)
    105
    106     def read_items(self, n_items, buf):
--> 107         raise NotImplementedError
    108
    109     def read_state_prefix(self, buf):

NotImplementedError:

After the fix the queries work and match output from clickhouse:

Clickhouse client

Debian-99-stretch-64-minimal :) CREATE TABLE test (id Int32, a Array(Array(Array(Int32)))) ENGINE MergeTree ORDER BY id;

CREATE TABLE test
(
    `id` Int32,
    `a` Array(Array(Array(Int32)))
)
ENGINE = MergeTree
ORDER BY id

Ok.

0 rows in set. Elapsed: 0.016 sec.

Debian-99-stretch-64-minimal :) INSERT INTO test VALUES
:-]     (1, []),
:-]     (2, [[]]),
:-]     (3, [[[]]]),
:-]     (4, [[[],[]]]),
:-]     (5, [[[],[]],[]]),
:-]     (6, [[[],[]],[[]]]),
:-]     (7, [[[],[]],[[],[]]]),
:-]     (8, [[[1],[2]],[[],[]]])
:-]     (9, [[[1,2],[2,3]],[[],[]]])
:-] ,;

INSERT INTO test VALUES

Ok.

9 rows in set. Elapsed: 0.005 sec.

Debian-99-stretch-64-minimal :) SELECT * FROM test;

SELECT *
FROM test

┌─id─┬─a───────────────────────┐
│  1 │ []                      │
│  2 │ [[]]                    │
│  3 │ [[[]]]                  │
│  4 │ [[[],[]]]               │
│  5 │ [[[],[]],[]]            │
│  6 │ [[[],[]],[[]]]          │
│  7 │ [[[],[]],[[],[]]]       │
│  8 │ [[[1],[2]],[[],[]]]     │
│  9 │ [[[1,2],[2,3]],[[],[]]] │
└────┴─────────────────────────┘

9 rows in set. Elapsed: 0.005 sec.
Debian-99-stretch-64-minimal :) SELECT * FROM test limit 2;

SELECT *
FROM test
LIMIT 2

┌─id─┬─a────┐
│  1 │ []   │
│  2 │ [[]] │
└────┴──────┘

2 rows in set. Elapsed: 0.004 sec.

Python client

In [3]: clickhouse_driver.Client(...).execute('select * from test')
Out[3]: 
[(1, []),
 (2, [[]]),
 (3, [[[]]]),
 (4, [[[], []]]),
 (5, [[[], []], []]),
 (6, [[[], []], [[]]]),
 (7, [[[], []], [[], []]]),
 (8, [[[1], [2]], [[], []]]),
 (9, [[[1, 2], [2, 3]], [[], []]])]

In [4]: clickhouse_driver.Client(...).execute('select * from test limit 2') # this query fails before
Out[4]: [(1, []), (2, [[]])]``` 

@coveralls
Copy link

Coverage Status

Coverage decreased (-0.03%) to 96.533% when pulling 503bbc4 on dourvaris:fix/empty-nested-array into d80c330 on mymarilyn:master.

@xzkostyan
Copy link
Member

PR looks good to me.

Thanks.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants