SNOW-592647: IndexError: list index out of range in snowflake/connector/result_batch.py:698: IndexError #1145

michaelk1c · 2022-05-19T19:00:52Z

Please answer these questions before submitting your issue. Thanks!

What version of Python are you using?

Python 3.8.12 (default, Oct 20 2021, 21:41:13)
[Clang 13.0.0 (clang-1300.0.29.3)]

What operating system and processor architecture are you using?

macOS-12.2.1-x86_64-i386-64bit

What are the component versions in the environment (pip freeze)?

WARNING: Ignoring invalid distribution -umpy (/Users/mkang/.local/share/virtualenvs/01-built-object-service-gvseloBk/lib/python3.8/site-packages)
WARNING: Ignoring invalid distribution -illow (/Users/mkang/.local/share/virtualenvs/01-built-object-service-gvseloBk/lib/python3.8/site-packages)
WARNING: Ignoring invalid distribution -etworkx (/Users/mkang/.local/share/virtualenvs/01-built-object-service-gvseloBk/lib/python3.8/site-packages)
WARNING: Ignoring invalid distribution -etuptools (/Users/mkang/.local/share/virtualenvs/01-built-object-service-gvseloBk/lib/python3.8/site-packages)
WARNING: Ignoring invalid distribution -bformat (/Users/mkang/.local/share/virtualenvs/01-built-object-service-gvseloBk/lib/python3.8/site-packages)
WARNING: Ignoring invalid distribution -bconvert (/Users/mkang/.local/share/virtualenvs/01-built-object-service-gvseloBk/lib/python3.8/site-packages)
affine==2.3.0
alembic==1.7.6
anyio==3.5.0
appdirs==1.4.4
appnope==0.1.3
argon2-cffi==21.3.0
argon2-cffi-bindings==21.2.0
asn1crypto==1.4.0
astroid==2.5.6
asttokens==2.0.5
attrs==21.4.0
autopep8==1.6.0
Babel==2.9.1
backcall==0.2.0
beautifulsoup4==4.10.0
black==20.8b1
bleach==4.1.0
cachetools==5.0.0
certifi==2021.10.8
cffi==1.15.0
cftime==1.5.2
chardet==4.0.0
charset-normalizer==2.0.12
click==8.0.3
click-plugins==1.1.1
cligj==0.7.2
contextily==1.2.0
coverage==6.3.1
coveralls==3.3.1
cryptography==36.0.1
cycler==0.11.0
Cython==0.29.27
datacomp @ git+https://github.com/kappamaki/datacomp@4ef24b16ff96fb06edcba6fdd85d8b7a4da31b0c
debugpy==1.5.1
decorator==5.1.1
defusedxml==0.7.1
diff-cover==6.4.4
distlib==0.3.4
docopt==0.6.2
entrypoints==0.4
et-xmlfile==1.1.0
executing==0.8.2
fastjsonschema==2.15.3
filelock==3.6.0
Fiona==1.8.21
Flask==2.0.2
fonttools==4.29.1
GeoAlchemy2==0.10.2
geographiclib==1.52
geopandas==0.10.2
geopy==2.2.0
google-api-core==2.7.2
google-auth==2.6.4
google-cloud-core==2.3.0
google-cloud-storage==2.2.1
google-crc32c==1.3.0
google-resumable-media==2.3.2
googleapis-common-protos==1.56.0
greenlet==1.1.2
gunicorn==20.1.0
httplib2==0.20.4
idna==3.3
importlib-metadata==4.11.0
importlib-resources==5.4.0
iniconfig==1.1.1
ipykernel==6.9.0
ipython==8.0.1
ipython-genutils==0.2.0
isort==5.10.1
itsdangerous==2.0.1
jedi==0.18.1
jellyfish==0.6.1
Jinja2==3.0.3
joblib==1.1.0
json5==0.9.6
jsonschema==4.4.0
jupyter-client==7.1.2
jupyter-core==4.9.1
jupyter-server==1.13.5
jupyterlab==3.2.9
jupyterlab-pygments==0.1.2
jupyterlab-server==2.10.3
kiwisolver==1.3.2
lazy-object-proxy==1.7.1
Mako==1.1.6
MarkupSafe==2.0.1
matplotlib==3.5.1
matplotlib-inline==0.1.3
mccabe==0.6.1
mercantile==1.2.1
mistune==0.8.4
munch==2.5.0
mypy==0.931
mypy-extensions==0.4.3
nbclassic==0.3.5
nbclient==0.5.10
nbconvert==6.4.2
nbformat==5.1.3
nest-asyncio==1.5.4
netCDF4==1.5.8
networkx==2.6.3
notebook==6.4.8
notebook-shim==0.1.0
numpy==1.22.2
oauth2client==4.1.3
openlocationcode==1.0.1
openpyxl==3.0.9
oscrypto==1.2.1
osmnx==1.1.2
packaging==21.3
pandas==1.3.5
pandocfilters==1.5.0
parso==0.8.3
pathspec==0.9.0
pexpect==4.8.0
pickleshare==0.7.5
Pillow==9.0.1
pipenv==2022.4.8
platformdirs==2.5.1
pluggy==1.0.0
-e git+ssh://[email protected]/oneconcern/plus-code-plus.git@2c4c926db58115acc1bac297cc3655cd0b1043d0#egg=pluscodeplus
prometheus-client==0.13.1
prompt-toolkit==3.0.28
protobuf==3.20.0
psutil==5.9.0
psycopg2==2.9.3
ptyprocess==0.7.0
pure-eval==0.2.2
py==1.11.0
py-cpuinfo==8.0.0
pyarrow==6.0.1
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycodestyle==2.8.0
pycparser==2.21
pycryptodomex==3.14.1
Pygments==2.11.2
PyJWT==2.3.0
pylint==2.8.3
pyOpenSSL==21.0.0
pyparsing==3.0.7
pyproj==3.3.0
pyrsistent==0.18.1
pytest==7.0.1
pytest-benchmark==3.4.1
pytest-cov==3.0.0
python-dateutil==2.8.2
python-editor==1.0.4
pytz==2021.3
PyYAML==5.4.1
pyzmq==22.3.0
rasterio==1.2.10
regex==2022.1.18
requests==2.27.1
rsa==4.8
Rtree==0.9.7
scikit-learn==0.24.2
scipy==1.8.0
-e git+ssh://[email protected]/oneconcern/seismic-resilience.git@2f30638811014fb49e791481cc04a102a594c882#egg=seismic_resilience
Send2Trash==1.8.0
Shapely==1.8.0
six==1.16.0
sniffio==1.2.0
snowflake-connector-python==2.7.4
snowflake-sqlalchemy==1.3.3
snuggs==1.4.7
soupsieve==2.3.1
SQLAlchemy==1.4.31
stack-data==0.1.4
structlog==21.5.0
tenacity==8.0.1
terminado==0.13.1
testpath==0.5.0
threadpoolctl==3.1.0
tinycss2==1.1.1
toml==0.10.2
tomli==2.0.1
tornado==6.1
tqdm==4.62.3
traitlets==5.1.1
typed-ast==1.5.2
typing_extensions==4.0.1
urllib3==1.26.8
us==2.0.2
virtualenv==20.14.1
virtualenv-clone==0.5.7
wcwidth==0.2.5
webencodings==0.5.1
websocket-client==1.2.3
Werkzeug==2.0.3
wget==3.2
wrapt==1.12.1
xlrd==1.2.0
xyzservices==2022.3.0
zipp==3.7.0

What did you do?

If possible, provide a recipe for reproducing the error.
A complete runnable program is good.

I was loading data with the following code for the snowflake table with lnglat GEOGRAPHY type in it.

        with db_engine.begin() as conn:
            stmt = f"""
                        SELECT {sql_columns} FROM {table_name};
                    """
            res = conn.execute(stmt)
            df_data = res.cursor.fetch_pandas_all()

Here is the complete error list:

(01-built-object-service) ➜  01-built-object-service (foo) ✔ pytest -vsm focus      
PG HOST: localhost
PG NAME: built-object-service
PG USER: postgres
PG PORT: 5432
===================================================================================================== test session starts =====================================================================================================
platform darwin -- Python 3.8.12, pytest-7.0.1, pluggy-1.0.0 -- /Users/mkang/.local/share/virtualenvs/01-built-object-service-gvseloBk/bin/python
cachedir: .pytest_cache
benchmark: 3.4.1 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: /Users/mkang/occ/prod/01-built-object-service, configfile: setup.cfg
plugins: benchmark-3.4.1, anyio-3.5.0, cov-3.0.0
collecting 111 items                                                                                                                                                                                                          Snowflake connection URL: snowflake://ONEC_DEV:***@jbulliw-ug92181/CI_BUILT_OBJECT_SERVICE_TEST/public?role=sysadmin&warehouse=COMPUTE_WH
collected 158 items / 157 deselected / 1 selected                                                                                                                                                                             

tests/pipelines/exporter_snowflake/test_exporter_snowflake.py::test_snowflake_exporter Copying file://tests/data/airports/jp/expected_output/.meta.yaml [Content-Type=application/octet-stream]...
/ [1 files][  562.0 B/  562.0 B]                                                
Operation completed over 1 objects/562.0 B.                                      
Copying file://tests/data/substations/jp/exp_sub_output/substations.parquet [Content-Type=application/octet-stream]...
- [1 files][786.2 KiB/786.2 KiB]                                                
Operation completed over 1 objects/786.2 KiB.                                    
2022-05-12 21:46:44,835 - MainProcess - src.pipelines.exporter_snowflake.__main__ - INFO - parsed command line args: Namespace(export_list_yaml_file_path=PosixPath('tests/data/exporter_snowflake/exporter_snowflake_list_test_samples.yaml'), input_gcs_bucket=PosixPath('built-object-service-dev'))
2022-05-12 21:46:44,854 - MainProcess - src.pipelines.exporter_snowflake.__main__ - INFO - >>> exporting substations to snowflake from gcs://built-object-service-dev/substations/jp/v0.0.0
Copying gs://built-object-service-dev/substations/jp/v0.0.0/.meta.yaml...
/ [1 files][  562.0 B/  562.0 B]                                                
Operation completed over 1 objects/562.0 B.                                      
{'uid': 'c6fac755-52b5-4868-971a-308406d4660d', 'namespace': 'built-object-service', 'name': 'airports-importer-jlsq6', 'serviceAccountName': 'built-object-service', 'creationTimestamp': '2022-04-21T17:48:22Z', 'parameters': [{'name': 'gcs-bucket', 'value': 'built-object-service-dev'}, {'name': 'input-datamon-bundle', 'value': '27zpshJwQDvUvDkqDvE1u4Vr64v'}, {'name': 'input-datamon-context', 'value': 'dev'}, {'name': 'input-datamon-repo', 'value': 'airport-JP'}, {'name': 'version', 'value': 'v1.0.0'}, {'name': 'docker-image-tag', 'value': 'v2.5.0-staging-3-gc29adea'}, {'name': 'datamon-config', 'value': 'global-onec-co-datamon-config'}]}
2022-05-12 21:46:49,769 - MainProcess - snowflake.connector.connection - INFO - Snowflake Connector for Python Version: 2.7.4, Python Version: 3.8.12, Platform: macOS-12.2.1-x86_64-i386-64bit
2022-05-12 21:46:49,769 - MainProcess - snowflake.connector.connection - INFO - This connection is in OCSP Fail Open Mode. TLS Certificates would be checked for validity and revocation status. Any other Certificate Revocation related exceptions or OCSP Responder failures would be disregarded in favor of connectivity.
2022-05-12 21:46:49,769 - MainProcess - snowflake.connector.connection - INFO - Setting use_openssl_only mode to False
2022-05-12 21:46:50,740 - MainProcess - snowflake.connector.cursor - INFO - query: [ROLLBACK]
2022-05-12 21:46:50,863 - MainProcess - snowflake.connector.cursor - INFO - query execution done
##############################
[JSONResultChunk(1)]
result_chunks: [JSONResultChunk(1)]
##############################
2022-05-12 21:46:50,864 - MainProcess - snowflake.connector.cursor - INFO - query: [CREATE OR REPLACE TABLE temp_table ( workflow_uid                        TEXT, w...]
2022-05-12 21:46:51,175 - MainProcess - snowflake.connector.cursor - INFO - query execution done
##############################
[JSONResultChunk(1)]
result_chunks: [JSONResultChunk(1)]
##############################
2022-05-12 21:46:51,176 - MainProcess - snowflake.connector.cursor - INFO - query: [INSERT INTO temp_table (workflow_uid, workflow_name, workflow_input, meta, targe...]
2022-05-12 21:46:52,404 - MainProcess - snowflake.connector.cursor - INFO - query execution done
##############################
[JSONResultChunk(1)]
result_chunks: [JSONResultChunk(1)]
##############################
2022-05-12 21:46:52,405 - MainProcess - snowflake.connector.cursor - INFO - query: [MERGE INTO data_events USING temp_table ON data_events.workflow_uid = temp_table...]
2022-05-12 21:46:53,736 - MainProcess - snowflake.connector.cursor - INFO - query execution done
##############################
[JSONResultChunk(1)]
result_chunks: [JSONResultChunk(1)]
##############################
2022-05-12 21:46:53,737 - MainProcess - snowflake.connector.cursor - INFO - query: [drop table temp_table;]
2022-05-12 21:46:54,135 - MainProcess - snowflake.connector.cursor - INFO - query execution done
##############################
[JSONResultChunk(1)]
result_chunks: [JSONResultChunk(1)]
##############################
2022-05-12 21:46:54,136 - MainProcess - snowflake.connector.cursor - INFO - query: [SELECT id FROM data_events WHERE workflow_uid = 'c6fac755-52b5-4868-971a-308406d...]
2022-05-12 21:46:55,066 - MainProcess - snowflake.connector.cursor - INFO - query execution done
##############################
[ArrowResultChunk(1)]
result_chunks: [ArrowResultChunk(1)]
##############################
2022-05-12 21:46:55,067 - MainProcess - src.pipelines.exporter_snowflake.data_events - INFO - data_event_id: 1
2022-05-12 21:46:55,067 - MainProcess - snowflake.connector.cursor - INFO - query: [COMMIT]
2022-05-12 21:46:55,186 - MainProcess - snowflake.connector.cursor - INFO - query execution done
##############################
[JSONResultChunk(1)]
result_chunks: [JSONResultChunk(1)]
##############################
2022-05-12 21:46:55,187 - MainProcess - snowflake.connector.cursor - INFO - query: [ROLLBACK]
2022-05-12 21:46:55,304 - MainProcess - snowflake.connector.cursor - INFO - query execution done
##############################
[JSONResultChunk(1)]
result_chunks: [JSONResultChunk(1)]
##############################
2022-05-12 21:46:55,305 - MainProcess - src.helpers.snowflake_helper - INFO - creating stage: substations, url: gcs://built-object-service-dev/substations/jp/v0.0.0
2022-05-12 21:46:55,305 - MainProcess - snowflake.connector.cursor - INFO - query: [CREATE OR REPLACE STAGE substations STORAGE_INTEGRATION = DEV_GCS_INT URL = 'gcs...]
2022-05-12 21:46:55,577 - MainProcess - snowflake.connector.cursor - INFO - query execution done
##############################
[JSONResultChunk(1)]
result_chunks: [JSONResultChunk(1)]
##############################
2022-05-12 21:46:55,577 - MainProcess - snowflake.connector.cursor - INFO - query: [ROLLBACK]
2022-05-12 21:46:55,691 - MainProcess - snowflake.connector.cursor - INFO - query execution done
##############################
[JSONResultChunk(1)]
result_chunks: [JSONResultChunk(1)]
##############################
2022-05-12 21:46:55,691 - MainProcess - snowflake.connector.cursor - INFO - query: [SELECT * FROM substations]
2022-05-12 21:46:55,817 - MainProcess - snowflake.connector.cursor - INFO - query execution done
##############################
[ArrowResultChunk(0)]
result_chunks: [ArrowResultChunk(0)]
##############################

        copy into substations (ID,LNGLAT,PLUSCODE,PLUSCODE4,PLUSCODE6,PLUSCODE8,SOURCE_VENDOR,COUNTRY,META,NAME,ADDRESS,ADMINISTRATIVE_AREA_LEVEL_1,INPUT_VOLT,OUTPUT_VOLT,SUBSTATION_TYPE)
            from (
            select $1:id,ST_MakePoint($1:longitude::REAL, $1:latitude::REAL),$1:pluscode,$1:pluscode4,$1:pluscode6,$1:pluscode8,$1:source_vendor,$1:country,$1:meta,$1:name,$1:address,$1:administrative_area_level_1,$1:input_volt,$1:output_volt,$1:substation_type
            from @substations)
            file_format = (type = parquet)
            pattern = '.*.parquet'
            FORCE = false
            ;
        
2022-05-12 21:46:55,817 - MainProcess - snowflake.connector.cursor - INFO - query: [copy into substations (ID,LNGLAT,PLUSCODE,PLUSCODE4,PLUSCODE6,PLUSCODE8,SOURCE_V...]
2022-05-12 21:46:57,220 - MainProcess - snowflake.connector.cursor - INFO - query execution done
##############################
[JSONResultChunk(1)]
result_chunks: [JSONResultChunk(1)]
##############################
2022-05-12 21:46:57,220 - MainProcess - snowflake.connector.cursor - INFO - query: [COMMIT]
2022-05-12 21:46:57,421 - MainProcess - snowflake.connector.cursor - INFO - query execution done
##############################
[JSONResultChunk(1)]
result_chunks: [JSONResultChunk(1)]
##############################
2022-05-12 21:46:57,421 - MainProcess - snowflake.connector.cursor - INFO - query: [ROLLBACK]
2022-05-12 21:46:57,541 - MainProcess - snowflake.connector.cursor - INFO - query execution done
##############################
[JSONResultChunk(1)]
result_chunks: [JSONResultChunk(1)]
##############################
2022-05-12 21:46:57,541 - MainProcess - snowflake.connector.cursor - INFO - query: [UPDATE substations SET data_event_id = 1 WHERE data_event_id = 0;]
2022-05-12 21:46:58,033 - MainProcess - snowflake.connector.cursor - INFO - query execution done
##############################
[JSONResultChunk(1)]
result_chunks: [JSONResultChunk(1)]
##############################
2022-05-12 21:46:58,033 - MainProcess - snowflake.connector.cursor - INFO - query: [COMMIT]
2022-05-12 21:46:58,241 - MainProcess - snowflake.connector.cursor - INFO - query execution done
##############################
[JSONResultChunk(1)]
result_chunks: [JSONResultChunk(1)]
##############################
2022-05-12 21:46:58,242 - MainProcess - snowflake.connector.cursor - INFO - query: [ROLLBACK]
2022-05-12 21:46:58,348 - MainProcess - snowflake.connector.cursor - INFO - query execution done
##############################
[JSONResultChunk(1)]
result_chunks: [JSONResultChunk(1)]
##############################
2022-05-12 21:46:58,348 - MainProcess - src.helpers.snowflake_helper - INFO - dropping stage: substations
2022-05-12 21:46:58,349 - MainProcess - snowflake.connector.cursor - INFO - query: [DROP STAGE substations]
2022-05-12 21:46:58,476 - MainProcess - snowflake.connector.cursor - INFO - query execution done
##############################
[JSONResultChunk(1)]
result_chunks: [JSONResultChunk(1)]
##############################
2022-05-12 21:46:58,476 - MainProcess - snowflake.connector.cursor - INFO - query: [ROLLBACK]
2022-05-12 21:46:58,591 - MainProcess - snowflake.connector.cursor - INFO - query execution done
##############################
[JSONResultChunk(1)]
result_chunks: [JSONResultChunk(1)]
##############################
2022-05-12 21:46:58,592 - MainProcess - src.pipelines.exporter_snowflake.__main__ - INFO - >>> exported substations from gcs://built-object-service-dev/substations/jp/v0.0.0 successfully
2022-05-12 21:46:58,592 - MainProcess - src.pipelines.exporter_snowflake.__main__ - INFO - >>> Export to Snowflake completed..
2022-05-12 21:46:58,592 - MainProcess - snowflake.connector.cursor - INFO - query: [SELECT * FROM substations;]
2022-05-12 21:46:58,858 - MainProcess - snowflake.connector.cursor - INFO - query execution done
##############################
[ArrowResultChunk(0), ArrowResultChunk(data_0_0_0), ArrowResultChunk(data_0_0_1), ArrowResultChunk(data_0_0_2), ArrowResultChunk(data_0_0_3)]
result_chunks: [ArrowResultChunk(0), ArrowResultChunk(data_0_0_0), ArrowResultChunk(data_0_0_1), ArrowResultChunk(data_0_0_2), ArrowResultChunk(data_0_0_3)]
##############################
s.name: ID s.type_code: 2 s: ResultMetadata(name='ID', type_code=2, display_size=None, internal_size=16777216, precision=None, scale=None, is_nullable=False)
s.name: LNGLAT s.type_code: 14 s: ResultMetadata(name='LNGLAT', type_code=14, display_size=None, internal_size=None, precision=None, scale=None, is_nullable=True)
s.name: PLUSCODE s.type_code: 2 s: ResultMetadata(name='PLUSCODE', type_code=2, display_size=None, internal_size=16777216, precision=None, scale=None, is_nullable=False)
s.name: PLUSCODE4 s.type_code: 2 s: ResultMetadata(name='PLUSCODE4', type_code=2, display_size=None, internal_size=16777216, precision=None, scale=None, is_nullable=False)
s.name: PLUSCODE6 s.type_code: 2 s: ResultMetadata(name='PLUSCODE6', type_code=2, display_size=None, internal_size=16777216, precision=None, scale=None, is_nullable=False)
s.name: PLUSCODE8 s.type_code: 2 s: ResultMetadata(name='PLUSCODE8', type_code=2, display_size=None, internal_size=16777216, precision=None, scale=None, is_nullable=False)
s.name: SOURCE_VENDOR s.type_code: 2 s: ResultMetadata(name='SOURCE_VENDOR', type_code=2, display_size=None, internal_size=16777216, precision=None, scale=None, is_nullable=False)
s.name: COUNTRY s.type_code: 2 s: ResultMetadata(name='COUNTRY', type_code=2, display_size=None, internal_size=16777216, precision=None, scale=None, is_nullable=False)
s.name: META s.type_code: 9 s: ResultMetadata(name='META', type_code=9, display_size=None, internal_size=None, precision=None, scale=None, is_nullable=True)
s.name: NAME s.type_code: 2 s: ResultMetadata(name='NAME', type_code=2, display_size=None, internal_size=16777216, precision=None, scale=None, is_nullable=False)
s.name: ADDRESS s.type_code: 9 s: ResultMetadata(name='ADDRESS', type_code=9, display_size=None, internal_size=None, precision=None, scale=None, is_nullable=True)
s.name: ADMINISTRATIVE_AREA_LEVEL_1 s.type_code: 2 s: ResultMetadata(name='ADMINISTRATIVE_AREA_LEVEL_1', type_code=2, display_size=None, internal_size=16777216, precision=None, scale=None, is_nullable=True)
s.name: INPUT_VOLT s.type_code: 1 s: ResultMetadata(name='INPUT_VOLT', type_code=1, display_size=None, internal_size=None, precision=None, scale=None, is_nullable=True)
s.name: OUTPUT_VOLT s.type_code: 1 s: ResultMetadata(name='OUTPUT_VOLT', type_code=1, display_size=None, internal_size=None, precision=None, scale=None, is_nullable=True)
s.name: SUBSTATION_TYPE s.type_code: 2 s: ResultMetadata(name='SUBSTATION_TYPE', type_code=2, display_size=None, internal_size=16777216, precision=None, scale=None, is_nullable=True)
s.name: DATA_EVENT_ID s.type_code: 0 s: ResultMetadata(name='DATA_EVENT_ID', type_code=0, display_size=None, internal_size=None, precision=38, scale=0, is_nullable=False)
s.name: CREATED_AT s.type_code: 8 s: ResultMetadata(name='CREATED_AT', type_code=8, display_size=None, internal_size=None, precision=0, scale=9, is_nullable=False)
2022-05-12 21:46:58,859 - MainProcess - snowflake.connector.cursor - INFO - query: [ROLLBACK]
2022-05-12 21:46:58,970 - MainProcess - snowflake.connector.cursor - INFO - query execution done
##############################
[JSONResultChunk(1)]
result_chunks: [JSONResultChunk(1)]
##############################
2022-05-12 21:46:58,970 - MainProcess - snowflake.connector.cursor - INFO - query: [ROLLBACK]
2022-05-12 21:46:59,089 - MainProcess - snowflake.connector.cursor - INFO - query execution done
##############################
[JSONResultChunk(1)]
result_chunks: [JSONResultChunk(1)]
##############################
FAILED

========================================================================================================== FAILURES ===========================================================================================================
___________________________________________________________________________________________________ test_snowflake_exporter ___________________________________________________________________________________________________

    @pytest.mark.focus
    def test_snowflake_exporter():
        bucket = "built-object-service-dev"  # using dev bucket for our test, keep it simple for now
        list_yaml_path = "tests/data/exporter_snowflake/exporter_snowflake_list_test_samples.yaml"
        meta_path = "tests/data/airports/jp/expected_output/.meta.yaml"
        with open(list_yaml_path) as file:
            export_list = yaml.full_load(file)
    
        # upload to gcs bucket
        for bucket_path, _, parquet_path in export_list:
            cmd = f"gsutil cp {meta_path} gs://{bucket}/{bucket_path}/"
            os.system(cmd)
            cmd2 = f"gsutil cp -r {parquet_path} gs://{bucket}/{bucket_path}"
            os.system(cmd2)
    
        main(
            [
                "--input-gcs-bucket",
                bucket,
                "--export-list-yaml-file-path",
                list_yaml_path,
            ]
        )
    
        for bucket_path, table_name, parquet_path in export_list:
            # delete the data in gcsbucket
            cmd = f"gsutil rm -rf gs://{bucket}/{bucket_path}"
            # os.system(cmd)
            parquet_path = Path(parquet_path)
    
            # actual dataframe
>           dfa = get_snowflake_dataframe(
                table_name,
            )

tests/pipelines/exporter_snowflake/test_exporter_snowflake.py:43: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
src/helpers/snowflake_helper.py:122: in get_snowflake_dataframe
    df_data = res.cursor.fetch_pandas_all()
../../../.local/share/virtualenvs/01-built-object-service-gvseloBk/lib/python3.8/site-packages/snowflake/connector/cursor.py:998: in fetch_pandas_all
    return self._result_set._fetch_pandas_all(**kwargs)
../../../.local/share/virtualenvs/01-built-object-service-gvseloBk/lib/python3.8/site-packages/snowflake/connector/result_set.py:199: in _fetch_pandas_all
    dataframes = list(self._fetch_pandas_batches())
../../../.local/share/virtualenvs/01-built-object-service-gvseloBk/lib/python3.8/site-packages/snowflake/connector/result_set.py:195: in _fetch_pandas_batches
    return self._create_iter(iter_unit=IterUnit.TABLE_UNIT, structure="pandas")
../../../.local/share/virtualenvs/01-built-object-service-gvseloBk/lib/python3.8/site-packages/snowflake/connector/result_set.py:233: in _create_iter
    first_batch_iter = self.batches[0].create_iter(**kwargs)
../../../.local/share/virtualenvs/01-built-object-service-gvseloBk/lib/python3.8/site-packages/snowflake/connector/result_batch.py:740: in create_iter
    return self._get_pandas_iter(connection=connection, **kwargs)
../../../.local/share/virtualenvs/01-built-object-service-gvseloBk/lib/python3.8/site-packages/snowflake/connector/result_batch.py:722: in _get_pandas_iter
    dataframe = self.to_pandas(connection=connection, **kwargs)
../../../.local/share/virtualenvs/01-built-object-service-gvseloBk/lib/python3.8/site-packages/snowflake/connector/result_batch.py:714: in to_pandas
    table = self.to_arrow(connection=connection)
../../../.local/share/virtualenvs/01-built-object-service-gvseloBk/lib/python3.8/site-packages/snowflake/connector/result_batch.py:707: in to_arrow
    return self._create_empty_table()
../../../.local/share/virtualenvs/01-built-object-service-gvseloBk/lib/python3.8/site-packages/snowflake/connector/result_batch.py:697: in _create_empty_table
    fields = [
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

.0 = <list_iterator object at 0x1466bfc70>

    fields = [
>       field(s.name, FIELD_TYPE_TO_PA_TYPE[s.type_code]) for s in self.schema
    ]
E   IndexError: list index out of range

../../../.local/share/virtualenvs/01-built-object-service-gvseloBk/lib/python3.8/site-packages/snowflake/connector/result_batch.py:698: IndexError
------------------------------------------------------------------------------------------------------ Captured log call ------------------------------------------------------------------------------------------------------
INFO     src.pipelines.exporter_snowflake.__main__:__main__.py:103 parsed command line args: Namespace(export_list_yaml_file_path=PosixPath('tests/data/exporter_snowflake/exporter_snowflake_list_test_samples.yaml'), input_gcs_bucket=PosixPath('built-object-service-dev'))
INFO     src.pipelines.exporter_snowflake.__main__:__main__.py:47 >>> exporting substations to snowflake from gcs://built-object-service-dev/substations/jp/v0.0.0
INFO     snowflake.connector.connection:connection.py:268 Snowflake Connector for Python Version: 2.7.4, Python Version: 3.8.12, Platform: macOS-12.2.1-x86_64-i386-64bit
INFO     snowflake.connector.connection:connection.py:878 This connection is in OCSP Fail Open Mode. TLS Certificates would be checked for validity and revocation status. Any other Certificate Revocation related exceptions or OCSP Responder failures would be disregarded in favor of connectivity.
INFO     snowflake.connector.connection:connection.py:896 Setting use_openssl_only mode to False
INFO     snowflake.connector.cursor:cursor.py:713 query: [ROLLBACK]
INFO     snowflake.connector.cursor:cursor.py:737 query execution done
INFO     snowflake.connector.cursor:cursor.py:713 query: [CREATE OR REPLACE TABLE temp_table ( workflow_uid                        TEXT, w...]
INFO     snowflake.connector.cursor:cursor.py:737 query execution done
INFO     snowflake.connector.cursor:cursor.py:713 query: [INSERT INTO temp_table (workflow_uid, workflow_name, workflow_input, meta, targe...]
INFO     snowflake.connector.cursor:cursor.py:737 query execution done
INFO     snowflake.connector.cursor:cursor.py:713 query: [MERGE INTO data_events USING temp_table ON data_events.workflow_uid = temp_table...]
INFO     snowflake.connector.cursor:cursor.py:737 query execution done
INFO     snowflake.connector.cursor:cursor.py:713 query: [drop table temp_table;]
INFO     snowflake.connector.cursor:cursor.py:737 query execution done
INFO     snowflake.connector.cursor:cursor.py:713 query: [SELECT id FROM data_events WHERE workflow_uid = 'c6fac755-52b5-4868-971a-308406d...]
INFO     snowflake.connector.cursor:cursor.py:737 query execution done
INFO     src.pipelines.exporter_snowflake.data_events:data_events.py:112 data_event_id: 1
INFO     snowflake.connector.cursor:cursor.py:713 query: [COMMIT]
INFO     snowflake.connector.cursor:cursor.py:737 query execution done
INFO     snowflake.connector.cursor:cursor.py:713 query: [ROLLBACK]
INFO     snowflake.connector.cursor:cursor.py:737 query execution done
INFO     src.helpers.snowflake_helper:snowflake_helper.py:79 creating stage: substations, url: gcs://built-object-service-dev/substations/jp/v0.0.0
INFO     snowflake.connector.cursor:cursor.py:713 query: [CREATE OR REPLACE STAGE substations STORAGE_INTEGRATION = DEV_GCS_INT URL = 'gcs...]
INFO     snowflake.connector.cursor:cursor.py:737 query execution done
INFO     snowflake.connector.cursor:cursor.py:713 query: [ROLLBACK]
INFO     snowflake.connector.cursor:cursor.py:737 query execution done
INFO     snowflake.connector.cursor:cursor.py:713 query: [SELECT * FROM substations]
INFO     snowflake.connector.cursor:cursor.py:737 query execution done
INFO     snowflake.connector.cursor:cursor.py:713 query: [copy into substations (ID,LNGLAT,PLUSCODE,PLUSCODE4,PLUSCODE6,PLUSCODE8,SOURCE_V...]
INFO     snowflake.connector.cursor:cursor.py:737 query execution done
INFO     snowflake.connector.cursor:cursor.py:713 query: [COMMIT]
INFO     snowflake.connector.cursor:cursor.py:737 query execution done
INFO     snowflake.connector.cursor:cursor.py:713 query: [ROLLBACK]
INFO     snowflake.connector.cursor:cursor.py:737 query execution done
INFO     snowflake.connector.cursor:cursor.py:713 query: [UPDATE substations SET data_event_id = 1 WHERE data_event_id = 0;]
INFO     snowflake.connector.cursor:cursor.py:737 query execution done
INFO     snowflake.connector.cursor:cursor.py:713 query: [COMMIT]
INFO     snowflake.connector.cursor:cursor.py:737 query execution done
INFO     snowflake.connector.cursor:cursor.py:713 query: [ROLLBACK]
INFO     snowflake.connector.cursor:cursor.py:737 query execution done
INFO     src.helpers.snowflake_helper:snowflake_helper.py:87 dropping stage: substations
INFO     snowflake.connector.cursor:cursor.py:713 query: [DROP STAGE substations]
INFO     snowflake.connector.cursor:cursor.py:737 query execution done
INFO     snowflake.connector.cursor:cursor.py:713 query: [ROLLBACK]
INFO     snowflake.connector.cursor:cursor.py:737 query execution done
INFO     src.pipelines.exporter_snowflake.__main__:__main__.py:97 >>> exported substations from gcs://built-object-service-dev/substations/jp/v0.0.0 successfully
INFO     src.pipelines.exporter_snowflake.__main__:__main__.py:112 >>> Export to Snowflake completed..
INFO     snowflake.connector.cursor:cursor.py:713 query: [SELECT * FROM substations;]
INFO     snowflake.connector.cursor:cursor.py:737 query execution done
INFO     snowflake.connector.cursor:cursor.py:713 query: [ROLLBACK]
INFO     snowflake.connector.cursor:cursor.py:737 query execution done
INFO     snowflake.connector.cursor:cursor.py:713 query: [ROLLBACK]
INFO     snowflake.connector.cursor:cursor.py:737 query execution done
====================================================================================================== warnings summary =======================================================================================================
../../../.local/share/virtualenvs/01-built-object-service-gvseloBk/lib/python3.8/site-packages/geopandas/_compat.py:18
  /Users/mkang/.local/share/virtualenvs/01-built-object-service-gvseloBk/lib/python3.8/site-packages/geopandas/_compat.py:18: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
    PANDAS_GE_10 = str(pd.__version__) >= LooseVersion("1.0.0")

../../../.local/share/virtualenvs/01-built-object-service-gvseloBk/lib/python3.8/site-packages/setuptools/_distutils/version.py:351
../../../.local/share/virtualenvs/01-built-object-service-gvseloBk/lib/python3.8/site-packages/setuptools/_distutils/version.py:351
../../../.local/share/virtualenvs/01-built-object-service-gvseloBk/lib/python3.8/site-packages/setuptools/_distutils/version.py:351
../../../.local/share/virtualenvs/01-built-object-service-gvseloBk/lib/python3.8/site-packages/setuptools/_distutils/version.py:351
../../../.local/share/virtualenvs/01-built-object-service-gvseloBk/lib/python3.8/site-packages/setuptools/_distutils/version.py:351
../../../.local/share/virtualenvs/01-built-object-service-gvseloBk/lib/python3.8/site-packages/setuptools/_distutils/version.py:351
../../../.local/share/virtualenvs/01-built-object-service-gvseloBk/lib/python3.8/site-packages/setuptools/_distutils/version.py:351
  /Users/mkang/.local/share/virtualenvs/01-built-object-service-gvseloBk/lib/python3.8/site-packages/setuptools/_distutils/version.py:351: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
    other = LooseVersion(other)

../../../.local/share/virtualenvs/01-built-object-service-gvseloBk/lib/python3.8/site-packages/geopandas/_compat.py:19
  /Users/mkang/.local/share/virtualenvs/01-built-object-service-gvseloBk/lib/python3.8/site-packages/geopandas/_compat.py:19: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
    PANDAS_GE_11 = str(pd.__version__) >= LooseVersion("1.1.0")

../../../.local/share/virtualenvs/01-built-object-service-gvseloBk/lib/python3.8/site-packages/geopandas/_compat.py:20
  /Users/mkang/.local/share/virtualenvs/01-built-object-service-gvseloBk/lib/python3.8/site-packages/geopandas/_compat.py:20: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
    PANDAS_GE_115 = str(pd.__version__) >= LooseVersion("1.1.5")

../../../.local/share/virtualenvs/01-built-object-service-gvseloBk/lib/python3.8/site-packages/geopandas/_compat.py:21
  /Users/mkang/.local/share/virtualenvs/01-built-object-service-gvseloBk/lib/python3.8/site-packages/geopandas/_compat.py:21: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
    PANDAS_GE_12 = str(pd.__version__) >= LooseVersion("1.2.0")

../../../.local/share/virtualenvs/01-built-object-service-gvseloBk/lib/python3.8/site-packages/geopandas/_compat.py:29
  /Users/mkang/.local/share/virtualenvs/01-built-object-service-gvseloBk/lib/python3.8/site-packages/geopandas/_compat.py:29: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
    SHAPELY_GE_17 = str(shapely.__version__) >= LooseVersion("1.7.0")

../../../.local/share/virtualenvs/01-built-object-service-gvseloBk/lib/python3.8/site-packages/geopandas/_compat.py:30
  /Users/mkang/.local/share/virtualenvs/01-built-object-service-gvseloBk/lib/python3.8/site-packages/geopandas/_compat.py:30: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
    SHAPELY_GE_18 = str(shapely.__version__) >= LooseVersion("1.8")

../../../.local/share/virtualenvs/01-built-object-service-gvseloBk/lib/python3.8/site-packages/geopandas/_compat.py:31
  /Users/mkang/.local/share/virtualenvs/01-built-object-service-gvseloBk/lib/python3.8/site-packages/geopandas/_compat.py:31: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
    SHAPELY_GE_20 = str(shapely.__version__) >= LooseVersion("2.0")

../../../.local/share/virtualenvs/01-built-object-service-gvseloBk/lib/python3.8/site-packages/geopandas/_compat.py:227
../../../.local/share/virtualenvs/01-built-object-service-gvseloBk/lib/python3.8/site-packages/geopandas/_compat.py:227
  /Users/mkang/.local/share/virtualenvs/01-built-object-service-gvseloBk/lib/python3.8/site-packages/geopandas/_compat.py:227: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
    PYPROJ_LT_3 = LooseVersion(pyproj.__version__) < LooseVersion("3")

../../../.local/share/virtualenvs/01-built-object-service-gvseloBk/lib/python3.8/site-packages/geopandas/_compat.py:228
../../../.local/share/virtualenvs/01-built-object-service-gvseloBk/lib/python3.8/site-packages/geopandas/_compat.py:228
  /Users/mkang/.local/share/virtualenvs/01-built-object-service-gvseloBk/lib/python3.8/site-packages/geopandas/_compat.py:228: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
    PYPROJ_GE_31 = LooseVersion(pyproj.__version__) >= LooseVersion("3.1")

../../../.local/share/virtualenvs/01-built-object-service-gvseloBk/lib/python3.8/site-packages/sklearn/utils/multiclass.py:14
  /Users/mkang/.local/share/virtualenvs/01-built-object-service-gvseloBk/lib/python3.8/site-packages/sklearn/utils/multiclass.py:14: DeprecationWarning: Please use `spmatrix` from the `scipy.sparse` namespace, the `scipy.sparse.base` namespace is deprecated.
    from scipy.sparse.base import spmatrix

../../../.local/share/virtualenvs/01-built-object-service-gvseloBk/lib/python3.8/site-packages/sklearn/utils/optimize.py:18
  /Users/mkang/.local/share/virtualenvs/01-built-object-service-gvseloBk/lib/python3.8/site-packages/sklearn/utils/optimize.py:18: DeprecationWarning: Please use `line_search_wolfe2` from the `scipy.optimize` namespace, the `scipy.optimize.linesearch` namespace is deprecated.
    from scipy.optimize.linesearch import line_search_wolfe2, line_search_wolfe1

../../../.local/share/virtualenvs/01-built-object-service-gvseloBk/lib/python3.8/site-packages/sklearn/utils/optimize.py:18
  /Users/mkang/.local/share/virtualenvs/01-built-object-service-gvseloBk/lib/python3.8/site-packages/sklearn/utils/optimize.py:18: DeprecationWarning: Please use `line_search_wolfe1` from the `scipy.optimize` namespace, the `scipy.optimize.linesearch` namespace is deprecated.
    from scipy.optimize.linesearch import line_search_wolfe2, line_search_wolfe1

<frozen importlib._bootstrap>:219
  <frozen importlib._bootstrap>:219: RuntimeWarning: numpy.ndarray size changed, may indicate binary incompatibility. Expected 16 from C header, got 96 from PyObject

tests/pipelines/exporter_snowflake/test_exporter_snowflake.py::test_snowflake_exporter
  /Users/mkang/occ/prod/01-built-object-service/src/pipelines/exporter_snowflake/data_events.py:82: SAWarning: Dialect snowflake:snowflake will not make use of SQL compilation caching as it does not set the 'supports_statement_cache' attribute to ``True``.  This can have significant performance implications including some performance degradations in comparison to prior SQLAlchemy versions.  Dialect maintainers should seek to set this attribute to True after appropriate development and testing for SQLAlchemy 1.4 caching support.   Alternatively, this attribute may be set to False which will disable this warning. (Background on this error at: https://sqlalche.me/e/14/cprf)
    conn.execute(stmt_create_temp)

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=================================================================================================== short test summary info ===================================================================================================
FAILED tests/pipelines/exporter_snowflake/test_exporter_snowflake.py::test_snowflake_exporter - IndexError: list index out of range
======================================================================================= 1 failed, 157 deselected, 23 warnings in 31.15s =======================================================================================
(01-built-object-service) ➜  01-built-object-service (foo) ✔

What did you expect to see?

What should have happened and what happened instead?

fetch_pandas_all() should return data correctly instead of throwing Index Error:

Can you set logging to DEBUG and collect the logs?

import logging
import os

for logger_name in ('snowflake.connector',):
    logger = logging.getLogger(logger_name)
    logger.setLevel(logging.DEBUG)
    ch = logging.StreamHandler()
    ch.setLevel(logging.DEBUG)
    ch.setFormatter(logging.Formatter('%(asctime)s - %(threadName)s %(filename)s:%(lineno)d - %(funcName)s() - %(levelname)s - %(message)s'))
    logger.addHandler(ch)

The text was updated successfully, but these errors were encountered:

sfc-gh-mkeller · 2022-05-19T22:34:19Z

Would it be possible for you to set a breakpoint in ../../../.local/share/virtualenvs/01-built-object-service-gvseloBk/lib/python3.8/site-packages/snowflake/connector/result_batch.py just before line 698 with for example: import pytest; pytest.set_trace() and tell us what s.type_code is right before your test crashes?
Is there any chance that you are working with GEOGRAPHY column type? Support for that was added in 2.7.4 and that would fit for your error

michaelk1c · 2022-05-20T17:19:30Z

Would it be possible for you to set a breakpoint in ../../../.local/share/virtualenvs/01-built-object-service-gvseloBk/lib/python3.8/site-packages/snowflake/connector/result_batch.py just before line 698 with for example: import pytest; pytest.set_trace() and tell us what s.type_code is right before your test crashes?
Is there any chance that you are working with GEOGRAPHY column type? Support for that was added in 2.7.4 and that would fit for your error

snowflake/connector/result_batch.py:697: in _create_empty_table
s.name: LNGLAT s.type_code: 14 s: ResultMetadata(name='LNGLAT',
type_code=14, display_size=None, internal_size=None,
precision=None, scale=None, is_nullable=True)

michaelk1c · 2022-05-20T17:22:44Z

Support for that was added in 2.7.4 and that would fit for your error

As you can see above on the package list, we're on 2.7.4. Should we upgrade this to 2.7.7?

michaelk1c added bug needs triage labels May 19, 2022

github-actions bot changed the title ~~IndexError: list index out of range in snowflake/connector/result_batch.py:698: IndexError~~ SNOW-592647: IndexError: list index out of range in snowflake/connector/result_batch.py:698: IndexError May 19, 2022

sfc-gh-mkeller removed the needs triage label May 27, 2022

sfc-gh-mkeller self-assigned this May 27, 2022

sfc-gh-mkeller mentioned this issue May 27, 2022

SNOW-592647 consolidate definitions and resolve circular dependency issues #1158

Merged

6 tasks

sfc-gh-mkeller closed this as completed in #1158 Jun 8, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SNOW-592647: IndexError: list index out of range in snowflake/connector/result_batch.py:698: IndexError #1145

SNOW-592647: IndexError: list index out of range in snowflake/connector/result_batch.py:698: IndexError #1145

michaelk1c commented May 19, 2022

sfc-gh-mkeller commented May 19, 2022

michaelk1c commented May 20, 2022

michaelk1c commented May 20, 2022

SNOW-592647: IndexError: list index out of range in snowflake/connector/result_batch.py:698: IndexError #1145

SNOW-592647: IndexError: list index out of range in snowflake/connector/result_batch.py:698: IndexError #1145

Comments

michaelk1c commented May 19, 2022

sfc-gh-mkeller commented May 19, 2022

michaelk1c commented May 20, 2022

michaelk1c commented May 20, 2022