duckdb.ConversionException: Conversion Error: Unimplemented type for cast (STRUCT("0" DOUBLE, "1" DOUBLE, ... , "23011" DOUBLE) -> DOUBLE)

[Yust0724]

How to reproduce:

In "Part 2 Instructions (Exercises)," when modifying and running a part of extract_historical_weather_data.py as specified in Exercise 2, the above error occurs.

coordinates = get_lat_long_for_city.expand(city=["Bern", "Luzern", "Zurich"])
historical_weather = get_historical_weather.expand(coordinates=coordinates)

Cause:

When you run type(historical_weather) to check the type of the variable historical_weather, it outputs the following:
<class 'airflow.models.xcom.LazyXComAccess'>

This means that the code is entering the else part of the if type(historical_weather) == list : statement in the turn_json_into_table function, which is causing this issue.

    def turn_json_into_table(
        duckdb_conn_id: str,
        historical_weather_table_name: str,
        historical_weather: dict,
    ):
        """
        Convert the JSON input with info about historical weather into a pandas
        DataFrame and load it into DuckDB.
        Args:
            duckdb_conn_id (str): The connection ID for the DuckDB connection.
            historical_weather_table_name (str): The name of the table to store the historical weather data.
            historical_weather (list): The historical weather data to load into DuckDB.
        """
        from duckdb_provider.hooks.duckdb_hook import DuckDBHook

        if type(historical_weather) == list:
            list_of_df = []

            for item in historical_weather:
                df = pd.DataFrame(item)
                list_of_df.append(df)

            historical_weather_df = pd.concat(list_of_df, ignore_index=True)
        else:
            historical_weather_df = pd.DataFrame(historical_weather)

Proposed Solution:

By commenting out the entire block of the above if statement and modifying the code to run the following part directly, it worked successfully. If you have any better suggestions for a fix, please let me know.

            list_of_df = []

            for item in historical_weather:
                df = pd.DataFrame(item)
                list_of_df.append(df)

            historical_weather_df = pd.concat(list_of_df, ignore_index=True)