Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bug(flink): memtables with nested data do not compile/execute correctly on Flink backend #8516

Closed
1 task done
chloeh13q opened this issue Mar 1, 2024 · 0 comments
Closed
1 task done
Assignees
Labels
bug Incorrect behavior inside of ibis flink Issues or PRs related to Flink

Comments

@chloeh13q
Copy link
Contributor

chloeh13q commented Mar 1, 2024

What happened?

ibis.memtable() expressions do not compile/execute correctly on Flink backend.

Examples:
1.

t = ibis.memtable({"arr": [[], ["a"], ["a", "b"]]})
con.execute(t)

compiles into

SELECT
  `arr`
FROM (VALUES
  (CAST(ARRAY[] AS ARRAY<STRING>)),
  (ARRAY['a']),
  (ARRAY['a', 'b'])) AS `ibis_pandas_memtable_odkwlixgsfdmtmxfcd6ezswwpe`(`arr`)

which throws

Caused by: org.apache.calcite.runtime.CalciteContextException: From line 4, column 9 to line 4, column 15: Require at least 1 argument
jobs = ibis.memtable(
    {
        "steps": [
            [
                {"status": "success"},
                {"status": "success"},
                {"status": None},
                {"status": "failure"},
            ],
            [
                {"status": None},
                {"status": "success"},
            ],
        ]
    },
    schema=dict(steps="array<struct<status: string>>"),
)
con.execute(jobs)

compiles into

SELECT
  `steps`
FROM (VALUES
  (ARRAY[STRUCT('success' AS `status`), STRUCT('success' AS `status`), STRUCT(CAST(NULL AS STRING) AS `status`), STRUCT('failure' AS `status`)]),
  (ARRAY[STRUCT(CAST(NULL AS STRING) AS `status`), STRUCT('success' AS `status`)])) AS `ibis_pandas_memtable_gnydymxcijastata6eh4tiuwhq`(`steps`)

which throws

Caused by: org.apache.calcite.sql.parser.SqlParseException: Encountered "AS" at line 4, column 27.

What version of ibis are you using?

8.0

What backend(s) are you using, if any?

Flink

Relevant log output

No response

Code of Conduct

  • I agree to follow this project's Code of Conduct
@chloeh13q chloeh13q added the bug Incorrect behavior inside of ibis label Mar 1, 2024
@chloeh13q chloeh13q added the flink Issues or PRs related to Flink label Mar 1, 2024
@jcrist jcrist changed the title bug(flink): memtables do not compile/execute correctly on Flink backend bug(flink): memtables with nested data do not compile/execute correctly on Flink backend Mar 4, 2024
@chloeh13q chloeh13q moved this from backlog to cooking in Ibis planning and roadmap Mar 21, 2024
@chloeh13q chloeh13q self-assigned this Mar 23, 2024
@chloeh13q chloeh13q moved this from cooking to review in Ibis planning and roadmap Mar 23, 2024
@github-project-automation github-project-automation bot moved this from review to done in Ibis planning and roadmap Mar 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Incorrect behavior inside of ibis flink Issues or PRs related to Flink
Projects
Archived in project
Development

No branches or pull requests

1 participant