Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Failure of getting column descriptions for some tables #508

Closed
yruslan opened this issue Oct 31, 2024 · 0 comments · Fixed by #509
Closed

Failure of getting column descriptions for some tables #508

yruslan opened this issue Oct 31, 2024 · 0 comments · Fixed by #509
Labels
bug Something isn't working

Comments

@yruslan
Copy link
Collaborator

yruslan commented Oct 31, 2024

Describe the bug

Fetching database metadata for JDBC sources sometimes gets 10s of gigabytes of data from the server.

Until this is fixed, let's disable fetching column descriptions for MS SQL Server.

The driver is: "com.microsoft.sqlserver.jdbc.SQLServerDriver"

Steps to reproduce

    val statement = connection.createStatement(ResultSet.TYPE_FORWARD_ONLY, ResultSet.CONCUR_READ_ONLY)

    val rs = statement.executeQuery("SELECT CAST(SERVERPROPERTY('EngineEdition') as INT)")

    rs.next()

    println(rs.getLong(1))

Expected behavior

Pramen pipelines should not fail with out of memory exceptions when loading column descriptions.

Root cause

The route cause is the the above query, but the fact that we query the data query to get the resultset to get metadata from. This is fixed by ensuring that the metadata query does not return any actual records.

@yruslan yruslan added the bug Something isn't working label Oct 31, 2024
@yruslan yruslan changed the title Failure of getting column descriptions for MS SQL Server Failure of getting column descriptions for some tables Nov 1, 2024
@yruslan yruslan closed this as completed Nov 1, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant