Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Configure drafter client to pass max-query-timeout #103

Open
Robsteranium opened this issue May 17, 2021 · 2 comments
Open

Configure drafter client to pass max-query-timeout #103

Robsteranium opened this issue May 17, 2021 · 2 comments
Assignees
Labels
etl Related to the etl/pipelines ops Related to devops/deployment of the app

Comments

@Robsteranium
Copy link
Contributor

Robsteranium commented May 17, 2021

Stardog times out before queries can finish and this can go unnoticed (if drafter swallows the error).

Drafter asks Stardog to time-out queries after 30s (although result may continue to be returned for 90s).

Stardog is configured to allow a maximum time-out of 15 mins.

Drafter will allow privileged users to provide a JWS-signed timeout parameter to their requests to lift the 30s timeout.

We need to generate a key, configure drafter's DRAFTER_JWS_SIGNING_KEY env var, and have the ook.etl/query function pass the signed parameter.

This will leave large results in stasher but they're in binary format (so orders of magnitude smaller than the text versions (which are gigabytes)) and they'll only survive until the next publish. Once we're loading changes only #17 this should be less of a concern.

An alternative would be to make requests directly to Stardog. For this we'd need to install ook on the muttnik box or otherwise tunnel to stardog. We'd also need to change the queries to exclude draft graphs.

@Robsteranium Robsteranium added etl Related to the etl/pipelines ops Related to devops/deployment of the app labels May 17, 2021
@Robsteranium Robsteranium self-assigned this May 17, 2021
@Robsteranium Robsteranium changed the title Stardog times out before queries can finish Configure drafter client to pass max-query-timeout May 20, 2021
@Robsteranium
Copy link
Contributor Author

Adding jws-signing led to HTTP/2 errors with curl. The max-query-timeout parameter was also missing from drafter's swagger schema which mean the Martian client didn't want to use it.

Paging the large observation-select.sparql by graphs ought to obviate the need for this, but it may recur (perhaps for the expensive codes-used-construct.sparql queries #69.

@Robsteranium Robsteranium reopened this May 20, 2021
@Robsteranium
Copy link
Contributor Author

We're seeing truncated responses on the code-pipeline now too. It passes with a select limit of 100 but this isn't ideal.

@RicSwirrl has also wondered if the load balancer is truncating responses.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
etl Related to the etl/pipelines ops Related to devops/deployment of the app
Projects
None yet
Development

No branches or pull requests

1 participant