Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Setting MaxExecutionTime doesn't get correctly passed to clickhouse query #6493

Open
sati-max opened this issue Nov 20, 2024 · 0 comments
Open

Comments

@sati-max
Copy link

Bug description

The config value of MaxExecutionTime in query Service isn't being correctly set in the passed query to clickhouse database and it's still hardcoded to 64 seconds. This results in queries being aborted after 64 seconds. In both scenarios - clickhouse provided by SigNoz or external clickhouse setup on separate host.

The setting:
additionalEnvs: ClickHouseMaxExecutionTime: "3600"

Actual query information from clickhouse system.query_log shows for SELECT queries:
{'connect_timeout_with_failover_ms':'1000','distributed_aggregation_memory_efficient':'1','log_queries':'1','max_execution_time':'64','log_comment':'{"alertID":"","client":"api","dashboardID":"","path":"","servicesTab":"","source":"","viewName":""}','parallel_view_processing':'1','allow_nondeterministic_mutations':'1','allow_experimental_window_functions':'1','default_database_engine':'Ordinary'}

Setting ClickHouseMaxExecutionTimeLeaf variable doesn't help - the query is being aborted after 64 seconds as max_execution_time has higher priority then max_execution_time_leaf setting:
{'connect_timeout':'3600','receive_timeout':'3600','send_timeout':'3600','load_balancing':'random','max_execution_time':'64','max_execution_time_leaf':'3600','max_memory_usage':'10000000000','log_comment':'{"alertID":"","client":"browser","dashboardID":"ea6d4f10-9816-4812-9867-44752708e296","email":"","path":"/dashboard/ea6d4f10-9816-4812-9867-44752708e296","servicesTab":"","source":"dashboards","viewName":""}'}

Tested on K8S setup with clickhouse database on k8s pod (first example) and also with external clickhouse database set outside of K8S workers (second example).

Expected behavior

max_execution_time variable is being correctly passed to clickhouse database and the query doesn't get aborted after 64 seconds.

How to reproduce

  1. Setup MaxExecutionTime to some value above 64
  2. Run anything in GUI that produces a SELECT query (log viwer, dashboard)
  3. Check clickhouse database system.query_log and filter for type = 'QueryFinish' and query_kind = 'Select'
  4. Verify Settings: column and the value of max_execution_time variable

Version information

  • Signoz version: 0.53.0 (chart 0.51.1) and 0.58.2 (chart 0.56.2)
  • Browser version: Chrome 130.0.6723.117
  • Your OS and version: Windows 11
  • Your CPU Architecture(ARM/Intel): Intel

Additional context

The feature to add max_execution_time was open here: #5393 and merged to 0.53.0 release here: #5683

Thank you for your bug report – we love squashing them!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant