Skip to content

Backend query timeout when forwarding long-range queries (24+ hours) to Prometheus/Thanos #353

Description

@brnmndl

Environment:
ASAP Query Version: 0.4.0
Deployment: Kubernetes (Helm Chart)
Backend: Thanos Query
Query range: 24+ hours with 5-minute steps
Issue:
When executing long-range queries with forwardUnsupportedQueries: true, ASAP times out while forwarding to Thanos backend.

Error:

2026-05-21T09:50:52.441377Z ERROR query_engine_rust::drivers::query::fallback::prometheus: asap-query-engine/src/drivers/query/fallback/prometheus.rs:151: Failed to forward range query to Prometheus: error sending request for url (http://x-thanos-query.x-thanos.svc.cluster.local:9090/api/v1/query_range?query=sum%28sum_over_time%28x_aggregated_logs%7Blog_type%3D%22access%22%2C+component%3D%7E%22x-lb%22%2C+region%3D%7E%22.*%22%2C+host%3D%7E%22.*%22%2C+environment%3D%7E%22.*%22%2C+server_name%3D%7E%22.*%22%2C+upstream_name%3D%7E%22.*%22%2C+status%3D%7E%22.*%22%7D%5B5m%5D%29%29&start=1779346200&end=1779357000&step=300): operation timed out

Impact:

Blocks long-range queries (24+ hours) through ASAP
Affects high-cardinality metrics
No current workaround without source code modification

Metadata

Metadata

Labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions