Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HCAD v2: add pagination #86

Closed
Tracked by #190
kaituo opened this issue Jun 1, 2021 · 0 comments
Closed
Tracked by #190

HCAD v2: add pagination #86

kaituo opened this issue Jun 1, 2021 · 0 comments
Labels
enhancement New feature or request

Comments

@kaituo
Copy link
Collaborator

kaituo commented Jun 1, 2021

Is your feature request related to a problem? Please describe.
The major problem of HCAD is scalability. HCAD issues a query to fetch feature data for each entity regularly. HCAD v1 limit the number of entity values returned per query to 1000, both for reasons of efficiency (fetching the overall entity values is expensive and may not finish before the next scheduled query run starts) and for reasons of ease-of-load-shedding (the top 1000 limit, in turn, curtails the memory used to host models, the disk access needed to read/write model checkpoints and anomaly results, the CPU used for entity metadata maintenance and model training/inference, and the garbage collection for deleted models and metadata). Users can increase the maximum entity setting at the cost of a linear growth resource usage, which opens the door to cluster instability. However, such a heuristic “best-effort” limit is detrimental to the scalability of entity monitoring. Even if a user scales out, the number of monitored entities does not increase.

Describe the solution you'd like
We increase the default maximum entities returned by the feature query from one thousand to one million. The huge number of entities are handled via pagination.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant