Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

GRRafana: Add support to query Client Breakdown Statistics #870

Merged
merged 21 commits into from
Oct 28, 2020

Conversation

tsehori
Copy link
Contributor

@tsehori tsehori commented Oct 21, 2020

This PR extends GRRafana HTTP server introduced in #832 by adding new metrics that can be fetched.
These new metrics are the ones that appear in GRR's Statistics page (for reference, you can check out one of the metrics fetched within this page here).

The metrics introduced to the server (and Grafana, in turn) are:

  • OS Release Version Breakdown
  • OS Platform Breakdown
  • Client Version

All metrics are available in the interface for 1, 7, 14 and 30 days, and are shown in a table format (similar to the way it currently is in the Statistics page within GRR AdminUI).

In order to support this new metric (which unlike Client Load Stats in #859, which must be provided with a GRR Client ID), we GRRafana to support multiple types of metrics. So we have introduced the abstract Metric class to encapsulate the appropriate logic.

Note that this PR may make the different cron jobs that build FleetStats objects and push them to the datastore obsolete. As Grafana issues a request to GRRafana, the latter will call CountClientPlatformReleasesByLabel (for example) the build the corresponding FleetStats object that will encapsulate the data. Then GRRafana will process it and send it back to Grafana.

Also note that in a future PR, we'd like to cache the results of the queries within GRRafana, as they may get quite time consuming with many labels and data.

Gilad Tsehori added 16 commits October 13, 2020 07:47
This abstract class encapsulates a single metric that can be fetched
from a FS-based GRR deployment. For instance, such a metric can be "Mean
Resident Memory MB".

This class is created so that we can introduce metrics which aren't
targeted at individual clients alone, but also multiple clients
(statistics page in GRR).
Tests were fixed/enhanced for the previous behaviour (only Client Load
Stats). I will nxt added unit tests for the aggregated statistics that
were added to GRRafana.
- Changed days buckets to constant _FLEET_BREAKDOWN_DAY_BUCKETS
- Added a comment on conversion from MiB to MB
- Changed some variable names to be consistent and more clear
grr/server/grr_response_server/bin/grrafana.py Outdated Show resolved Hide resolved
grr/server/grr_response_server/bin/grrafana.py Outdated Show resolved Hide resolved
grr/server/grr_response_server/bin/grrafana.py Outdated Show resolved Hide resolved
grr/server/grr_response_server/bin/grrafana.py Outdated Show resolved Hide resolved
grr/server/grr_response_server/bin/grrafana.py Outdated Show resolved Hide resolved
grr/server/grr_response_server/bin/grrafana.py Outdated Show resolved Hide resolved
grr/server/grr_response_server/bin/grrafana.py Outdated Show resolved Hide resolved
grr/server/grr_response_server/bin/grrafana.py Outdated Show resolved Hide resolved
grr/server/grr_response_server/bin/grrafana_test.py Outdated Show resolved Hide resolved
Gilad Tsehori added 3 commits October 26, 2020 07:30
* Import package abc itself instead of methods out of it
* Callable type mentions full arg/return types
* Hide attributes by using underscores
* _Datapoint is now a NamedTuple
* Renamed AVAILABLE_METRICS_DICT to AVAILABLE_METRICS_BY_NAME
* Changed all 'extend's to AVAILABLE_METRICS_LIST with a more readable
loop
* Using context manager in testing for KeyError
@tsehori
Copy link
Contributor Author

tsehori commented Oct 26, 2020

PTAL @mbushkov @panhania

grr/server/grr_response_server/bin/grrafana.py Outdated Show resolved Hide resolved
grr/server/grr_response_server/bin/grrafana.py Outdated Show resolved Hide resolved
grr/server/grr_response_server/bin/grrafana.py Outdated Show resolved Hide resolved
Gilad Tsehori added 2 commits October 26, 2020 16:15
* Ignore override code issues and add TODO
* Make __init__ not abstract
* Use FrozenSet instead of frozenset
@tsehori
Copy link
Contributor Author

tsehori commented Oct 26, 2020

PTAL @panhania

@tsehori tsehori requested a review from panhania October 27, 2020 12:31
@mbushkov mbushkov merged commit 5641de4 into google:grrafana Oct 28, 2020
@tsehori tsehori deleted the clients_statistics branch October 28, 2020 15:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants