Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prometheus exporter #294

Merged
Merged
Show file tree
Hide file tree
Changes from 48 commits
Commits
Show all changes
53 commits
Select commit Hold shift + click to select a range
f839e21
Squashing commits
MarceloAquino7 Aug 14, 2018
f89b60a
Few adjustments due to Emmanuel's review
MarceloAquino7 Aug 20, 2018
ed6f363
Fix due to Angela's reply | Created unit test for the new transport s…
MarceloAquino7 Aug 22, 2018
4ec6dc3
Renaming file
MarceloAquino7 Aug 22, 2018
15f687f
Sync transport unit test
MarceloAquino7 Aug 22, 2018
9397a8a
Lint fixes
MarceloAquino7 Aug 22, 2018
750fd30
Lint fixes
MarceloAquino7 Aug 22, 2018
1edfeba
Merge branch 'master' into stats_core_changes
Aug 24, 2018
123fc3b
Add tests for situations with exporter registered
MarceloAquino7 Aug 27, 2018
ecaacc3
Merge branch 'master' into stats_core_changes
liyanhui1228 Aug 27, 2018
ec8cbf3
Bringing content from another branch
MarceloAquino7 Aug 28, 2018
6ce4afb
Some fixes due to code review
MarceloAquino7 Aug 30, 2018
a14667e
support W3C distributed tracing (part 1) (#251)
reyang Aug 9, 2018
033a1ea
Fix typos in the README. (#252)
songy23 Aug 9, 2018
3722f70
Squashing commits
MarceloAquino7 Aug 14, 2018
a492f88
Tags: Set and get current tag_map in thread local. (#257)
songy23 Aug 14, 2018
8a9417e
add tracestate to spancontext (#259)
reyang Aug 16, 2018
ff0b9a0
Stats: Allow recording against implicit tag_map. (#258)
songy23 Aug 17, 2018
eb8ba4f
Few adjustments due to Emmanuel's review
MarceloAquino7 Aug 20, 2018
05f653c
support tracestate HTTP propagation (#261)
reyang Aug 20, 2018
5c7460f
Fix #263 (#264)
ocervell Aug 20, 2018
9515063
Prepare for 0.1.6 release (#267)
liyanhui1228 Aug 21, 2018
3827a23
Fix due to Angela's reply | Created unit test for the new transport s…
MarceloAquino7 Aug 22, 2018
4fb0061
Renaming file
MarceloAquino7 Aug 22, 2018
3fde245
Sync transport unit test
MarceloAquino7 Aug 22, 2018
3fd06bd
Lint fixes
MarceloAquino7 Aug 22, 2018
338f173
Lint fixes
MarceloAquino7 Aug 22, 2018
1d1d4c8
Set SpanKind in extensions (#271)
Aug 22, 2018
549a5e1
Rename attributes to the default naming schema and use standard http …
Aug 23, 2018
ad877b0
Fix bug in updating aggregation map. (#253)
songy23 Aug 23, 2018
936022a
Document 'google-cloud-trace' and BackgroundThreadTransport (#262)
stewartreichling Aug 24, 2018
64a3f95
Add tests for situations with exporter registered
MarceloAquino7 Aug 27, 2018
35b201b
Log a detailed exception when integration fails (#284)
stewartreichling Aug 27, 2018
a444034
Bringing content from another branch
MarceloAquino7 Aug 28, 2018
3de80f8
Some fixes due to code review
MarceloAquino7 Aug 30, 2018
6225dbf
Merge branch 'prometheus_exporter' of https://github.com/CESARBR/open…
MarceloAquino7 Aug 30, 2018
ae3425c
Merge branch 'master' into prometheus_exporter
vcasadei Aug 31, 2018
7a89d7b
Fixing merge issues
vcasadei Aug 31, 2018
cd0787b
Fixes after merge
vcasadei Sep 3, 2018
1a65e5d
Adding more tests and documentation
vcasadei Sep 4, 2018
79ce303
Removing unused code
vcasadei Sep 4, 2018
6db375c
Updating Readme
vcasadei Sep 4, 2018
bc9e485
Fixing readme
vcasadei Sep 4, 2018
2f5984e
Fixing readme
vcasadei Sep 4, 2018
5dd3c92
Merge branch 'master' into prometheus_exporter
vcasadei Sep 6, 2018
52067bd
Fixing readme
vcasadei Sep 6, 2018
92fb0cc
cover
vcasadei Sep 6, 2018
006f94b
Lint
vcasadei Sep 6, 2018
5bef7f0
Fix readme
vcasadei Sep 6, 2018
f442bc5
lint
vcasadei Sep 10, 2018
a905d2b
Merge branch 'master' into prometheus_exporter
vcasadei Sep 10, 2018
ed73ffd
Addresses review comments
eduardoemery Sep 17, 2018
191738b
Merge branch 'master' into prometheus_exporter
liyanhui1228 Sep 19, 2018
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
78 changes: 68 additions & 10 deletions README.rst
Original file line number Diff line number Diff line change
Expand Up @@ -415,28 +415,29 @@ The use of vendoring or a dependency management tool is recommended.

.. _Stackdriver: https://app.google.stackdriver.com/metrics-explorer

Exporter Usage
~~~~~~~~~~~~~~
Stackdriver Exporter Usage
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Import
******
Stackdriver Import
************************

.. code:: python

from opencensus.stats.exporters import stackdriver_exporter as stackdriver
from opencensus.stats import stats as stats_module

Prerequisites
*************
Stackdriver Prerequisites
**************************

- OpenCensus Python libraries require Python 2.7 or later.
- Google Cloud Platform account and project.
- Google Stackdriver Monitoring enabled on your project (Need help? `Click here`_).

.. _Click here: https://opencensus.io/codelabs/stackdriver

Register the exporter
*********************
Register the Stackdriver exporter
**********************************

.. code:: python

stats = stats_module.Stats()
Expand All @@ -447,8 +448,8 @@ Register the exporter
...


Code Reference
**************
Stackdriver Code Reference
******************************

In the *examples* folder, you can find all the necessary steps to get the exporter, register a view, put tags on the measure, and see the values against the Stackdriver monitoring tool once you have defined the *project_id*.

Expand All @@ -462,6 +463,62 @@ For further details for the Stackdriver implementation, see the file *stackdrive
| opencensus/stats/exporters/stackdriver_exporter.py | Stats implementation for Stackdriver|
+----------------------------------------------------+-------------------------------------+

Prometheus Stats
-----------------

The OpenCensus `Prometheus`_ Stats Exporter allows users
to export metrics to Prometheus monitoring solution.
The API of this project is still evolving.
The use of vendoring or a dependency management tool is recommended.

.. _Prometheus: https://prometheus.io/

Prometheus Exporter Usage
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Prometheus Import
********************

.. code:: python

from opencensus.stats.exporters import prometheus_exporter as prometheus
from opencensus.stats import stats as stats_module

Prometheus Prerequisites
***************************

- OpenCensus Python libraries require Python 2.7 or later.
- Prometheus up and running.

Register the Prometheus exporter
***********************************

.. code:: python

stats = stats_module.Stats()
view_manager = stats.view_manager

exporter = prometheus.new_stats_exporter(prometheus.Options(namespace="<namespace>"))
view_manager.register_exporter(exporter)
...


Prometheus Code Reference
***************************

In the *examples* folder, you can find all the necessary steps to get the exporter, register a view, put tags on the measure, and see the values against the Prometheus monitoring tool.

For further details for the Prometheus implementation, see the file *prometheus_exporter.py*.


+----------------------------------------------------+-------------------------------------+
| Path & File | Short Description |
+====================================================+=====================================+
| examples/stats/exporter/prometheus.py | End to end example |
+----------------------------------------------------+-------------------------------------+
| opencensus/stats/exporters/prometheus_exporter.py | Stats implementation for Prometheus |
+----------------------------------------------------+-------------------------------------+

------------------
Additional Info
------------------
Expand Down Expand Up @@ -516,3 +573,4 @@ Disclaimer
----------

This is not an official Google product.
gi
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Spurious?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed it

80 changes: 80 additions & 0 deletions examples/stats/exporter/prometheus.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
#!/usr/bin/env python

# Copyright 2018, OpenCensus Authors
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

import random
import time

from opencensus.stats import aggregation as aggregation_module
from opencensus.stats.exporters import prometheus_exporter as prometheus
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why don't we keep the file name as prometheus? Instead of prometheus_exporter, which already resides under opencensus/stats/exporters, hence the _exporter suffix here there is redundant.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I got your point, but this is how the exporters files where named in trace and stats before. I believe they were just following the pattern. But I like your suggestion, I think we can open an issue about this.

from opencensus.stats import measure as measure_module
from opencensus.stats import stats as stats_module
from opencensus.stats import view as view_module
from opencensus.tags import tag_key as tag_key_module
from opencensus.tags import tag_map as tag_map_module
from opencensus.tags import tag_value as tag_value_module
from pprint import pprint

MiB = 1 << 20
FRONTEND_KEY = tag_key_module.TagKey("my.org/keys/frontend")
VIDEO_SIZE_MEASURE = measure_module.MeasureInt(
"my.org/measures/video_size", "size of processed videos", "By")
VIDEO_SIZE_VIEW_NAME = "my.org/views/video_size"
VIDEO_SIZE_DISTRIBUTION = aggregation_module.DistributionAggregation(
[0.0, 16.0 * MiB, 256.0 * MiB])
VIDEO_SIZE_VIEW = view_module.View(VIDEO_SIZE_VIEW_NAME,
"processed video size over time",
[FRONTEND_KEY],
VIDEO_SIZE_MEASURE,
VIDEO_SIZE_DISTRIBUTION)


def main():
stats = stats_module.Stats()
view_manager = stats.view_manager
stats_recorder = stats.stats_recorder

exporter = prometheus.new_stats_exporter(prometheus.Options(namespace="opencensus"))
view_manager.register_exporter(exporter)

# Register view.
view_manager.register_view(VIDEO_SIZE_VIEW)

# Sleep for [0, 10] milliseconds to fake work.
time.sleep(random.randint(1, 10) / 1000.0)

# Process video.
# Record the processed video size.
tag_value = tag_value_module.TagValue(str(random.randint(1, 10000)))
tag_map = tag_map_module.TagMap()
tag_map.insert(FRONTEND_KEY, tag_value)
measure_map = stats_recorder.new_measurement_map()
measure_map.measure_int_put(VIDEO_SIZE_MEASURE, 25 * MiB)
measure_map.record(tag_map)

# Use the line below to see the data on prometheus
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remove the lines here?

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, I'm not sure. If we remove, the user may not be able to access Prometheus before the code finishes. This is just a way to time.sleep() for an uncertain amount of time and allow the user to see the collected data on Prometheus. Do you have any suggestions on how to do that in a better way?

# while True:
# pass

# Get aggregated stats and print it to console.
view_data = view_manager.get_view(VIDEO_SIZE_VIEW_NAME)
pprint(vars(view_data))
for k, v in view_data._tag_value_aggregation_data_map.items():
pprint(k)
pprint(vars(v))


if __name__ == '__main__':
main()
1 change: 0 additions & 1 deletion opencensus/stats/aggregation_data.py
Original file line number Diff line number Diff line change
Expand Up @@ -230,7 +230,6 @@ def increment_bucket_count(self, value):
return i

self._counts_per_bucket[(len(self._bounds))-1] += 1

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Spurious line? Please remove it.

return i


Expand Down
Empty file modified opencensus/stats/exporters/__init__.py
100644 → 100755
Empty file.
4 changes: 2 additions & 2 deletions opencensus/stats/exporters/base.py
100644 → 100755
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,7 @@ def on_register_view(self, view):
:param object of opencensus.stats.view.View view:
View object to register
"""
raise NotImplementedError
raise NotImplementedError # pragma: NO COVER

def emit(self, view_datas):
"""Send view and measurement to exporter record method,
Expand All @@ -39,4 +39,4 @@ def emit(self, view_datas):
:param list of opencensus.stats.view_data.ViewData ViewData:
list of ViewData object to send to Stackdriver Monitoring
"""
raise NotImplementedError
raise NotImplementedError # pragma: NO COVER
Loading