examples: add python span profiling (#3764)

* examples: add python span profiling

docs/sources/configure-client/trace-span-profiles/python-span-profiles.md

@@ -75,4 +75,6 @@ Refer to the [data source configuration documentation](https://grafana.com/docs/
 
 ## Examples
 
-Check out the [examples](https://github.com/grafana/pyroscope/tree/main/examples/tracing/tempo) directory for a complete demo application of span profiles in multiple languages.
+Check out these demo applications for span profiles:
+- [Python example](https://github.com/grafana/pyroscope/tree/main/examples/tracing/tempo/python)
+- [Other examples](https://github.com/grafana/pyroscope/tree/main/examples/tracing/tempo) in multiple languages

examples/tracing/python/Dockerfile

@@ -0,0 +1,10 @@
+FROM python:3.9
+
+RUN pip3 install flask pyroscope-io==0.8.8 pyroscope-otel==0.4.0
+RUN pip3 install opentelemetry-api opentelemetry-sdk opentelemetry-instrumentation-flask opentelemetry-exporter-otlp-proto-grpc
+
+ENV FLASK_ENV=development
+ENV PYTHONUNBUFFERED=1
+
+COPY lib ./lib
+CMD [ "python", "lib/server.py" ]

examples/tracing/python/README.md

@@ -0,0 +1,54 @@
+# Span Profiles with Grafana Tempo and Pyroscope
+
+The docker compose consists of:
+- Three Python Rideshare App instances (us-east, eu-north, ap-south regions)
+- Tempo for trace collection
+- Pyroscope for continuous profiling
+- Grafana for visualization
+- Load Generator for simulating traffic
+
+For a detailed guide about Python span profiles configuration, refer to the docs [Pyroscope documentation](https://grafana.com/docs/pyroscope/latest/configure-client/trace-span-profiles/python-span-profiles/).
+
+The `rideshare` app generates traces and profiling data that should be available in Grafana.
+Pyroscope and Tempo datasources are provisioned automatically.
+
+### Build and run
+
+The project can be run locally with the following commands:
+
+```shell
+# Pull latest pyroscope and grafana images:
+docker pull grafana/pyroscope:latest
+docker pull grafana/grafana:latest
+
+docker compose up
+```
+The load generator will automatically start sending requests to all regional instances.
+
+### Viewing Traces and Profiles
+
+Navigate to the [Explore page](http://localhost:3000/explore?schemaVersion=1&panes=%7B%22yM9%22:%7B%22datasource%22:%22tempo%22,%22queries%22:%5B%7B%22refId%22:%22A%22,%22datasource%22:%7B%22type%22:%22tempo%22,%22uid%22:%22tempo%22%7D,%22queryType%22:%22traceqlSearch%22,%22limit%22:20,%22tableType%22:%22traces%22,%22filters%22:%5B%7B%22id%22:%22e73a615e%22,%22operator%22:%22%3D%22,%22scope%22:%22span%22%7D,%7B%22id%22:%22service-name%22,%22tag%22:%22service.name%22,%22operator%22:%22%3D%22,%22scope%22:%22resource%22,%22value%22:%5B%22rideshare.python.push.app%22%5D,%22valueType%22:%22string%22%7D%5D%7D%5D,%22range%22:%7B%22from%22:%22now-6h%22,%22to%22:%22now%22%7D%7D%7D&orgId=1), select a trace and click on a span that has a linked profile:
+
+![image](https://github.com/grafana/otel-profiling-go/assets/12090599/31e33cd1-818b-4116-b952-c9ec7b1fb593)
+
+By default, only the root span gets labeled (the first span created locally): such spans are marked with the link icon
+and have `pyroscope.profile.id` attribute set to the corresponding span ID.
+Please note that presence of the attribute does not necessarily
+indicate that the span has a profile: stack trace samples might not be collected, if the utilized CPU time is
+less than the sample interval (10ms).
+
+### Grafana Tempo configuration
+
+In order to correlate trace spans with profiling data, the Tempo datasource should have the following configured:
+- The profiling data source
+- Tags to use when making profiling queries
+
+![image](https://github.com/grafana/pyroscope/assets/12090599/380ac574-a298-440d-acfb-7bc0935a3a7c)
+
+While tags are optional, configuring them is highly recommended for optimizing query performance.
+In our example, we configured the `service.name` tag for use in Pyroscope queries as the `service_name` label.
+This configuration restricts the data set for lookup, ensuring that queries remain
+consistently fast. Note that the tags you configure must be present in the span attributes or resources
+for a trace to profiles span link to appear.
+
+Please refer to our [documentation](https://grafana.com/docs/grafana/next/datasources/tempo/configure-tempo-data-source/#trace-to-profiles) for more details.

examples/tracing/python/docker-compose.yaml

@@ -0,0 +1,85 @@
+services:
+  pyroscope:
+    image: grafana/pyroscope
+    ports:
+      - "4040:4040"
+
+  us-east:
+    ports:
+      - "5000"
+    hostname: us-east
+    environment: &env
+      OTLP_URL: tempo:4318
+      OTLP_INSECURE: 1
+      DEBUG_LOGGER: 1
+      OTEL_TRACES_EXPORTER: otlp
+      OTEL_EXPORTER_OTLP_ENDPOINT: http://tempo:4317
+      OTEL_SERVICE_NAME: rideshare.python.push.app
+      OTEL_METRICS_EXPORTER: none
+      OTEL_TRACES_SAMPLER: always_on
+      OTEL_PROPAGATORS: tracecontext
+      PYROSCOPE_LABELS: hostname=us-east
+      REGION: us-east
+      PYROSCOPE_SERVER_ADDRESS: http://pyroscope:4040
+      PYTHONUNBUFFERED: 1  # Python-specific: Ensures logging output isn't buffered
+    build:
+      context: .
+
+  eu-north:
+    ports:
+      - "5000"
+    hostname: eu-north
+    environment:
+      <<: *env
+      REGION: eu-north
+      PYROSCOPE_LABELS: hostname=eu-north
+    build:
+      context: .
+
+  ap-south:
+    ports:
+      - "5000"
+    hostname: ap-south
+    environment:
+      <<: *env
+      REGION: ap-south
+      PYROSCOPE_LABELS: hostname=ap-south
+    build:
+      context: .
+
+  load-generator:
+    environment: *env
+    build:
+      context: ../../language-sdk-instrumentation/golang-push/rideshare
+      dockerfile: Dockerfile.load-generator
+    command:
+      - ./loadgen
+      - http://ap-south:5000
+      - http://eu-north:5000
+      - http://us-east:5000
+
+  grafana:
+    image: grafana/grafana:latest
+    environment:
+      - GF_INSTALL_PLUGINS=grafana-pyroscope-app
+      - GF_AUTH_ANONYMOUS_ENABLED=true
+      - GF_AUTH_ANONYMOUS_ORG_ROLE=Admin
+      - GF_AUTH_DISABLE_LOGIN_FORM=true
+      - GF_FEATURE_TOGGLES_ENABLE=traceToProfiles tracesEmbeddedFlameGraph
+    volumes:
+      - ./grafana-provisioning:/etc/grafana/provisioning
+    ports:
+      - "3000:3000"
+
+  tempo:
+    image: grafana/tempo:latest
+    command: [ "-config.file=/etc/tempo.yml" ]
+    volumes:
+      - ./tempo/tempo.yml:/etc/tempo.yml
+    ports:
+      - "14268:14268"  # jaeger ingest
+      - "3200:3200"   # tempo
+      - "9095:9095" # tempo grpc
+      - "4317:4317"  # otlp grpc
+      - "4318:4318"  # otlp http
+      - "9411:9411"   # zipkin

examples/tracing/python/grafana-provisioning/datasources/pyroscope.yml

@@ -0,0 +1,31 @@
+---
+apiVersion: 1
+datasources:
+  - name: Tempo
+    type: tempo
+    access: proxy
+    orgId: 1
+    url: http://tempo:3200
+    basicAuth: false
+    isDefault: true
+    version: 1
+    editable: false
+    apiVersion: 1
+    uid: tempo
+    jsonData:
+      httpMethod: GET
+      serviceMap:
+        datasourceUid: prometheus
+      tracesToProfiles:
+        customQuery: false
+        datasourceUid: "pyroscope"
+        profileTypeId: "process_cpu:cpu:nanoseconds:cpu:nanoseconds"
+        tags:
+          - key: "service.name"
+            value: "service_name"
+  - uid: pyroscope
+    type: grafana-pyroscope-datasource
+    name: Pyroscope
+    url: http://pyroscope:4040
+    jsonData:
+      keepCookies: [pyroscope_git_session]

examples/tracing/python/grafana-provisioning/plugins/explore-profiles.yml

@@ -0,0 +1,7 @@
+---
+apiVersion: 1
+apps:
+  - type: grafana-pyroscope-app
+    jsonData:
+      backendUrl: http://pyroscope:4040
+    secureJsonData:

examples/tracing/python/lib/__init__.py

@@ -0,0 +1 @@
+

examples/tracing/python/lib/bike/__init__.py

@@ -0,0 +1 @@
+

examples/tracing/python/lib/bike/bike.py

@@ -0,0 +1,4 @@
+from utility.utility import find_nearest_vehicle
+
+def order_bike(search_radius):
+    find_nearest_vehicle(search_radius, "bike")

examples/tracing/python/lib/car/__init__.py

@@ -0,0 +1 @@
+

examples/tracing/python/lib/car/car.py

@@ -0,0 +1,4 @@
+from utility.utility import find_nearest_vehicle
+
+def order_car(search_radius):
+    find_nearest_vehicle(search_radius, "car")

examples/tracing/python/lib/scooter/__init__.py

@@ -0,0 +1 @@
+

examples/tracing/python/lib/scooter/scooter.py

@@ -0,0 +1,4 @@
+from utility.utility import find_nearest_vehicle
+
+def order_scooter(search_radius):
+    find_nearest_vehicle(search_radius, "scooter")

examples/tracing/python/lib/server.py

@@ -0,0 +1,68 @@
+import os
+
+import pyroscope
+from flask import Flask
+
+# OpenTelemetry
+from opentelemetry.instrumentation.flask import FlaskInstrumentor
+from opentelemetry import trace
+from opentelemetry.sdk.trace import TracerProvider
+from opentelemetry.sdk.trace.export import BatchSpanProcessor
+from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
+from pyroscope.otel import PyroscopeSpanProcessor
+
+from bike.bike import order_bike
+from car.car import order_car
+from scooter.scooter import order_scooter
+
+provider = TracerProvider()
+provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
+provider.add_span_processor(PyroscopeSpanProcessor())
+
+# Sets the global default tracer provider
+trace.set_tracer_provider(provider)
+
+app_name = os.getenv("PYROSCOPE_APPLICATION_NAME", "rideshare.python.push.app")
+server_addr = os.getenv("PYROSCOPE_SERVER_ADDRESS", "http://pyroscope:4040")
+basic_auth_username = os.getenv("PYROSCOPE_BASIC_AUTH_USER", "")
+basic_auth_password = os.getenv("PYROSCOPE_BASIC_AUTH_PASSWORD", "")
+port = int(os.getenv("RIDESHARE_LISTEN_PORT", "5000"))
+
+pyroscope.configure(
+	application_name = app_name,
+	server_address   = server_addr,
+    basic_auth_username = basic_auth_username, # for grafana cloud
+    basic_auth_password = basic_auth_password, # for grafana cloud
+	tags             = {
+        "region":   f'{os.getenv("REGION")}',
+	}
+)
+
+app = Flask(__name__)
+FlaskInstrumentor().instrument_app(app)
+
+@app.route("/bike")
+def bike():
+    order_bike(0.2)
+    return "<p>Bike ordered</p>"
+
+@app.route("/scooter")
+def scooter():
+    order_scooter(0.3)
+    return "<p>Scooter ordered</p>"
+
+@app.route("/car")
+def car():
+    order_car(0.4)
+    return "<p>Car ordered</p>"
+
+
+@app.route("/")
+def environment():
+    result = "<h1>environment vars:</h1>"
+    for key, value in os.environ.items():
+        result +=f"<p>{key}={value}</p>"
+    return result
+
+if __name__ == '__main__':
+    app.run(threaded=False, processes=1, host='0.0.0.0', port=port, debug=False)

examples/tracing/python/lib/utility/__init__.py

@@ -0,0 +1 @@
+

examples/tracing/python/lib/utility/utility.py

@@ -0,0 +1,35 @@
+import time
+import pyroscope
+import os
+from datetime import datetime
+
+
+def mutex_lock(n):
+    i = 0
+    start_time = time.time()
+    while time.time() - start_time < n * 10:
+        i += 1
+
+def check_driver_availability(n):
+    i = 0
+    start_time = time.time()
+    while time.time() - start_time < n / 2:
+        i += 1
+
+    # Every 4 minutes this will artificially create make requests in eu-north region slow
+    # this is just for demonstration purposes to show how performance impacts show up in the
+    # flamegraph
+
+    force_mutex_lock = datetime.today().minute * 4 % 8 == 0
+    if os.getenv("REGION") == "eu-north" and force_mutex_lock:
+        mutex_lock(n)
+
+
+def find_nearest_vehicle(n, vehicle):
+    with pyroscope.tag_wrapper({ "vehicle": vehicle}):
+        i = 0
+        start_time = time.time()
+        while time.time() - start_time < n:
+            i += 1
+        if vehicle == "car":
+            check_driver_availability(n)

examples/tracing/python/tempo/tempo.yml

@@ -0,0 +1,39 @@
+server:
+  http_listen_port: 3200
+
+query_frontend:
+  search:
+    duration_slo: 5s
+    throughput_bytes_slo: 1.073741824e+09
+  trace_by_id:
+    duration_slo: 5s
+
+distributor:
+  receivers:                           # this configuration will listen on all ports and protocols that tempo is capable of.
+    jaeger:                            # the receives all come from the OpenTelemetry collector.  more configuration information can
+      protocols:                       # be found there: https://github.com/open-telemetry/opentelemetry-collector/tree/main/receiver
+        thrift_http:                   #
+        grpc:                          # for a production deployment you should only enable the receivers you need!
+        thrift_binary:
+        thrift_compact:
+    zipkin:
+    otlp:
+      protocols:
+        http:
+        grpc:
+    opencensus:
+
+ingester:
+  max_block_duration: 5m               # cut the headblock when this much time passes. this is being set for demo purposes and should probably be left alone normally
+
+compactor:
+  compaction:
+    block_retention: 1h                # overall Tempo trace retention. set for demo purposes
+
+storage:
+  trace:
+    backend: local                     # backend configuration to use
+    wal:
+      path: /tmp/tempo/wal             # where to store the wal locally
+    local:
+      path: /tmp/tempo/blocks