bug-1889185: implement stage submitter and fake collector #6617
Conversation
| @@ -54,6 +54,7 @@ echo ">>> run tests" | |||
|
|
|||
| # Run socorro tests | |||
| "${PYTEST}" | |||
| CLOUD_PROVIDER=GCP "${PYTEST}" -m gcp | |||
There was a problem hiding this comment.
The submitter tests use queue_helper and storage_helper, so we need to do AWS and GCP tests now for socorro tests.
| @@ -103,6 +103,10 @@ SESSION_COOKIE_SECURE=False | |||
| # ----------- | |||
| SENTRY_PORT=8090 | |||
|
|
|||
| # stage_submitter | |||
| # --------------- | |||
| STAGE_SUBMITTER_DESTINATIONS=http://fakecollector:8000/submit|20 | |||
There was a problem hiding this comment.
This sets up a single destination sampled at 20%.
willkg
force-pushed
the
willkg-bug-1889185-submitter
branch
from
1155fc2 to
04fad15
Compare
| from werkzeug.exceptions import HTTPException, MethodNotAllowed | ||
| from werkzeug.formparser import parse_form_data | ||
| from werkzeug.routing import Map, Rule | ||
| from werkzeug.wrappers import Request, Response |
There was a problem hiding this comment.
The only web framework we have available is Django. I didn't want to use Django for the fake collector because that involves a whole bunch of additional set up for the db I didn't want to do.
Flask is an ergonomic shell on top of werkzeug. We don't need templates, localizations, extensions, etc so we can use raw werkzeug to build a very minimal API for /submit.
| content_encoding = request.headers.get("Content-Encoding", "") | ||
| self.logger.info("content_encoding: %s", content_encoding) | ||
|
|
||
| stream, form, files = parse_form_data(request.environ, max_content_length=content_length) |
There was a problem hiding this comment.
This worked surprisingly well.
Even so, I want to see how parse_form_data deals with some of the malformed data issues we deal with in Antenna. Depending on that, switching Antenna from Falcon (which uses the deprecated-to-be-removed cgi module) to Flask seems viable.
| ] | ||
|
|
||
|
|
||
| def remove_collector_keys(raw_crash): |
There was a problem hiding this comment.
Large portions of this file were copied verbatim from socorro-submitter repository where they've been running for years.
| return repr(thing).encode("utf-8") | ||
|
|
||
|
|
||
| def multipart_encode(raw_crash, dumps, payload_type, payload_compressed): |
There was a problem hiding this comment.
This is copied from socorro-submitter.
| if dest.sample >= 100 or random.randint(0, 100) > dest.sample | ||
| ] | ||
|
|
||
| def process(self, crash): |
There was a problem hiding this comment.
The structure of this was copied from the socorro-submitter.
|
|
||
| self.shutdown() | ||
|
|
||
| def run_once(self): |
There was a problem hiding this comment.
run_once is used in tests.
| @@ -0,0 +1,594 @@ | |||
| # This Source Code Form is subject to the terms of the Mozilla Public | |||
There was a problem hiding this comment.
The tests were lifted from socorro-submitter and then adjusted to work with the SubmitterApp instead of the socorro-submitter lambda function.
This implements a stage submitter that pulls crash ids from QUEUE and crash data from STORAGE, packages that up into a crash report payload, and submits it to STAGE_SUBMITTER_DESTINATIONS. This creates a `make runsubmitter` system that runs the stage submitter and a fake collector allowing you to use `bin/process_crash.sh` to set everything up and publish a crash id to the standard queue which the submitter will see and submit a crash to the fake collector. The fake collector will parse the payload and log headers, annotations, and files. This adds stage submitter tests.
willkg
force-pushed
the
willkg-bug-1889185-submitter
branch
from
4c006d9 to
d269f85
Compare
| addopts = -rsxX --tb=native -p no:django -m 'not gcp' | ||
| norecursedirs = .git docs config docker __pycache__ | ||
| testpaths = socorro/ |
There was a problem hiding this comment.
pretty sure marker definitions need to be copied from webapp/pytest.ini as well:
| addopts = -rsxX --tb=native -p no:django -m 'not gcp' | |
| norecursedirs = .git docs config docker __pycache__ | |
| testpaths = socorro/ | |
| addopts = -rsxX --tb=native -p no:django -m 'not gcp' | |
| norecursedirs = .git docs config docker __pycache__ | |
| testpaths = socorro/ | |
| markers = | |
| aws: tests that require aws backends to be configured in the environment. this is the default. | |
| gcp: tests that require gcp backends to be configured in the environment. skipped unless explicitly requested. |
There was a problem hiding this comment.
I'll add that. I'm not sure how it works currently without those markers.
| def generate_s3_key(kind, crash_id): | ||
| """Generates the key in S3 for this object kind |
There was a problem hiding this comment.
this isn't just used for s3, i suggest we rename it:
| def generate_s3_key(kind, crash_id): | |
| """Generates the key in S3 for this object kind | |
| def generate_storage_key(kind, crash_id): | |
| """Generates the key in cloud storage for this object kind |
There was a problem hiding this comment.
Whoops--this is a copy/paste fail.
|
Thank you! |
This implements a stage submitter that pulls crash ids from QUEUE and crash data from STORAGE, packages that up into a crash report payload, and submits it to STAGE_SUBMITTER_DESTINATIONS.
This creates a
make runsubmittersystem that runs the stage submitter and a fake collector allowing you to usebin/process_crash.shto set everything up and publish a crash id to the standard queue which the submitter will see and submit a crash to the fake collector. The fake collector will parse the payload and log headers, annotations, and files.This adds stage submitter tests.
To test:
make buildmake setupmake runservicesmake runsubmittermake shelland then you can usebin/process_crash.shto download crash data, put it in storage, and publish the crash id to the standard queuesocorro.submitter.sampleincr metricsocorro.submitter.acceptincr metric and you'll see the headers, annotations, and files logged by the fake collectorLight documentation is provided in
docs/services/stage_submitter.rst.