Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Failing tests #1398

Closed
samuela opened this issue Apr 18, 2022 · 9 comments
Closed

Failing tests #1398

samuela opened this issue Apr 18, 2022 · 9 comments
Labels
Component: Tests Dealing with tests

Comments

@samuela
Copy link

samuela commented Apr 18, 2022

How do you use Sentry?

Self-hosted/on-premise

Version

1.5.10

Steps to Reproduce

  1. Checkout sentry-sdk at version 1.5.10.
  2. Attempt to run the test suite.
  3. Observe the following errors:
=========================== short test summary info ============================
FAILED tests/integrations/django/test_basic.py::test_user_captured
FAILED tests/integrations/bottle/test_bottle.py::test_transaction_style[/message-endpoint-hi]
FAILED tests/integrations/bottle/test_bottle.py::test_transaction_style[/message-url-/message]
FAILED tests/integrations/bottle/test_bottle.py::test_transaction_style[/message-named-route-endpoint-hi]
FAILED tests/integrations/bottle/test_bottle.py::test_large_json_request
FAILED tests/integrations/bottle/test_bottle.py::test_empty_json_request[empty-dict]
FAILED tests/integrations/bottle/test_bottle.py::test_empty_json_request[empty-list]
FAILED tests/integrations/bottle/test_bottle.py::test_medium_formdata_request
FAILED tests/integrations/bottle/test_bottle.py::test_too_large_raw_request[a0]
FAILED tests/integrations/bottle/test_bottle.py::test_too_large_raw_request[a1]
FAILED tests/integrations/bottle/test_bottle.py::test_files_and_form
FAILED tests/integrations/bottle/test_bottle.py::test_500
FAILED tests/integrations/django/test_basic.py::test_does_not_capture_403[permission_denied_exc]
FAILED tests/integrations/django/test_basic.py::test_request_captured
FAILED tests/integrations/django/test_basic.py::test_render_spans
FAILED tests/integrations/django/test_basic.py::test_transaction_with_class_view
FAILED tests/integrations/django/test_basic.py::test_middleware_spans
FAILED tests/integrations/django/test_basic.py::test_custom_error_handler_request_context
FAILED tests/integrations/django/test_basic.py::test_middleware_spans_disabled
FAILED tests/integrations/django/test_basic.py::test_500
FAILED tests/integrations/django/test_basic.py::test_csrf
FAILED tests/integrations/django/test_basic.py::test_transaction_style[function_name-tests.integrations.django.myapp.views.message]
FAILED tests/integrations/django/test_basic.py::test_custom_urlconf_middleware
FAILED tests/integrations/django/test_basic.py::test_transaction_style[url-/message]
FAILED tests/integrations/django/test_basic.py::test_request_body
FAILED tests/integrations/flask/test_flask.py::test_flask_medium_formdata_request
FAILED tests/integrations/flask/test_flask.py::test_flask_formdata_request_appear_transaction_body
FAILED tests/integrations/flask/test_flask.py::test_flask_too_large_raw_request[a0]
FAILED tests/integrations/django/test_basic.py::test_template_exception[with_executing_integration0]
FAILED tests/integrations/flask/test_flask.py::test_flask_too_large_raw_request[a1]
FAILED tests/integrations/flask/test_flask.py::test_flask_files_and_form
FAILED tests/integrations/django/test_basic.py::test_template_exception[with_executing_integration1]
FAILED tests/integrations/flask/test_flask.py::test_cli_commands_raise
FAILED tests/integrations/pyramid/test_pyramid.py::test_flask_empty_json_request[empty-dict]
FAILED tests/integrations/pyramid/test_pyramid.py::test_flask_empty_json_request[empty-list]
FAILED tests/integrations/pyramid/test_pyramid.py::test_errorhandler_500
FAILED tests/integrations/flask/test_flask.py::test_sentry_trace_context
FAILED tests/integrations/flask/test_flask.py::test_dont_override_sentry_trace_context
FAILED tests/integrations/trytond/test_trytond.py::test_rpc_error_page
ERROR tests/integrations/flask/test_flask.py::test_errors_not_reported_twice[integrations0]
ERROR tests/integrations/flask/test_flask.py::test_transaction_style[endpoint-hi]
ERROR tests/integrations/flask/test_flask.py::test_transaction_style[url-/message]
ERROR tests/integrations/flask/test_flask.py::test_errors_not_reported_twice[integrations1]
ERROR tests/integrations/flask/test_flask.py::test_logging
ERROR tests/integrations/flask/test_flask.py::test_errors[auto-True-True]
ERROR tests/integrations/flask/test_flask.py::test_errors[auto-True-False]
ERROR tests/integrations/flask/test_flask.py::test_500
ERROR tests/integrations/flask/test_flask.py::test_error_in_errorhandler
ERROR tests/integrations/flask/test_flask.py::test_errors[auto-False-True]
ERROR tests/integrations/flask/test_flask.py::test_flask_login_not_configured[manual]
ERROR tests/integrations/flask/test_flask.py::test_flask_login_partially_configured[auto]
ERROR tests/integrations/flask/test_flask.py::test_errors[auto-False-False]
ERROR tests/integrations/flask/test_flask.py::test_flask_login_partially_configured[manual]
ERROR tests/integrations/flask/test_flask.py::test_flask_login_configured[auto-None-True]
ERROR tests/integrations/flask/test_flask.py::test_errors[manual-True-True]
ERROR tests/integrations/flask/test_flask.py::test_flask_login_configured[auto-None-False]
ERROR tests/integrations/flask/test_flask.py::test_errors[manual-True-False]
ERROR tests/integrations/flask/test_flask.py::test_flask_login_configured[auto-42-True]
ERROR tests/integrations/flask/test_flask.py::test_flask_login_configured[auto-42-False]
ERROR tests/integrations/flask/test_flask.py::test_flask_login_configured[auto-3-True]
ERROR tests/integrations/flask/test_flask.py::test_errors[manual-False-True]
ERROR tests/integrations/flask/test_flask.py::test_flask_login_configured[auto-3-False]
ERROR tests/integrations/flask/test_flask.py::test_flask_login_configured[manual-None-True]
ERROR tests/integrations/flask/test_flask.py::test_errors[manual-False-False]
ERROR tests/integrations/flask/test_flask.py::test_flask_login_configured[manual-None-False]
ERROR tests/integrations/flask/test_flask.py::test_flask_login_not_installed[auto]
ERROR tests/integrations/flask/test_flask.py::test_flask_login_configured[manual-42-True]
ERROR tests/integrations/flask/test_flask.py::test_flask_login_configured[manual-42-False]
ERROR tests/integrations/flask/test_flask.py::test_flask_login_not_installed[manual]
ERROR tests/integrations/flask/test_flask.py::test_flask_login_configured[manual-3-True]
ERROR tests/integrations/flask/test_flask.py::test_flask_login_configured[manual-3-False]
ERROR tests/integrations/flask/test_flask.py::test_flask_login_not_configured[auto]
ERROR tests/integrations/flask/test_flask.py::test_flask_session_tracking
ERROR tests/integrations/flask/test_flask.py::test_tracing_success
ERROR tests/integrations/flask/test_flask.py::test_tracing_error
ERROR tests/integrations/flask/test_flask.py::test_class_based_views
ERROR tests/integrations/flask/test_flask.py::test_sentry_trace_context
ERROR tests/integrations/flask/test_flask.py::test_dont_override_sentry_trace_context
= 39 failed, 561 passed, 31 skipped, 2 xfailed, 5 warnings, 39 errors in 48.49s =

Full error log available here: https://gist.github.com/samuela/0c1701f2169c2e7b9c48822be8947865.

I'm using the following versions

  • sentry-sdk v1.5.10. I've also reproduced the same failures on v1.5.8.
  • aiohttp 3.8.1
  • certifi 2021.10.08
  • celery 5.2.3
  • urllib3 1.26.9
  • asttokens 2.0.5
  • blinker 1.4
  • botocore 1.24.33
  • bottle 0.12.19
  • chalice 1.26.6
  • Django 3.2.13
  • executing 0.8.2
  • fakeredis 1.7.1
  • falcon unstable-2018-10-23 (falconpl/falcon@637e2d5)
  • flask-login 0.6.0
  • gevent 21.12.0
  • httpx 1.2.1
  • jsonschema 4.4.0
  • pure-eval 0.2.1
  • pyramid 2.0
  • pyspark 3.2.1
  • pytest-django 4.5.2
  • pytest-forked 1.4.0
  • pytest-localserver 0.5.1.post0
  • rq 1.0.2
  • sanic 21.12.1
  • sanic-testing 0.8.2
  • sqlalchemy 1.4.35
  • tornado 6.1
  • trytond 6.2.6
  • werkzeug 2.1.0

As an alternative to checking out these exact versions manually, you can reproduce by cloning NixOS/nixpkgs@bf4f857, and running nix-build -A python3Packages.sentry-sdk.

Expected Result

The test suite to pass.

Actual Result

39 test failures.
@samuela samuela mentioned this issue Apr 18, 2022
Closed
@gador
Copy link

gador commented Apr 19, 2022

I noticed a requirement of an older version of werkzeug. When using, the tests mostly pass.

@samuela
Copy link
Author

samuela commented Apr 19, 2022

Where is this requirement? I'm not doing any patching of the requirement version constraints.

@gador
Copy link

gador commented Apr 19, 2022

Here

sentry-python/test-requirements.txt

Line 5 in 1878f9b

Werkzeug<2.1.0

@samuela
Copy link
Author

samuela commented Apr 19, 2022

Ahhh, I see... that's why it wasn't enforced automatically. Ok good catch!

@antonpirker
Copy link
Member

Hello!

So you ran the tests with a newer Werkzeug (version 2.1.0) and the tests failed. So the test suite is working because the Werkzeug version is pinned in the test_requirements.txt. So I will close this issue. Thanks @samuela for reporting and @gador for helping with a solution.

PS: The test suite is currently not in the best shape and we (Sentry) know this. But we plan to improve it a lot.

@antonpirker
Copy link
Member

If you both @samuela and @gador send me a your mailing address to anton (dot) pirker {at} sentry [dot] io I can send you a couple of stickers as a little "Thank you".

@antonpirker antonpirker added Component: Tests Dealing with tests and removed Status: Untriaged labels Apr 19, 2022
@samuela
Copy link
Author

samuela commented Apr 19, 2022

Thanks for the update @antonpirker! And for your generous offer of stickers! I'll take a stab with an older werkzeug.

@gador
Copy link

gador commented Apr 19, 2022

@antonpirker thanks 😄
@samuela: The older werkzeug still fails with test_cli_commands_raise. Unsure why and will investigate further. This probably has something to do with other requirements of sentry being updated.

@gador
Copy link

gador commented Apr 19, 2022

ok, last update:
As posted here NixOS/nixpkgs#169130 (comment)
the issue is in NixOS using the newer version of flask and werkzeug which break the test suite here.

I am unsure of the impact of the SDK changes in werkzeug but they do appear to break a lot of tests.

samuela referenced this issue in NixOS/nixpkgs Apr 19, 2022
@mweinelt
python3Packages.Flask: 2.0.3 -> 2.1.1
3caf52b
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Component: Tests Dealing with tests
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@antonpirker @samuela @gador