The trace you are looking for was not found.

[hariprasadiit]

Self-Hosted Version

23.5.2

CPU Architecture

x86_64

Docker Version

20.10.23

Docker Compose Version

2.15.1

Steps to Reproduce

Help needed in debugging this issue.

Expected Result

Trace of any event to be shown.

Actual Result

Actual error from chrome network tab

detail: "Internal error. Your query failed to run."

Logs at the time of this error

sentry-self-hosted-web-1                                       | /usr/local/lib/python3.8/site-packages/sentry/models/avatars/base.py:74: DeprecationWarning: LANCZOS is deprecated and will be removed in Pillow 10 (2023-07-01). Use Resampling.LANCZOS instead.
sentry-self-hosted-web-1                                       |   image = image.resize((size, size), Image.LANCZOS)
sentry-self-hosted-web-1                                       | /usr/local/lib/python3.8/site-packages/sentry/web/frontend/base.py:662: ResourceWarning: unclosed file <_io.BufferedReader name='/data/files/7d/797e/8c890a4b66b6d2a25510ba2501'>
sentry-self-hosted-web-1                                       |   photo_file = avatar.get_cached_photo(size)
sentry-self-hosted-web-1                                       | ResourceWarning: Enable tracemalloc to get the object allocation traceback
sentry-self-hosted-web-1                                       | Traceback (most recent call last):
sentry-self-hosted-web-1                                       |   File "/usr/local/lib/python3.8/site-packages/sentry/snuba/referrer.py", line 630, in validate_referrer
sentry-self-hosted-web-1                                       |     raise Exception(error_message)
sentry-self-hosted-web-1                                       | Exception: referrer api.trace-view.count-performance-issues is not part of Referrer Enum
sentry-self-hosted-web-1                                       | 12:00:26 [WARNING] sentry.snuba.referrer: referrer api.trace-view.count-performance-issues is not part of Referrer Enum
sentry-self-hosted-web-1                                       | 12:00:26 [INFO] sentry.access.api: api.access (method='GET' view='sentry.api.endpoints.organization_events_trace.OrganizationEventsTraceEndpoint' response=200 user_id='1' is_app='None' token_type='None' is_frontend_request='True' organization_id='1' auth_id='None' path='/api/0/organizations/vizmo/events-trace/296452efc9894d7ea176cfbb54554eab/' caller_ip='172.23.0.36' user_agent='Mozilla/5.0 (Macintosh; Intel Mac OS X 13_3_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36 Edg/113.0.0.0' rate_limited='False' rate_limit_category='None' request_duration_seconds=0.20128297805786133 rate_limit_type='DNE' concurrent_limit='None' concurrent_requests='None' reset_time='None' group='None' limit='None' remaining='None')
sentry-self-hosted-web-1                                       | 12:00:26 [INFO] sentry.access.api: api.access (method='GET' view='sentry.api.endpoints.organization_events_trace.OrganizationEventsTraceMetaEndpoint' response=500 user_id='1' is_app='None' token_type='None' is_frontend_request='True' organization_id='1' auth_id='None' path='/api/0/organizations/vizmo/events-trace-meta/296452efc9894d7ea176cfbb54554eab/' caller_ip='172.23.0.36' user_agent='Mozilla/5.0 (Macintosh; Intel Mac OS X 13_3_1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/112.0.0.0 Safari/537.36 Edg/113.0.0.0' rate_limited='False' rate_limit_category='None' request_duration_seconds=0.23880314826965332 rate_limit_type='DNE' concurrent_limit='None' concurrent_requests='None' reset_time='None' group='None' limit='None' remaining='None')
sentry-self-hosted-web-1                                       | 12:00:26 [ERROR] django.request: Internal Server Error: /api/0/organizations/vizmo/events-trace-meta/296452efc9894d7ea176cfbb54554eab/ (status_code=500 request=<WSGIRequest: GET '/api/0/organizations/vizmo/events-trace-meta/296452efc9894d7ea176cfbb54554eab/?end=2023-06-02T23%3A55%3A21.508&project=-1&query=&start=2023-06-01T23%3A55%3A21.508'>)

Recently I've fixed issues with Kafka out of range errors

One more thing we've noticed is that, at the same time notifications stopped working. We're no longer receiving error event related email notifications

This issue started occurring after upgrading to 23.5.1 from 23.3.1

[hariprasadiit]

https://sentry.vizmo.dev/api/0/organizations/vizmo/events-trace/a39313f97a164153a59c94c070a49b1a/?detailed=1&end=2023-06-03T00:34:20.449&project=-1&query=&start=2023-06-02T00:34:20.449

this request is successful while

https://sentry.vizmo.dev/api/0/organizations/vizmo/events-trace-meta/a39313f97a164153a59c94c070a49b1a/?end=2023-06-03T00:34:20.449&project=-1&query=&start=2023-06-02T00:34:20.449 request is failing

[hubertdeng123]

Just to confirm, you're getting a 500 error upon clicking into any issue? Can you check snuba logs to see if there's anything useful there?

[hariprasadiit]

I can see all the issue details, however when I click on View full trace to view trace of the relevant request, it shows not found error message.

When sentry tries to load events-trace-meta it fails but events-trace loads successfully as mentioned in the above comment

I can see exception from snuba api. not sure if that is the cause of it

sentry-self-hosted-snuba-api-1  | 12. DB::TCPHandler::runImpl() @ 0x9024489 in /usr/bin/clickhouse
sentry-self-hosted-snuba-api-1  | 13. DB::TCPHandler::run() @ 0x9025470 in /usr/bin/clickhouse
sentry-self-hosted-snuba-api-1  | 14. Poco::Net::TCPServerConnection::start() @ 0xe3ac69b in /usr/bin/clickhouse
sentry-self-hosted-snuba-api-1  | 15. Poco::Net::TCPServerDispatcher::run() @ 0xe3acb1d in /usr/bin/clickhouse
sentry-self-hosted-snuba-api-1  | 16. Poco::PooledThread::run() @ 0x105c3317 in /usr/bin/clickhouse
sentry-self-hosted-snuba-api-1  | 17. Poco::ThreadImpl::runnableEntry(void*) @ 0x105bf11c in /usr/bin/clickhouse
sentry-self-hosted-snuba-api-1  | 18. ? @ 0x105c0abd in /usr/bin/clickhouse
sentry-self-hosted-snuba-api-1  | 19. start_thread @ 0x76db in /lib/x86_64-linux-gnu/libpthread-2.27.so
sentry-self-hosted-snuba-api-1  | 20. __clone @ 0x12188f in /lib/x86_64-linux-gnu/libc-2.27.so
sentry-self-hosted-snuba-api-1  |
sentry-self-hosted-snuba-api-1  |
sentry-self-hosted-snuba-api-1  | The above exception was the direct cause of the following exception:
sentry-self-hosted-snuba-api-1  |
sentry-self-hosted-snuba-api-1  | Traceback (most recent call last):
sentry-self-hosted-snuba-api-1  |   File "/usr/src/snuba/./snuba/web/db_query.py", line 572, in _raw_query
sentry-self-hosted-snuba-api-1  |     result = execute_query_with_query_id(
sentry-self-hosted-snuba-api-1  |   File "/usr/src/snuba/./snuba/utils/metrics/util.py", line 70, in wrapper
sentry-self-hosted-snuba-api-1  |     return func(*args, **kwargs)
sentry-self-hosted-snuba-api-1  |   File "/usr/src/snuba/./snuba/web/db_query.py", line 368, in execute_query_with_query_id
sentry-self-hosted-snuba-api-1  |     return execute_query_with_readthrough_caching(
sentry-self-hosted-snuba-api-1  |   File "/usr/src/snuba/./snuba/utils/metrics/util.py", line 70, in wrapper
sentry-self-hosted-snuba-api-1  |     return func(*args, **kwargs)
sentry-self-hosted-snuba-api-1  |   File "/usr/src/snuba/./snuba/web/db_query.py", line 465, in execute_query_with_readthrough_caching
sentry-self-hosted-snuba-api-1  |     return cache_partition.get_readthrough(
sentry-self-hosted-snuba-api-1  |   File "/usr/src/snuba/./snuba/state/cache/redis/backend.py", line 198, in get_readthrough
sentry-self-hosted-snuba-api-1  |     raise e
sentry-self-hosted-snuba-api-1  |   File "/usr/src/snuba/./snuba/state/cache/redis/backend.py", line 176, in get_readthrough
sentry-self-hosted-snuba-api-1  |     value = self.__executor.submit(function).result(task_timeout)
sentry-self-hosted-snuba-api-1  |   File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 444, in result
sentry-self-hosted-snuba-api-1  |     return self.__get_result()
sentry-self-hosted-snuba-api-1  |   File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 389, in __get_result
sentry-self-hosted-snuba-api-1  |     raise self._exception
sentry-self-hosted-snuba-api-1  |   File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run
sentry-self-hosted-snuba-api-1  |     result = self.fn(*self.args, **self.kwargs)
sentry-self-hosted-snuba-api-1  |   File "/usr/src/snuba/./snuba/utils/metrics/util.py", line 70, in wrapper
sentry-self-hosted-snuba-api-1  |     return func(*args, **kwargs)
sentry-self-hosted-snuba-api-1  |   File "/usr/src/snuba/./snuba/web/db_query.py", line 314, in execute_query_with_rate_limits
sentry-self-hosted-snuba-api-1  |     return execute_query(
sentry-self-hosted-snuba-api-1  |   File "/usr/src/snuba/./snuba/utils/metrics/util.py", line 70, in wrapper
sentry-self-hosted-snuba-api-1  |     return func(*args, **kwargs)
sentry-self-hosted-snuba-api-1  |   File "/usr/src/snuba/./snuba/web/db_query.py", line 194, in execute_query
sentry-self-hosted-snuba-api-1  |     result = reader.execute(
sentry-self-hosted-snuba-api-1  |   File "/usr/src/snuba/./snuba/clickhouse/native.py", line 499, in execute
sentry-self-hosted-snuba-api-1  |     execute_func(
sentry-self-hosted-snuba-api-1  |   File "/usr/src/snuba/./snuba/clickhouse/native.py", line 268, in execute
sentry-self-hosted-snuba-api-1  |     raise ClickhouseError(e.message, code=e.code) from e
sentry-self-hosted-snuba-api-1  | snuba.clickhouse.errors.ClickhouseError: DB::Exception: Table default.search_issues_local_v2 doesn't exist.. Stack trace:
sentry-self-hosted-snuba-api-1  |
sentry-self-hosted-snuba-api-1  | 0. Poco::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0x105351b0 in /usr/bin/clickhouse
sentry-self-hosted-snuba-api-1  | 1. DB::Exception::Exception(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, int) @ 0x8f4172d in /usr/bin/clickhouse
sentry-self-hosted-snuba-api-1  | 2. DB::Context::getTableImpl(DB::StorageID const&, std::__1::optional<DB::Exception>*) const @ 0xcfe2a24 in /usr/bin/clickhouse
sentry-self-hosted-snuba-api-1  | 3. DB::Context::getTable(DB::StorageID const&) const @ 0xcfe2bbb in /usr/bin/clickhouse
sentry-self-hosted-snuba-api-1  | 4. DB::Context::getTable(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&) const @ 0xcfe2c7d in /usr/bin/clickhouse
sentry-self-hosted-snuba-api-1  | 5. DB::JoinedTables::getLeftTableStorage() @ 0xd454892 in /usr/bin/clickhouse
sentry-self-hosted-snuba-api-1  | 6. DB::InterpreterSelectQuery::InterpreterSelectQuery(std::__1::shared_ptr<DB::IAST> const&, DB::Context const&, std::__1::shared_ptr<DB::IBlockInputStream> const&, std::__1::optional<DB::Pipe>, std::__1::shared_ptr<DB::IStorage> const&, DB::SelectQueryOptions const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) @ 0xd13b6d1 in /usr/bin/clickhouse
sentry-self-hosted-snuba-api-1  | 7. DB::InterpreterSelectQuery::InterpreterSelectQuery(std::__1::shared_ptr<DB::IAST> const&, DB::Context const&, DB::SelectQueryOptions const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) @ 0xd13c619 in /usr/bin/clickhouse
sentry-self-hosted-snuba-api-1  | 8. DB::InterpreterSelectWithUnionQuery::InterpreterSelectWithUnionQuery(std::__1::shared_ptr<DB::IAST> const&, DB::Context const&, DB::SelectQueryOptions const&, std::__1::vector<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >, std::__1::allocator<std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > > > const&) @ 0xd341686 in /usr/bin/clickhouse
sentry-self-hosted-snuba-api-1  | 9. DB::InterpreterFactory::get(std::__1::shared_ptr<DB::IAST>&, DB::Context&, DB::QueryProcessingStage::Enum) @ 0xd0909b4 in /usr/bin/clickhouse
sentry-self-hosted-snuba-api-1  | 10. ? @ 0xd550655 in /usr/bin/clickhouse
sentry-self-hosted-snuba-api-1  | 11. DB::executeQuery(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, DB::Context&, bool, DB::QueryProcessingStage::Enum, bool, bool) @ 0xd553441 in /usr/bin/clickhouse
sentry-self-hosted-snuba-api-1  | 12. DB::TCPHandler::runImpl() @ 0x9024489 in /usr/bin/clickhouse
sentry-self-hosted-snuba-api-1  | 13. DB::TCPHandler::run() @ 0x9025470 in /usr/bin/clickhouse
sentry-self-hosted-snuba-api-1  | 14. Poco::Net::TCPServerConnection::start() @ 0xe3ac69b in /usr/bin/clickhouse
sentry-self-hosted-snuba-api-1  | 15. Poco::Net::TCPServerDispatcher::run() @ 0xe3acb1d in /usr/bin/clickhouse
sentry-self-hosted-snuba-api-1  | 16. Poco::PooledThread::run() @ 0x105c3317 in /usr/bin/clickhouse
sentry-self-hosted-snuba-api-1  | 17. Poco::ThreadImpl::runnableEntry(void*) @ 0x105bf11c in /usr/bin/clickhouse
sentry-self-hosted-snuba-api-1  | 18. ? @ 0x105c0abd in /usr/bin/clickhouse
sentry-self-hosted-snuba-api-1  | 19. start_thread @ 0x76db in /lib/x86_64-linux-gnu/libpthread-2.27.so
sentry-self-hosted-snuba-api-1  | 20. __clone @ 0x12188f in /lib/x86_64-linux-gnu/libc-2.27.so

[hubertdeng123]

Can you check your clickhouse db manually to see if the table search_issues_local_v2 exists? Looks like it was part of a snuba migration that was added a few months ago.

[kadet1090]

I had the same issue, you need to force migration in the snuba because for some reason it's not executed by default even if the storage set "search_issues" is specified in the config. You can force this migration group using:

snuba migrations migrate -g search_issues --force

[hariprasadiit]

Can you check your clickhouse db manually to see if the table search_issues_local_v2 exists? Looks like it was part of a snuba migration that was added a few months ago.

Yes, the table was missing.

I had the same issue, you need to force migration in the snuba because for some reason it's not executed by default even if the storage set "search_issues" is specified in the config. You can force this migration group using:

snuba migrations migrate -g search_issues --force

This fixed the issue.

Thanks for the help!!