Error when viewing DB in UI #103501

jrote1 · 2023-05-17T09:35:53Z

I'm getting an error when trying to view the database page, i'm not sure what's causing this but i'm happy to provide any log files that are required. All other databases can be viewed in the UI, it's just this one with the error

Please describe the issue you observed, and any steps we can take to reproduce it:

To Reproduce

I'm not sure what caused this error I just created a database and imported a postgres backup

Additional data / screenshots

All nodes are live with no errors

If a node in your cluster encountered a fatal error, supply the contents of the
log directories (at minimum of the affected node(s), but preferably all nodes).

Note that log files can contain confidential information. Please continue
creating this issue, but contact [email protected] to submit the log
files in private.

If applicable, add screenshots to help explain your problem.

Environment:

CockroachDB version: 23.1.1
Server OS: Linux/Docker
Client app: UI

Additional context
Unable to view the DB page

Add any other context about the problem here.

Jira issue: CRDB-28035

blathers-crl · 2023-05-17T09:35:57Z

Hello, I am Blathers. I am here to help you get the issue triaged.

It looks like you have not filled out the issue in the format of any of our templates. To best assist you, we advise you to use one of these templates.

I have CC'd a few people who may be able to assist you:

@cockroachdb/disaster-recovery (found keywords: backup)
@cockroachdb/sql-queries (found keywords: import)

If we have not gotten back to your issue within a few business days, you can try the following:

Join our community slack channel and ask on #cockroachdb.
Try find someone from here if you know they worked closely on the area and CC them.

_{🦉 Hoot! I am a Blathers, a bot for CockroachDB. My owner is dev-inf.}

blathers-crl · 2023-05-17T09:36:01Z

cc @cockroachdb/disaster-recovery

jrote1 · 2023-05-17T21:33:09Z

I think this is related to #103443

florence-crl · 2023-05-22T21:14:37Z

cc @kevin-v-ngo for triage
I also think this is related to #103443
In 23.1.1 with a single-node secure local cluster with a non-admin user without VIEWACTIVITY privilege, I also get the error:

This page had an unexpected error while loading databases.
 Reload this page

From the Console:

Failed to load resource: the server responded with a status of 403 (Forbidden)
bundle.js:1895 Error Loading databases: Error: Error while retrieving database details information:  only users with MODIFYCLUSTERSETTING, MODIFYSQLCLUSTERSETTING or VIEWCLUSTERSETTING system privileges are allowed to read crdb_internal.cluster_settings
a @ bundle.js:1895
:8080/_status/nodes_ui:1     Failed to load resource: the server responded with a status of 401 (Unauthorized)
:8080/api/v2/sql/:1     Failed to load resource: the server responded with a status of 401 (Unauthorized)
shared.js:77 Uncaught Error: Attempting to use a disconnected port object
    at a.post (shared.js:77:1110937)
    at e.post (shared.js:77:86484)
    at Object.getTabInfo (shared.js:77:1078778)
    at s.getLogoUrls (shared.js:77:1117878)
    at Nt.PageMonitor.H (shared.js:77:1239941)
    at b (shared.js:77:24341)
    at Nt.PageMonitor.trigger (shared.js:77:23693)
    at Nt.PageMonitor.scan (shared.js:77:1233628)
    at Object.onChange (shared.js:77:1232061)
    at u (shared.js:77:112987)
2bundle.js:1895 Error Loading databases: RequestError: Unauthorized

This is a regression, because in 22.2.9, I get the info message:

RequestError: this operation requires the VIEWACTIVITY system privilege [Learn more](https://www.cockroachlabs.com/docs/stable/ui-overview.html#db-console-access)

THardy98 · 2023-05-22T22:33:34Z

Are you able to load the page successfully as admin/root user or a user with VIEWACTIVITY privilege?

My suspicion is that this isn't related to #103443 but would like to see if that is the case.

Additionally, if you could share the response being sent back on the console, that would be helpful.

florence-crl · 2023-05-23T13:27:28Z

ok, I gave a non-admin user the following privilege:

grant SYSTEM VIEWACTIVITY to max;

In 23.1.1

From console:

5bundle.js:1895 Error Loading databases: Error: Error while retrieving database details information:  crdb_internal.get_database_id()
a @ bundle.js:1895
bundle.js:1895 Error Loading databases: Error: Error while retrieving database details information:  only users with MODIFYCLUSTERSETTING, MODIFYSQLCLUSTERSETTING or VIEWCLUSTERSETTING system privileges are allowed to read crdb_internal.cluster_settings
a @ bundle.js:1895
499Failed to decode downloaded font: <URL>
499OTS parsing error: invalid sfntVersion: -2147375342
bundle.js:1895 Error Loading databases: Error: Error while retrieving database details information:  crdb_internal.get_database_id()
a @ bundle.js:1895
:8080/_status/nodes_ui:1     Failed to load resource: the server responded with a status of 401 (Unauthorized)
2bundle.js:1895 Error Loading databases: Error: Error while retrieving database details information:  crdb_internal.get_database_id()

From console:

Failed to load resource: the server responded with a status of 401 (Unauthorized)
:8080/_admin/v1/settings?unredacted_values=true:1     Failed to load resource: the server responded with a status of 403 (Forbidden)
5bundle.js:1895 Error Loading databases: Error: Error while retrieving database details information:  crdb_internal.get_database_id()
a @ bundle.js:1895
bundle.js:1895 Error Loading databases: Error: Error while retrieving database details information:  only users with MODIFYCLUSTERSETTING, MODIFYSQLCLUSTERSETTING or VIEWCLUSTERSETTING system privileges are allowed to read crdb_internal.cluster_settings

In 22.2.9

Snippet from console:

bundle.js:1895 Error Loading databases: Error: Error while retrieving database details information:  crdb_internal.get_database_id()
a @ bundle.js:1895
:8080/_status/nodes_ui:1     Failed to load resource: the server responded with a status of 401 (Unauthorized)
2bundle.js:1895 Error Loading databases: Error: Error while retrieving database details information:  crdb_internal.get_database_id()

GET https://localhost:8081/_admin/v1/settings?unredacted_values=true 403 (Forbidden)
kn @ bundle.js:27177
wn @ bundle.js:27177
(anonymous) @ bundle.js:27182
(anonymous) @ bundle.js:27324
(anonymous) @ bundle.js:27188
V @ bundle.js:15458
(anonymous) @ bundle.js:27325
(anonymous) @ bundle.js:16850
(anonymous) @ bundle.js:27324
dispatch @ bundle.js:15458
(anonymous) @ bundle.js:27182
bundle.js:27177     GET https://localhost:8081/_admin/v1/settings?unredacted_values=true 403 (Forbidden)

jrote1 · 2023-05-23T14:02:10Z

@THardy98 i'm logged in as an admin and am seeing the following error in the console

with the following error observed in the sql request

{
   "code":"XXUUU",
   "message":"executing stmt 7: run-query-via-api: error requesting iterating nodes for span stats from node 1 (NODE_STATUS_LIVE): grpc: invalid key-range specified (start \u003e end) [code 2/Unknown]",
   "severity":"ERROR",
   "source":{
      "file":"status.go",
      "line":2886,
      "function":"func1"
   }
}

Also i'm only getting this error for a specific DB the rest are working okay, if there's any over information I can provide please just let me know

jrote1 · 2023-05-23T14:07:06Z

It looks like this is the offending statement as it's statement 7 in the list, but i'll confirm by running it when I get back later

SELECT
  sum(range_count) as range_count,
  sum(approximate_disk_bytes) as approximate_disk_bytes,
  sum(live_bytes) as live_bytes,
  sum(total_bytes) as total_bytes
FROM crdb_internal.tenant_span_stats((SELECT crdb_internal.get_database_id('nextcloud')))

I did run it via the API and got the response I saw above

THardy98 · 2023-05-23T16:11:26Z

@florence-crl my mistake, non-admin users need at least both VIEWACTIVITY and VIEWCLUSTERSETTING privileges to view the databases pages.

@jrote1 Yes this is what I was thinking was causing the problem, this is not a privilege issue.

@jbowens this looks similar to the key-range ordering issue we encountered before. Key ordering issues would occur when we tried to get disk size, though it could also be occuring when we try to get mvcc stats.

jbowens · 2023-05-23T16:16:06Z

@THardy98 the bug fix was only backported to the release-23.1 branch and not release.23.1.0, so it won’t be fixed until 23.1.2

jrote1 · 2023-05-23T23:05:10Z

@jbowens @THardy98 does this mean this is expected to be fixed in the next release? If so do you have a release date and i'm happy to respond if the issue is fixed :)

THardy98 · 2023-05-24T13:47:54Z

Yes, release date is May 30th

jrote1 · 2023-05-26T07:31:14Z

@THardy98 i've updated to the latest version and the issue is fixed. Thanks for your help

jrote1 added the C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. label May 17, 2023

blathers-crl bot added A-disaster-recovery O-community Originated from the community X-blathers-triaged blathers was able to find an owner T-disaster-recovery labels May 17, 2023

blathers-crl bot added the T-cluster-observability label May 17, 2023

yuzefovich removed A-disaster-recovery T-disaster-recovery labels May 17, 2023

florence-crl added the A-cluster-observability Related to cluster observability label May 22, 2023

exalate-issue-sync bot mentioned this issue May 22, 2023

PR #95303 - ui: hide reset options for non-admins cockroachdb/docs#16012

Open

maryliag assigned THardy98 May 23, 2023

jrote1 closed this as completed May 26, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Error when viewing DB in UI #103501

Error when viewing DB in UI #103501

jrote1 commented May 17, 2023 •

edited

Loading

blathers-crl bot commented May 17, 2023

blathers-crl bot commented May 17, 2023

jrote1 commented May 17, 2023

florence-crl commented May 22, 2023

THardy98 commented May 22, 2023 •

edited

Loading

florence-crl commented May 23, 2023

jrote1 commented May 23, 2023

jrote1 commented May 23, 2023 •

edited

Loading

THardy98 commented May 23, 2023

jbowens commented May 23, 2023

jrote1 commented May 23, 2023

THardy98 commented May 24, 2023

jrote1 commented May 26, 2023

Error when viewing DB in UI #103501

Error when viewing DB in UI #103501

Comments

jrote1 commented May 17, 2023 • edited Loading

blathers-crl bot commented May 17, 2023

blathers-crl bot commented May 17, 2023

jrote1 commented May 17, 2023

florence-crl commented May 22, 2023

THardy98 commented May 22, 2023 • edited Loading

florence-crl commented May 23, 2023

jrote1 commented May 23, 2023

jrote1 commented May 23, 2023 • edited Loading

THardy98 commented May 23, 2023

jbowens commented May 23, 2023

jrote1 commented May 23, 2023

THardy98 commented May 24, 2023

jrote1 commented May 26, 2023

jrote1 commented May 17, 2023 •

edited

Loading

THardy98 commented May 22, 2023 •

edited

Loading

jrote1 commented May 23, 2023 •

edited

Loading