kvclient: v22.2.6: attempting to append refresh spans after the tracked timestamp has moved forward #99093

cockroach-teamcity · 2023-03-21T06:17:08Z

This issue was autofiled by Sentry. It represents a crash or reported error on a live cluster with telemetry enabled.

Sentry link: https://cockroach-labs.sentry.io/issues/4023271542/?referrer=webhooks_plugin

Panic message:

txn_interceptor_span_refresher.go:687: log.Fatal: attempting to append refresh spans after the tracked timestamp has moved forward. batchTimestamp: 1679379412.106937991,0 refreshedTimestamp: 1679379412.122678360,2
(1) attached stack trace
-- stack trace:
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnSpanRefresher).importLeafFinalState
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go:687
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*TxnCoordSender).UpdateRootWithLeafFinalState
| github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_coord_sender.go:1259
| github.com/cockroachdb/cockroach/pkg/kv.(*Txn).UpdateRootWithLeafFinalState
| github.com/cockroachdb/cockroach/pkg/kv/txn.go:1345
| github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLReceiver).pushMeta
| github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1129
| github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLReceiver).Push
| github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1253
| github.com/cockroachdb/cockroach/pkg/sql/execinfra.Run
| github.com/cockroachdb/cockroach/pkg/sql/execinfra/base.go:189
| github.com/cockroachdb/cockroach/pkg/sql/execinfra.(*ProcessorBaseNoHelper).Run
| github.com/cockroachdb/cockroach/pkg/sql/execinfra/processorsbase.go:729
| github.com/cockroachdb/cockroach/pkg/sql/flowinfra.(*FlowBase).Run
| github.com/cockroachdb/cockroach/pkg/sql/flowinfra/flow.go:480
| github.com/cockroachdb/cockroach/pkg/sql/colflow.(*vectorizedFlow).Run
| github.com/cockroachdb/cockroach/pkg/sql/colflow/vectorized_flow.go:307
| github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).Run
| github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:731
| github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRun
| github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1698
| github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRunAll
| github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1421
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithDistSQLEngine
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1587
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).dispatchToExecutionEngine
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1201
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmtInOpenState
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:696
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt.func1
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:131
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithProfiling
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:2428
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:130
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execPortal
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:220
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd.func2
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:2050
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:2052
| github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1898
| github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn
| github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:857
| github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).processCommandsAsync.func1
| github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:728
| runtime.goexit
| GOROOT/src/runtime/asm_amd64.s:1594
Wraps: (2) secondary error attachment
| attempting to append refresh spans after the tracked timestamp has moved forward. batchTimestamp: 1679379412.106937991,0 refreshedTimestamp: 1679379412.122678360,2
| (1) assertion failure
| Wraps: (2) attached stack trace
| -- stack trace:
| | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnSpanRefresher).assertRefreshSpansAtInvalidTimestamp
| | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go:646
| | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*txnSpanRefresher).importLeafFinalState
| | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go:686
| | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord.(*TxnCoordSender).UpdateRootWithLeafFinalState
| | github.com/cockroachdb/cockroach/pkg/kv/kvclient/kvcoord/txn_coord_sender.go:1259
| | github.com/cockroachdb/cockroach/pkg/kv.(*Txn).UpdateRootWithLeafFinalState
| | github.com/cockroachdb/cockroach/pkg/kv/txn.go:1345
| | github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLReceiver).pushMeta
| | github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1129
| | github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLReceiver).Push
| | github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1253
| | github.com/cockroachdb/cockroach/pkg/sql/execinfra.Run
| | github.com/cockroachdb/cockroach/pkg/sql/execinfra/base.go:189
| | github.com/cockroachdb/cockroach/pkg/sql/execinfra.(*ProcessorBaseNoHelper).Run
| | github.com/cockroachdb/cockroach/pkg/sql/execinfra/processorsbase.go:729
| | github.com/cockroachdb/cockroach/pkg/sql/flowinfra.(*FlowBase).Run
| | github.com/cockroachdb/cockroach/pkg/sql/flowinfra/flow.go:480
| | github.com/cockroachdb/cockroach/pkg/sql/colflow.(*vectorizedFlow).Run
| | github.com/cockroachdb/cockroach/pkg/sql/colflow/vectorized_flow.go:307
| | github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).Run
| | github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:731
| | github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRun
| | github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1698
| | github.com/cockroachdb/cockroach/pkg/sql.(*DistSQLPlanner).PlanAndRunAll
| | github.com/cockroachdb/cockroach/pkg/sql/distsql_running.go:1421
| | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithDistSQLEngine
| | github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1587
| | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).dispatchToExecutionEngine
| | github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:1201
| | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmtInOpenState
| | github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:696
| | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt.func1
| | github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:131
| | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execWithProfiling
| | github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:2428
| | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execStmt
| | github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:130
| | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execPortal
| | github.com/cockroachdb/cockroach/pkg/sql/conn_executor_exec.go:220
| | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd.func2
| | github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:2050
| | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).execCmd
| | github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:2052
| | github.com/cockroachdb/cockroach/pkg/sql.(*connExecutor).run
| | github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:1898
| | github.com/cockroachdb/cockroach/pkg/sql.(*Server).ServeConn
| | github.com/cockroachdb/cockroach/pkg/sql/conn_executor.go:857
| | github.com/cockroachdb/cockroach/pkg/sql/pgwire.(*conn).processCommandsAsync.func1
| | github.com/cockroachdb/cockroach/pkg/sql/pgwire/conn.go:728
| | runtim...

Stacktrace (expand for inline code snippets):

cockroach/pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go

Lines 686 to 688 in d5a076f

    
           if err := sr.assertRefreshSpansAtInvalidTimestamp(tfs.Txn.ReadTimestamp); err != nil { 
        
           	log.Fatalf(ctx, "%s", err) 
        
           }

in pkg/kv/kvclient/kvcoord.(*txnSpanRefresher).importLeafFinalState

cockroach/pkg/kv/kvclient/kvcoord/txn_coord_sender.go

Lines 1258 to 1260 in d5a076f

    
           for _, reqInt := range tc.interceptorStack { 
        
           	reqInt.importLeafFinalState(ctx, tfs) 
        
           }

in pkg/kv/kvclient/kvcoord.(*TxnCoordSender).UpdateRootWithLeafFinalState

cockroach/pkg/kv/txn.go

Lines 1344 to 1346 in d5a076f

    
           defer txn.mu.Unlock() 
        
           txn.mu.sender.UpdateRootWithLeafFinalState(ctx, tfs) 
        
           return nil

in pkg/kv.(*Txn).UpdateRootWithLeafFinalState

cockroach/pkg/sql/distsql_running.go

Lines 1128 to 1130 in d5a076f

    
           if r.txn.ID() == meta.LeafTxnFinalState.Txn.ID { 
        
           	if err := r.txn.UpdateRootWithLeafFinalState(r.ctx, meta.LeafTxnFinalState); err != nil { 
        
           		r.SetError(err)

in pkg/sql.(*DistSQLReceiver).pushMeta

cockroach/pkg/sql/distsql_running.go

Lines 1252 to 1254 in d5a076f

    
           if meta != nil { 
        
           	return r.pushMeta(meta) 
        
           }

in pkg/sql.(*DistSQLReceiver).Push

cockroach/pkg/sql/execinfra/base.go

Lines 188 to 190 in d5a076f

    
           if row != nil || meta != nil { 
        
           	switch dst.Push(row, meta) { 
        
           	case NeedMoreRows:

in pkg/sql/execinfra.Run

cockroach/pkg/sql/execinfra/processorsbase.go

Lines 728 to 730 in d5a076f

    
           	pb.self.Start(ctx) 
        
           	Run(pb.ctx, pb.self, pb.Output) 
        
           }

in pkg/sql/execinfra.(*ProcessorBaseNoHelper).Run

cockroach/pkg/sql/flowinfra/flow.go

Lines 479 to 481 in d5a076f

    
           	log.VEventf(ctx, 1, "running %T in the flow's goroutine", headProc) 
        
           	headProc.Run(ctx) 
        
           }

in pkg/sql/flowinfra.(*FlowBase).Run

cockroach/pkg/sql/colflow/vectorized_flow.go

Lines 306 to 308 in d5a076f

    
           // as the root, so we run this flow with the default implementation. 
        
           f.FlowBase.Run(ctx, doneFn) 
        
           return

in pkg/sql/colflow.(*vectorizedFlow).Run

cockroach/pkg/sql/distsql_running.go

Lines 730 to 732 in d5a076f

    
           // TODO(radu): this should go through the flow scheduler. 
        
           flow.Run(ctx, func() {})

in pkg/sql.(*DistSQLPlanner).Run

cockroach/pkg/sql/distsql_running.go

Lines 1697 to 1699 in d5a076f

    
           recv.expectedRowsRead = int64(physPlan.TotalEstimatedScannedRows) 
        
           runCleanup := dsp.Run(ctx, planCtx, txn, physPlan, recv, evalCtx, nil /* finishedSetupFn */) 
        
           return func() {

in pkg/sql.(*DistSQLPlanner).PlanAndRun

cockroach/pkg/sql/distsql_running.go

Lines 1420 to 1422 in d5a076f

    
           // the planner whether or not to plan remote table readers. 
        
           cleanup := dsp.PlanAndRun( 
        
           	ctx, evalCtx, planCtx, planner.txn, planner.curPlan.main, recv,

in pkg/sql.(*DistSQLPlanner).PlanAndRunAll

cockroach/pkg/sql/conn_executor_exec.go

Lines 1586 to 1588 in d5a076f

    
           } 
        
           err := ex.server.cfg.DistSQLPlanner.PlanAndRunAll(ctx, evalCtx, planCtx, planner, recv, evalCtxFactory) 
        
           return *recv.stats, err

in pkg/sql.(*connExecutor).execWithDistSQLEngine

cockroach/pkg/sql/conn_executor_exec.go

Lines 1200 to 1202 in d5a076f

    
           ex.sessionTracing.TraceExecStart(ctx, "distributed") 
        
           stats, err = ex.execWithDistSQLEngine( 
        
           	ctx, planner, stmt.AST.StatementReturnType(), res, distribute, progAtomic,

in pkg/sql.(*connExecutor).dispatchToExecutionEngine

cockroach/pkg/sql/conn_executor_exec.go

Lines 695 to 697 in d5a076f

    
           if err := ex.dispatchToExecutionEngine(stmtCtx, p, res); err != nil { 
        
           	stmtThresholdSpan.Finish()

in pkg/sql.(*connExecutor).execStmtInOpenState

cockroach/pkg/sql/conn_executor_exec.go

Lines 130 to 132 in d5a076f

    
           err = ex.execWithProfiling(ctx, ast, prepared, func(ctx context.Context) error { 
        
           	ev, payload, err = ex.execStmtInOpenState(ctx, parserStmt, prepared, pinfo, res, canAutoCommit) 
        
           	return err

in pkg/sql.(*connExecutor).execStmt.func1

cockroach/pkg/sql/conn_executor_exec.go

Lines 2427 to 2429 in d5a076f

    
           } else { 
        
           	err = op(ctx) 
        
           }

in pkg/sql.(*connExecutor).execWithProfiling

cockroach/pkg/sql/conn_executor_exec.go

Lines 129 to 131 in d5a076f

    
           case stateOpen: 
        
           	err = ex.execWithProfiling(ctx, ast, prepared, func(ctx context.Context) error { 
        
           		ev, payload, err = ex.execStmtInOpenState(ctx, parserStmt, prepared, pinfo, res, canAutoCommit)

in pkg/sql.(*connExecutor).execStmt

cockroach/pkg/sql/conn_executor_exec.go

Lines 219 to 221 in d5a076f

    
           } 
        
           ev, payload, err = ex.execStmt(ctx, portal.Stmt.Statement, portal.Stmt, pinfo, stmtRes, canAutoCommit) 
        
           // Portal suspension is supported via a "side" state machine

in pkg/sql.(*connExecutor).execPortal

cockroach/pkg/sql/conn_executor.go

Lines 2049 to 2051 in d5a076f

    
           canAutoCommit := ex.implicitTxn() && tcmd.FollowedBySync 
        
           ev, payload, err = ex.execPortal(ctx, portal, portalName, stmtRes, pinfo, canAutoCommit) 
        
           return err

in pkg/sql.(*connExecutor).execCmd.func2

cockroach/pkg/sql/conn_executor.go

Lines 2051 to 2053 in d5a076f

    
           	return err 
        
           }() 
        
           // Note: we write to ex.statsCollector.phaseTimes, instead of ex.phaseTimes,

in pkg/sql.(*connExecutor).execCmd

cockroach/pkg/sql/conn_executor.go

Lines 1897 to 1899 in d5a076f

    
           var err error 
        
           if err = ex.execCmd(); err != nil { 
        
           	if errors.IsAny(err, io.EOF, errDrainingComplete) {

in pkg/sql.(*connExecutor).run

cockroach/pkg/sql/conn_executor.go

Lines 856 to 858 in d5a076f

    
           	}(ctx, h) 
        
           	return h.ex.run(ctx, s.pool, reserved, cancel) 
        
           }

in pkg/sql.(*Server).ServeConn

cockroach/pkg/sql/pgwire/conn.go

Lines 727 to 729 in d5a076f

    
           	reservedOwned = false // We're about to pass ownership away. 
        
           	retErr = sqlServer.ServeConn(ctx, connHandler, reserved, cancelConn) 
        
           }()

in pkg/sql/pgwire.(*conn).processCommandsAsync.func1
GOROOT/src/runtime/asm_amd64.s#L1593-L1595 in runtime.goexit

pkg/kv/kvclient/kvcoord/txn_interceptor_span_refresher.go in pkg/kv/kvclient/kvcoord.(*txnSpanRefresher).importLeafFinalState at line 687
pkg/kv/kvclient/kvcoord/txn_coord_sender.go in pkg/kv/kvclient/kvcoord.(*TxnCoordSender).UpdateRootWithLeafFinalState at line 1259
pkg/kv/txn.go in pkg/kv.(*Txn).UpdateRootWithLeafFinalState at line 1345
pkg/sql/distsql_running.go in pkg/sql.(*DistSQLReceiver).pushMeta at line 1129
pkg/sql/distsql_running.go in pkg/sql.(*DistSQLReceiver).Push at line 1253
pkg/sql/execinfra/base.go in pkg/sql/execinfra.Run at line 189
pkg/sql/execinfra/processorsbase.go in pkg/sql/execinfra.(*ProcessorBaseNoHelper).Run at line 729
pkg/sql/flowinfra/flow.go in pkg/sql/flowinfra.(*FlowBase).Run at line 480
pkg/sql/colflow/vectorized_flow.go in pkg/sql/colflow.(*vectorizedFlow).Run at line 307
pkg/sql/distsql_running.go in pkg/sql.(*DistSQLPlanner).Run at line 731
pkg/sql/distsql_running.go in pkg/sql.(*DistSQLPlanner).PlanAndRun at line 1698
pkg/sql/distsql_running.go in pkg/sql.(*DistSQLPlanner).PlanAndRunAll at line 1421
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execWithDistSQLEngine at line 1587
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).dispatchToExecutionEngine at line 1201
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execStmtInOpenState at line 696
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execStmt.func1 at line 131
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execWithProfiling at line 2428
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execStmt at line 130
pkg/sql/conn_executor_exec.go in pkg/sql.(*connExecutor).execPortal at line 220
pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).execCmd.func2 at line 2050
pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).execCmd at line 2052
pkg/sql/conn_executor.go in pkg/sql.(*connExecutor).run at line 1898
pkg/sql/conn_executor.go in pkg/sql.(*Server).ServeConn at line 857
pkg/sql/pgwire/conn.go in pkg/sql/pgwire.(*conn).processCommandsAsync.func1 at line 728
GOROOT/src/runtime/asm_amd64.s in runtime.goexit at line 1594

Tag	Value
Cockroach Release	`v22.2.6`
Cockroach SHA:	`d5a076f`
Platform	linux amd64
Distribution	CCL
Environment	v22.2.6
Command	server
Go Version	``
# of CPUs
# of Goroutines

Jira issue: CRDB-25797

The text was updated successfully, but these errors were encountered:

cucaroach · 2023-03-22T17:43:47Z

GIST:

[email protected]:26257/defaultdb> select crdb_internal.decode_plan_gist('AgGK////nxkAABoAAAADBwoUADACBgEFDgUCBgI=');                                           
          crdb_internal.decode_plan_gist
---------------------------------------------------
  • lookup join
  │ table: comments@primary
  │ equality: (_, _, _) = (type,object_id,sub_id)
  │ equality cols are key
  │
  └── • render
      │
      └── • filter
          │
          └── • virtual table
                table: @primary
(11 rows)

Time: 4ms total (execution 3ms / network 0ms)

cockroach-teamcity added C-bug Code not up to spec/doc, specs & docs deemed correct. Solution expected to change code/behavior. O-sentry Originated from an in-the-wild panic report. labels Mar 21, 2023

blathers-crl bot added the T-sql-queries SQL Queries Team label Mar 22, 2023

yuzefovich self-assigned this Mar 23, 2023

yuzefovich mentioned this issue Mar 23, 2023

sql: crash when reading from system.comments after having manually inserted a row #99316

Closed

craig bot closed this as completed in eb45930 Mar 23, 2023

craig bot closed this as completed in #99325 Mar 23, 2023

This was referenced Mar 23, 2023

release-23.1: sql: disable the streamer for queries which might use internal executor #99443

Merged

release-22.2: sql: disable the streamer for queries which might use internal executor #99445

Merged

yuzefovich mentioned this issue Jun 10, 2023

sentry: txn_interceptor_span_refresher.go:650: attempting to append refresh spans after the tracked timestamp has moved forward. batchTimestamp: 1686192818.725925502,0 refreshedTimestamp: 1686192818.729068450... #104572

Closed

mgartner added this to SQL Queries Jul 24, 2023

mgartner moved this to Done in SQL Queries Jul 24, 2023

yuzefovich mentioned this issue Sep 7, 2023

kv,jobs: txn_interceptor_span_refresher error when performing job operations #99125

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

kvclient: v22.2.6: attempting to append refresh spans after the tracked timestamp has moved forward #99093

kvclient: v22.2.6: attempting to append refresh spans after the tracked timestamp has moved forward #99093

cockroach-teamcity commented Mar 21, 2023 •

edited by cockroach-jira-scripts

Loading

cucaroach commented Mar 22, 2023 •

edited by yuzefovich

Loading

kvclient: v22.2.6: attempting to append refresh spans after the tracked timestamp has moved forward #99093

kvclient: v22.2.6: attempting to append refresh spans after the tracked timestamp has moved forward #99093

Comments

cockroach-teamcity commented Mar 21, 2023 • edited by cockroach-jira-scripts Loading

cucaroach commented Mar 22, 2023 • edited by yuzefovich Loading

cockroach-teamcity commented Mar 21, 2023 •

edited by cockroach-jira-scripts

Loading

cucaroach commented Mar 22, 2023 •

edited by yuzefovich

Loading