Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

*: add foreign key check/cascade runtime stats information in explain analyze result #39203

Merged
merged 55 commits into from
Nov 30, 2022
Merged
Show file tree
Hide file tree
Changes from 45 commits
Commits
Show all changes
55 commits
Select commit Hold shift + click to select a range
756ffcb
*: support explain with foreign key check/cascade
crazycs520 Nov 15, 2022
1a5a715
refine code
crazycs520 Nov 15, 2022
cee28ef
tiny fix
crazycs520 Nov 15, 2022
4c31703
add normalize plan with foreign key test
crazycs520 Nov 15, 2022
0a15163
tiny refine
crazycs520 Nov 15, 2022
a187b9e
fix ci
crazycs520 Nov 15, 2022
0776206
Merge branch 'master' into fk-explain
crazycs520 Nov 15, 2022
a64d6b5
fix ci
crazycs520 Nov 15, 2022
15bbefd
Merge branch 'fk-explain' of https://github.com/crazycs520/tidb into …
crazycs520 Nov 15, 2022
bcd8d5e
init
crazycs520 Nov 16, 2022
9d57e91
add test and fix bug
crazycs520 Nov 16, 2022
b0c2bcb
fix bug
crazycs520 Nov 16, 2022
1167519
fix test
crazycs520 Nov 16, 2022
76f02b2
move test to address comment
crazycs520 Nov 17, 2022
06f6151
address comment
crazycs520 Nov 18, 2022
41cc9e7
address comment
crazycs520 Nov 18, 2022
6d68562
Merge branch 'master' of https://github.com/pingcap/tidb into fk-explain
crazycs520 Nov 18, 2022
8d453e9
make ci stable
crazycs520 Nov 18, 2022
e5f9543
Merge branch 'master' of https://github.com/pingcap/tidb into fk-explain
crazycs520 Nov 18, 2022
4c0bf4c
remove bin
crazycs520 Nov 18, 2022
bea9776
fix bug
crazycs520 Nov 18, 2022
0ba0320
add more test
crazycs520 Nov 18, 2022
c45b10c
add more test
crazycs520 Nov 18, 2022
ffbee96
Merge branch 'master' into fk-explain
crazycs520 Nov 18, 2022
ffea8f1
Merge branch 'master' into fk-explain
hawkingrei Nov 18, 2022
e541b93
Merge branch 'master' into fk-explain
crazycs520 Nov 21, 2022
e4a16a2
Merge branch 'fk-explain' into fk-explain-analyze
crazycs520 Nov 21, 2022
af250e3
refine code
crazycs520 Nov 21, 2022
e5cacef
Merge branch 'master' of https://github.com/pingcap/tidb into fk-expl…
crazycs520 Nov 22, 2022
a41d3a9
Merge branch 'master' into fk-explain-analyze
crazycs520 Nov 22, 2022
4361260
fix ci
crazycs520 Nov 22, 2022
bde4ef4
Merge branch 'fk-explain-analyze' of https://github.com/crazycs520/ti…
crazycs520 Nov 22, 2022
e1bf6d5
add test case
crazycs520 Nov 23, 2022
2a4f693
address comment
crazycs520 Nov 24, 2022
7309ae7
address comment
crazycs520 Nov 25, 2022
a31b376
Merge branch 'master' of https://github.com/pingcap/tidb into fk-expl…
crazycs520 Nov 28, 2022
2f039fc
address comment
crazycs520 Nov 28, 2022
4ef09bb
Merge branch 'master' of https://github.com/pingcap/tidb into fk-expl…
crazycs520 Nov 28, 2022
2fb059e
Merge branch 'master' into fk-explain-analyze
crazycs520 Nov 29, 2022
c096672
Merge branch 'master' into fk-explain-analyze
crazycs520 Nov 29, 2022
9b4d2a7
fix bug and add test
crazycs520 Nov 29, 2022
3f4a0c5
add comment
crazycs520 Nov 29, 2022
e8ca848
Merge branch 'master' into fk-explain-analyze
crazycs520 Nov 29, 2022
ce31478
Merge branch 'master' into fk-explain-analyze
crazycs520 Nov 29, 2022
ad2a6ca
Merge branch 'master' into fk-explain-analyze
crazycs520 Nov 30, 2022
6964610
fix test
crazycs520 Nov 30, 2022
18b0be4
refine code
crazycs520 Nov 30, 2022
f1e4293
Merge branch 'master' into fk-explain-analyze
crazycs520 Nov 30, 2022
1d0423a
fix test
crazycs520 Nov 30, 2022
8f16430
Merge branch 'fk-explain-analyze' of https://github.com/crazycs520/ti…
crazycs520 Nov 30, 2022
a11d047
Merge branch 'master' into fk-explain-analyze
crazycs520 Nov 30, 2022
5a2aa03
Merge branch 'master' into fk-explain-analyze
crazycs520 Nov 30, 2022
30db64e
Merge branch 'master' into fk-explain-analyze
crazycs520 Nov 30, 2022
fd22612
Merge branch 'master' into fk-explain-analyze
crazycs520 Nov 30, 2022
03eb5bd
Merge branch 'master' into fk-explain-analyze
ti-chi-bot Nov 30, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
14 changes: 10 additions & 4 deletions executor/adapter.go
Original file line number Diff line number Diff line change
Expand Up @@ -592,10 +592,6 @@ func (a *ExecStmt) handleForeignKeyTrigger(ctx context.Context, e Executor, dept
if !ok {
return nil
}
a.Ctx.GetSessionVars().StmtCtx.InHandleForeignKeyTrigger = true
defer func() {
a.Ctx.GetSessionVars().StmtCtx.InHandleForeignKeyTrigger = false
}()
fkChecks := exec.GetFKChecks()
for _, fkCheck := range fkChecks {
err := fkCheck.doCheck(ctx)
Expand Down Expand Up @@ -631,6 +627,16 @@ func (a *ExecStmt) handleForeignKeyCascade(ctx context.Context, fkc *FKCascadeEx
if depth > maxForeignKeyCascadeDepth {
return ErrForeignKeyCascadeDepthExceeded.GenWithStackByArgs(maxForeignKeyCascadeDepth)
}
a.Ctx.GetSessionVars().StmtCtx.InHandleForeignKeyTrigger = true
defer func() {
a.Ctx.GetSessionVars().StmtCtx.InHandleForeignKeyTrigger = false
}()
if fkc.stats != nil {
start := time.Now()
defer func() {
fkc.stats.Total += time.Since(start)
}()
}
for {
e, err := fkc.buildExecutor(ctx)
if err != nil || e == nil {
Expand Down
344 changes: 333 additions & 11 deletions executor/fktest/foreign_key_test.go

Large diffs are not rendered by default.

144 changes: 139 additions & 5 deletions executor/foreign_key.go
Original file line number Diff line number Diff line change
Expand Up @@ -15,8 +15,11 @@
package executor

import (
"bytes"
"context"
"strconv"
"sync/atomic"
"time"

"github.com/pingcap/errors"
"github.com/pingcap/tidb/kv"
Expand All @@ -31,6 +34,7 @@ import (
"github.com/pingcap/tidb/types"
driver "github.com/pingcap/tidb/types/parser_driver"
"github.com/pingcap/tidb/util/codec"
"github.com/pingcap/tidb/util/execdetails"
"github.com/pingcap/tidb/util/set"
"github.com/tikv/client-go/v2/txnkv/txnsnapshot"
)
Expand Down Expand Up @@ -60,12 +64,16 @@ type FKCheckExec struct {

// FKCheckRuntimeStats contains the FKCheckExec runtime stats.
type FKCheckRuntimeStats struct {
Keys int
Total time.Duration
Check time.Duration
Lock time.Duration
Keys int
}

// FKCascadeExec uses to execute foreign key cascade behaviour.
type FKCascadeExec struct {
*fkValueHelper
plan *plannercore.FKCascade
b *executorBuilder
tp plannercore.FKCascadeType
referredFK *model.ReferredFKInfo
Expand All @@ -78,6 +86,8 @@ type FKCascadeExec struct {
fkValues [][]types.Datum
// new-value-key => UpdatedValuesCouple
fkUpdatedValuesMap map[string]*UpdatedValuesCouple

stats *FKCascadeRuntimeStats
}

// UpdatedValuesCouple contains the updated new row the old rows, exporting for test.
Expand All @@ -86,6 +96,12 @@ type UpdatedValuesCouple struct {
OldValuesList [][]types.Datum
}

// FKCascadeRuntimeStats contains the FKCascadeExec runtime stats.
type FKCascadeRuntimeStats struct {
Total time.Duration
Keys int
}

func buildTblID2FKCheckExecs(sctx sessionctx.Context, tblID2Table map[int64]table.Table, tblID2FKChecks map[int64][]*plannercore.FKCheck) (map[int64][]*FKCheckExec, error) {
fkChecksMap := make(map[int64][]*FKCheckExec)
for tid, tbl := range tblID2Table {
Expand Down Expand Up @@ -129,11 +145,16 @@ func buildFKCheckExec(sctx sessionctx.Context, tbl table.Table, fkCheck *planner
colsOffsets: colsOffsets,
fkValuesSet: set.NewStringSet(),
}
return &FKCheckExec{
e := &FKCheckExec{
ctx: sctx,
FKCheck: fkCheck,
fkValueHelper: helper,
}, nil
}
if sctx.GetSessionVars().StmtCtx.RuntimeStatsColl != nil {
e.stats = &FKCheckRuntimeStats{}
sctx.GetSessionVars().StmtCtx.RuntimeStatsColl.RegisterStats(fkCheck.ID(), e.stats)
}
return e, nil
}

func (fkc *FKCheckExec) insertRowNeedToCheck(sc *stmtctx.StatementContext, row []types.Datum) error {
Expand Down Expand Up @@ -175,6 +196,16 @@ func (fkc *FKCheckExec) addRowNeedToCheck(sc *stmtctx.StatementContext, row []ty
}

func (fkc *FKCheckExec) doCheck(ctx context.Context) error {
if len(fkc.toBeCheckedKeys) == 0 && len(fkc.toBeCheckedPrefixKeys) == 0 {
return nil
}
start := time.Now()
if fkc.stats != nil {
defer func() {
fkc.stats.Keys = len(fkc.toBeCheckedKeys) + len(fkc.toBeCheckedPrefixKeys)
fkc.stats.Total = time.Since(start)
}()
}
txn, err := fkc.ctx.Txn(false)
if err != nil {
return err
Expand All @@ -187,6 +218,9 @@ func (fkc *FKCheckExec) doCheck(ctx context.Context) error {
if err != nil {
return err
}
if fkc.stats != nil {
fkc.stats.Check = time.Since(start)
}
if len(fkc.toBeLockedKeys) == 0 {
return nil
}
Expand All @@ -202,6 +236,9 @@ func (fkc *FKCheckExec) doCheck(ctx context.Context) error {
// doLockKeys may set TxnCtx.ForUpdate to 1, then if the lock meet write conflict, TiDB can't retry for update.
// So reset TxnCtx.ForUpdate to 0 then can be retry if meet write conflict.
atomic.StoreUint32(&sessVars.TxnCtx.ForUpdate, forUpdate)
if fkc.stats != nil {
fkc.stats.Lock = time.Since(start) - fkc.stats.Check
}
return err
}

Expand Down Expand Up @@ -586,17 +623,23 @@ func (b *executorBuilder) buildFKCascadeExec(tbl table.Table, fkCascade *planner
colsOffsets: colsOffsets,
fkValuesSet: set.NewStringSet(),
}
return &FKCascadeExec{
e := &FKCascadeExec{
b: b,
fkValueHelper: helper,
plan: fkCascade,
tp: fkCascade.Tp,
referredFK: fkCascade.ReferredFK,
childTable: fkCascade.ChildTable.Meta(),
fk: fkCascade.FK,
fkCols: fkCascade.FKCols,
fkIdx: fkCascade.FKIdx,
fkUpdatedValuesMap: make(map[string]*UpdatedValuesCouple),
}, nil
}
if b.ctx.GetSessionVars().StmtCtx.RuntimeStatsColl != nil {
e.stats = &FKCascadeRuntimeStats{}
b.ctx.GetSessionVars().StmtCtx.RuntimeStatsColl.RegisterStats(fkCascade.ID(), e.stats)
}
crazycs520 marked this conversation as resolved.
Show resolved Hide resolved
return e, nil
}

func (fkc *FKCascadeExec) onDeleteRow(sc *stmtctx.StatementContext, row []types.Datum) error {
Expand Down Expand Up @@ -641,6 +684,7 @@ func (fkc *FKCascadeExec) buildExecutor(ctx context.Context) (Executor, error) {
if err != nil || p == nil {
return nil, err
}
fkc.plan.CascadePlans = append(fkc.plan.CascadePlans, p)
e := fkc.b.build(p)
return e, fkc.b.err
}
Expand Down Expand Up @@ -672,6 +716,9 @@ func (fkc *FKCascadeExec) buildFKCascadePlan(ctx context.Context) (plannercore.P
case model.ReferOptionCascade:
couple := fkc.fetchUpdatedValuesCouple()
if couple != nil && len(couple.NewValues) != 0 {
if fkc.stats != nil {
fkc.stats.Keys += len(couple.OldValuesList)
}
stmtNode = GenCascadeUpdateAST(fkc.referredFK.ChildSchema, fkc.childTable.Name, indexName, fkc.fkCols, couple)
}
case model.ReferOptionSetNull:
Expand Down Expand Up @@ -703,6 +750,9 @@ func (fkc *FKCascadeExec) fetchOnDeleteOrUpdateFKValues() [][]types.Datum {
fkValues = fkc.fkValues[:maxHandleFKValueInOneCascade]
fkc.fkValues = fkc.fkValues[maxHandleFKValueInOneCascade:]
}
if fkc.stats != nil {
fkc.stats.Keys += len(fkValues)
}
return fkValues
}

Expand Down Expand Up @@ -815,3 +865,87 @@ func genWhereConditionAstForMultiColumn(cols []*model.ColumnInfo, fkValues [][]t
List: valueList,
}
}

// String implements the RuntimeStats interface.
func (s *FKCheckRuntimeStats) String() string {
buf := bytes.NewBuffer(make([]byte, 0, 32))
buf.WriteString("total:")
buf.WriteString(execdetails.FormatDuration(s.Total))
if s.Check > 0 {
buf.WriteString(", check:")
buf.WriteString(execdetails.FormatDuration(s.Check))
}
if s.Lock > 0 {
buf.WriteString(", lock:")
buf.WriteString(execdetails.FormatDuration(s.Lock))
}
if s.Keys > 0 {
buf.WriteString(", foreign_keys:")
buf.WriteString(strconv.Itoa(s.Keys))
}
return buf.String()
}

// Clone implements the RuntimeStats interface.
func (s *FKCheckRuntimeStats) Clone() execdetails.RuntimeStats {
newRs := &FKCheckRuntimeStats{
Total: s.Total,
Check: s.Check,
Lock: s.Lock,
Keys: s.Keys,
}
return newRs
}

// Merge implements the RuntimeStats interface.
func (s *FKCheckRuntimeStats) Merge(other execdetails.RuntimeStats) {
tmp, ok := other.(*FKCheckRuntimeStats)
if !ok {
return
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

any log info?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No. The other same logic place also doesn't have a log.

}
s.Total += tmp.Total
s.Check += tmp.Check
s.Lock += tmp.Lock
s.Keys += tmp.Keys
}

// Tp implements the RuntimeStats interface.
func (s *FKCheckRuntimeStats) Tp() int {
return execdetails.TpFKCheckRuntimeStats
}

// String implements the RuntimeStats interface.
func (s *FKCascadeRuntimeStats) String() string {
buf := bytes.NewBuffer(make([]byte, 0, 32))
buf.WriteString("total:")
buf.WriteString(execdetails.FormatDuration(s.Total))
if s.Keys > 0 {
buf.WriteString(", foreign_keys:")
buf.WriteString(strconv.Itoa(s.Keys))
}
return buf.String()
}

// Clone implements the RuntimeStats interface.
func (s *FKCascadeRuntimeStats) Clone() execdetails.RuntimeStats {
newRs := &FKCascadeRuntimeStats{
Total: s.Total,
Keys: s.Keys,
}
return newRs
}

// Merge implements the RuntimeStats interface.
func (s *FKCascadeRuntimeStats) Merge(other execdetails.RuntimeStats) {
tmp, ok := other.(*FKCascadeRuntimeStats)
if !ok {
return
}
s.Total += tmp.Total
s.Keys += tmp.Keys
}

// Tp implements the RuntimeStats interface.
func (s *FKCascadeRuntimeStats) Tp() int {
return execdetails.TpFKCascadeRuntimeStats
}
15 changes: 2 additions & 13 deletions executor/insert_common.go
Original file line number Diff line number Diff line change
Expand Up @@ -1092,10 +1092,6 @@ func (e *InsertValues) collectRuntimeStatsEnabled() bool {
BasicRuntimeStats: e.runtimeStats,
SnapshotRuntimeStats: snapshotStats,
AllocatorRuntimeStats: autoid.NewAllocatorRuntimeStats(),
FKCheckStats: &FKCheckRuntimeStats{},
}
for _, fkc := range e.fkChecks {
fkc.stats = e.stats.FKCheckStats
}
}
return true
Expand Down Expand Up @@ -1299,7 +1295,6 @@ type InsertRuntimeStat struct {
CheckInsertTime time.Duration
Prefetch time.Duration
FKCheckTime time.Duration
FKCheckStats *FKCheckRuntimeStats
}

func (e *InsertRuntimeStat) String() string {
Expand Down Expand Up @@ -1341,10 +1336,8 @@ func (e *InsertRuntimeStat) String() string {
execdetails.FormatDuration(e.CheckInsertTime),
execdetails.FormatDuration(e.CheckInsertTime-e.Prefetch),
execdetails.FormatDuration(e.Prefetch)))
if e.FKCheckStats != nil && e.FKCheckStats.Keys > 0 {
buf.WriteString(fmt.Sprintf(", fk_check: %v, fk_num: %v",
execdetails.FormatDuration(e.FKCheckTime),
e.FKCheckStats.Keys))
if e.FKCheckTime > 0 {
buf.WriteString(fmt.Sprintf(", fk_check: %v", execdetails.FormatDuration(e.FKCheckTime)))
}
if e.SnapshotRuntimeStats != nil {
if rpc := e.SnapshotRuntimeStats.String(); len(rpc) > 0 {
Expand Down Expand Up @@ -1376,10 +1369,6 @@ func (e *InsertRuntimeStat) Clone() execdetails.RuntimeStats {
if e.AllocatorRuntimeStats != nil {
newRs.AllocatorRuntimeStats = e.AllocatorRuntimeStats.Clone()
}
if e.FKCheckStats != nil {
fkCheckStats := *e.FKCheckStats
newRs.FKCheckStats = &fkCheckStats
}
return newRs
}

Expand Down
3 changes: 1 addition & 2 deletions executor/insert_test.go
Original file line number Diff line number Diff line change
Expand Up @@ -1056,8 +1056,7 @@ func TestInsertRuntimeStat(t *testing.T) {
stats.Merge(stats.Clone())
require.Equal(t, "prepare: 6s, check_insert: {total_time: 4s, mem_insert_time: 2s, prefetch: 2s}", stats.String())
stats.FKCheckTime = time.Second
stats.FKCheckStats = &executor.FKCheckRuntimeStats{Keys: 20}
require.Equal(t, "prepare: 6s, check_insert: {total_time: 4s, mem_insert_time: 2s, prefetch: 2s, fk_check: 1s, fk_num: 20}", stats.String())
require.Equal(t, "prepare: 6s, check_insert: {total_time: 4s, mem_insert_time: 2s, prefetch: 2s, fk_check: 1s}", stats.String())
}

func TestDuplicateEntryMessage(t *testing.T) {
Expand Down
3 changes: 2 additions & 1 deletion planner/core/encode.go
Original file line number Diff line number Diff line change
Expand Up @@ -275,11 +275,12 @@ func NormalizeFlatPlan(flat *FlatPhysicalPlan) (normalized string, digest *parse
// assume an operator costs around 30 bytes, preallocate space for them
d.buf.Grow(30 * len(selectPlan))
depthOffset := len(flat.Main) - len(selectPlan)
loop1:
for _, op := range selectPlan {
switch op.Origin.(type) {
case *FKCheck, *FKCascade:
// Generate plan digest doesn't need to contain the foreign key check/cascade plan, so just break the loop.
continue
break loop1
crazycs520 marked this conversation as resolved.
Show resolved Hide resolved
}
taskTypeInfo := plancodec.EncodeTaskTypeForNormalize(op.IsRoot, op.StoreType)
p := op.Origin.(PhysicalPlan)
Expand Down
7 changes: 7 additions & 0 deletions planner/core/flat_plan.go
Original file line number Diff line number Diff line change
Expand Up @@ -398,6 +398,13 @@ func (f *FlatPhysicalPlan) flattenRecursively(p Plan, info *operatorCtx, target
target, childIdx = f.flattenRecursively(plan.TargetPlan, initInfo, target)
childIdxs = append(childIdxs, childIdx)
}
case *FKCascade:
for i, child := range plan.CascadePlans {
childCtx.label = Empty
childCtx.isLastChild = i == len(plan.CascadePlans)-1
target, childIdx = f.flattenRecursively(child, childCtx, target)
childIdxs = append(childIdxs, childIdx)
}
}
if flat != nil {
flat.ChildrenIdx = childIdxs
Expand Down
4 changes: 4 additions & 0 deletions planner/core/foreign_key.go
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,10 @@ type FKCascade struct {
FK *model.FKInfo
FKCols []*model.ColumnInfo
FKIdx *model.IndexInfo
// CascadePlans contains the child cascade plan.
// CascadePlans will be filled during execution, so only `explain analyze` statement result contains the cascade plan,
// `explain` statement result doesn't contain the cascade plan.
CascadePlans []Plan
crazycs520 marked this conversation as resolved.
Show resolved Hide resolved
crazycs520 marked this conversation as resolved.
Show resolved Hide resolved
}

// FKCascadeType indicates in which (delete/update) statements.
Expand Down
6 changes: 4 additions & 2 deletions planner/core/point_get_plan.go
Original file line number Diff line number Diff line change
Expand Up @@ -533,8 +533,10 @@ func TryFastPlan(ctx sessionctx.Context, node ast.Node) (p Plan) {
return nil
}

ctx.GetSessionVars().PlanID = 0
ctx.GetSessionVars().PlanColumnID = 0
if !ctx.GetSessionVars().StmtCtx.InHandleForeignKeyTrigger {
ctx.GetSessionVars().PlanID = 0
ctx.GetSessionVars().PlanColumnID = 0
}
switch x := node.(type) {
case *ast.SelectStmt:
defer func() {
Expand Down
Loading