Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Show the maximum available CPU/Memory on TiFlash Grafana panel #3852

Closed
wants to merge 5,289 commits into from
Closed
Changes from 1 commit
Commits
Show all changes
5289 commits
Select commit Hold shift + click to select a range
0ec758c
Use CRC64 as checksum algo for LogFile (#4261)
JaySon-Huang Mar 15, 2022
14185f7
Clean deprecated directories and internal links (#4278)
solotzg Mar 15, 2022
0e40b9e
fix bug that ExchangeReceiver is not cancelled if exception happens i…
windtalker Mar 15, 2022
0daf533
Support dayofweek and dayofyear pushdown to tiflash (#4222)
gengliqi Mar 16, 2022
146a99a
refine tiflash shutdown logic (#4291)
bestwoody Mar 16, 2022
6141520
WALStore support multi path (#4085)
jiaqizho Mar 16, 2022
b9379c4
Limit build release parallelism (#4301)
solotzg Mar 16, 2022
502dd4a
add `HashJoinProbeBlockInputStream` (#4246)
SeaRise Mar 16, 2022
6ffc427
Add CI license check flow config file (#4294)
yibin87 Mar 16, 2022
f030da8
make mpp_fail test stable (#4300)
windtalker Mar 16, 2022
d385f0a
README.md: Rename tics to tiflash. (#4296)
JinheLin Mar 16, 2022
946b72e
fix potential asan error for string tests (#4279)
SchrodingerZhu Mar 16, 2022
de7bc99
MPP: ExchangeReceiver supports async gRPC (#3715)
fuzhe1989 Mar 16, 2022
0948c8a
Replace old CK license with standard Apache 2.0 License (#4306)
yibin87 Mar 16, 2022
59d9ed0
Fix crash when read ColumnFileTiny in MemTableSet (#4297)
lidezhu Mar 16, 2022
8723b25
cleanup benchmark (#4233)
guo-shaoge Mar 16, 2022
c0c63f3
Min tso metrics (#4260)
fzhedu Mar 16, 2022
a53783d
fix bug of handling zero date in dayofweek and dayofyear (#4314)
windtalker Mar 17, 2022
19c3e48
use the right way to wait tiflash replica ready in least_greatest.tes…
windtalker Mar 17, 2022
4e52ac8
print table_id for table which is applying snapshot (#4310)
lidezhu Mar 17, 2022
aacd920
Scheduler: release threads by tasks (#4307)
fzhedu Mar 17, 2022
fe5a3d7
Fix bug that TiFlash will crash when enable both async_grpc_client an…
fuzhe1989 Mar 17, 2022
dfc6dcd
MPP: add metrics for ExchangeReceiver (#4327)
fuzhe1989 Mar 17, 2022
75b2bf2
test: check "," when execute wait-table (#4319)
fuzhe1989 Mar 17, 2022
a78531c
feat: improve symbolization (#4325)
SchrodingerZhu Mar 17, 2022
ebf7396
Fix randomly build failed because mpp.pb.h not found (#4331)
guo-shaoge Mar 17, 2022
2686d08
fix(ut): use valid UTF8 sequence instead (#4315)
SchrodingerZhu Mar 17, 2022
a806456
Tracing: Add tracing_id to PageStorage snapshot (#4330)
JaySon-Huang Mar 18, 2022
00ac988
Fix problems introduced by symbolization (#4334)
SchrodingerZhu Mar 18, 2022
06dff09
fix: remove basename (#4343)
SchrodingerZhu Mar 18, 2022
1a7599a
share PageStorage V3 inside tiflash instance (#4220)
lidezhu Mar 19, 2022
c8faa90
*: Introduce Logger for replacing LogWithPrefix (#4354)
fuzhe1989 Mar 21, 2022
ca68f33
fix cannot find path error when pass relative data path to dt_workloa…
lidezhu Mar 21, 2022
6f5ad57
Fix `max_caps` may not been update after GC (#4323)
jiaqizho Mar 21, 2022
7d77e22
Grafana: correct figure name (#4341)
fzhedu Mar 21, 2022
69fdebb
update client-c to fix random keepalive timeout issue (#4357)
windtalker Mar 21, 2022
a0f0262
Improve LogWithPrefix of mpp (#4209)
SeaRise Mar 21, 2022
3269732
Fix restore problems in PageStorage. (#4350)
jiaqizho Mar 21, 2022
63fadc8
refine `appendFinalProjectFor(Non)RootQueryBlock` in analyzer (#4052)
SeaRise Mar 21, 2022
5053677
Fix ThreadSanitizer: `std::localtime()` is not thread-safe. (#4344)
Lloyd-Pottiger Mar 21, 2022
25c912d
tune default setting: async_pollers_per_cq (#4359)
bestwoody Mar 21, 2022
315c606
feat: more symbolization support (#4361)
SchrodingerZhu Mar 21, 2022
4d278c2
fix: elim sign comparison in basedaemon (#4368)
SchrodingerZhu Mar 22, 2022
78c7fa1
Fix special NewDecimal type passed from TiDB (#4329)
yibin87 Mar 22, 2022
ddc825f
Add metric to record end to end time to pre-handle snapshot (#3978)
CalvinNeo Mar 22, 2022
e22fc89
Fix wrong result of cast(float as decimal) when overflow happens (#4380)
guo-shaoge Mar 23, 2022
4ad93ae
*: replace LogWithPrefix by Logger (#4356)
fuzhe1989 Mar 23, 2022
4429765
Storages/DeltaMerge/tools: Refine statistical data. (#4363)
JinheLin Mar 23, 2022
5fc8794
make DeltaValueSpace::info thread safe (#4393)
lidezhu Mar 23, 2022
f887620
Remove useless file scanning when using v3 (#4395)
JaySon-Huang Mar 23, 2022
02d567d
Scheduler no throw in destruction and avoid updated min-tso query han…
fzhedu Mar 23, 2022
681447c
Fix lint and remove unused member in `FunctionsString` (#4382)
JaySon-Huang Mar 23, 2022
1a9a68b
Improve log in `dbms/src/client`, `dbms/src/common`, and `dbms/src/da…
Lloyd-Pottiger Mar 23, 2022
8264f65
Clean useless code in `dbms/src/Flash/DiagnosticsService.cpp` (#4401)
solotzg Mar 23, 2022
4a6130d
replace `LOG_XXX` with `LOG_FMT_XXX` in `dbms/src/DataStreams`, `dbms…
Lloyd-Pottiger Mar 24, 2022
6bc103d
Logger: Refactor dmfile reader and add tracing_logger (#4379)
JaySon-Huang Mar 24, 2022
afda84f
`Poco_INCLUDE_DIRS` should not be a cached entry (#4384)
gengliqi Mar 24, 2022
7c48f6d
add more checks for background segment split (#4406)
lidezhu Mar 24, 2022
2944ca0
Remove duplicated streams code. (#4413)
ywqzzy Mar 24, 2022
c4b2384
*: add macro RUNTIME_CHECK and RUNTIME_ASSERT (#4398)
fuzhe1989 Mar 24, 2022
da21480
Fix data race in gtest_rate_limiter.cpp (#4421)
JinheLin Mar 24, 2022
f236760
Add some log to make it easier to observe data skew (#4404)
windtalker Mar 25, 2022
53d467b
chores(cmake): bump version to 3.21.0 (#4162)
SchrodingerZhu Mar 25, 2022
be63f8e
Improve storages log (#4364)
Lloyd-Pottiger Mar 25, 2022
4689bd0
replace `LOG_XXX` with `LOG_FMT_XXX` in `dbms/src/Dictionaries`, `dbm…
Lloyd-Pottiger Mar 25, 2022
9e86037
update TiFlash Proxy to fix disk mount point name in dashboard (#4431)
CalvinNeo Mar 25, 2022
55f6dc2
fix a bug that ExchangeReceiver can't be canceled (#4441)
bestwoody Mar 25, 2022
ea34978
Exception msg (#4443)
fzhedu Mar 25, 2022
dd7af3a
Fix license checker (#4446)
solotzg Mar 26, 2022
932c2a4
Fix hardlink may cause dead loop, also disable config gc_hardlink_rat…
JaySon-Huang Mar 28, 2022
cb20f96
Add env for using the bundled protoc for building tiflash-proxy (#4461)
JaySon-Huang Mar 28, 2022
b67c7ab
Fix data race of DeltaIndexManager. (#4459)
JinheLin Mar 28, 2022
1b3b651
disable stacktrace related test for sanitized UT (#4463)
SchrodingerZhu Mar 28, 2022
f30ac85
override `IProfilingBlockInputStream.readSuffixImpl` instead of `IPro…
SeaRise Mar 28, 2022
ce227c1
*: Fix saving reference to temporary object (#4477)
JaySon-Huang Mar 29, 2022
0c01acb
replace LOG_TRACE with LOG_FMT_TRACE. (#4375)
ywqzzy Mar 29, 2022
9ace6aa
doc: Use collapsed blocks to make it more friendly for readers (#4370)
breezewish Mar 29, 2022
8c0f4df
Replace `LOG_XXX` with `LOG_FMT_XXX` (#4480)
Lloyd-Pottiger Mar 29, 2022
6293629
BlobStore support multipath. (#3938)
jiaqizho Mar 29, 2022
10c9e97
update tiflash proxy version to adopt bugfix from TiKV (#4453)
CalvinNeo Mar 30, 2022
9f8f0a5
Logger: Remove stringstream in total (#4484)
JaySon-Huang Mar 30, 2022
8947637
PageStorage V3 GC add metrics. (#4397)
jiaqizho Mar 30, 2022
4dbcbef
stable async setting (#4485)
bestwoody Mar 30, 2022
eb75500
Remove default arguments in DB::IConnectionPool and its derived class…
Lloyd-Pottiger Mar 30, 2022
d754df0
fix empty segment cannot merge after gc and avoid write index data fo…
lidezhu Mar 30, 2022
1ed83f1
*: refine RUNTIME_ASSERT (#4504)
fuzhe1989 Mar 30, 2022
e2a9949
*: use MPMCQueue in UnionBlockInputStream and SharedQueryBlockInputSt…
fuzhe1989 Mar 30, 2022
fb0ccb9
accelerate dumpTree by cache tree_id inside BlockInputStream and repl…
SeaRise Mar 31, 2022
87e360b
Add a log prefix into BlobStore and WALStore (#4492)
jiaqizho Mar 31, 2022
47e6483
*: remove redundant `<std::mutex>` in local variable definitions (#4510)
fuzhe1989 Mar 31, 2022
c38b7df
*: do not wrap functions into tasks in `DynamicThreadManager::schedul…
fuzhe1989 Mar 31, 2022
25ad728
Support drop table for PageStorage V3 (#4428)
lidezhu Mar 31, 2022
019ca96
Remove default arguments of virtual function in DB::PageStorage and i…
Lloyd-Pottiger Mar 31, 2022
bece4b0
Update README.md to collapse parts for internal developers (#4533)
solotzg Mar 31, 2022
cbf0334
Remove useless member of `CreatingSetsBlockInputStream` (#4535)
fuzhe1989 Mar 31, 2022
0cfed74
submit local_delta memory usage when need to pass memory_tracker betw…
windtalker Mar 31, 2022
e1132f1
remove useless clickhouse applications (#4539)
fuzhe1989 Mar 31, 2022
06f1e19
remove embeded compiler (#4542)
fuzhe1989 Mar 31, 2022
6f7c69c
remove useless RemoteBlockInput/OutputStream (#4547)
fuzhe1989 Mar 31, 2022
f06f9f7
doc: remove internal stuffs (#4543)
SchrodingerZhu Mar 31, 2022
6c1584d
Fix build failure introduced by #4542 (#4550)
fuzhe1989 Mar 31, 2022
7959675
Celebrate open-source (#4551)
solotzg Apr 1, 2022
72f1dca
doc: add apt.llvm.org in README (#4555)
Lloyd-Pottiger Apr 1, 2022
75dc910
PageStorage: Fix bug for reporting BlobFile disk usage (#4540)
JaySon-Huang Apr 1, 2022
3925075
PageStorage: Fix data race in BlobStore (#4548)
JaySon-Huang Apr 1, 2022
3c7898d
Fix typo in README/build (#4561)
qian0817 Apr 1, 2022
aa500f4
Unify the url of submodule to https url (#4566)
Lloyd-Pottiger Apr 2, 2022
d230df6
Remove invalid blob file after BlobStore restored. (#4475)
jiaqizho Apr 2, 2022
fca58ac
Remove redundant use of marco __PRETTY_FUNCTION__ (#4486)
ywqzzy Apr 2, 2022
0a4e413
Remove useless TODO in `dbms/src/Storages/Page/V3/tests/gtest_wal_log…
Lloyd-Pottiger Apr 2, 2022
3355702
Agg: replace the usage of `ThreadPool` by `ThreadPoolManager`. (#4582)
fuzhe1989 Apr 4, 2022
2e6cac4
Tiflash crash when scanning partition table with `time` column (#4563)
SeaRise Apr 6, 2022
6bc46ae
doc: add instructions for runing unit tests (#4576)
Lloyd-Pottiger Apr 6, 2022
6856628
Add a stale snapshots check, also add some logs (#4537)
jiaqizho Apr 6, 2022
2220906
Tracing: Add read_tso as tracing id to PageStorage snapshot (#4288)
JaySon-Huang Apr 8, 2022
5580f2c
Inhibit warning messages in tiflash-proxy (#4613)
solotzg Apr 11, 2022
3fde58c
Split AstToExecutor from dbgFuncCoprocessor. (#4610)
ywqzzy Apr 12, 2022
373b8d6
Update client c (#4625)
Lloyd-Pottiger Apr 12, 2022
e8ac8e9
Fix can not find right path when blobstore restore (#4608)
jiaqizho Apr 12, 2022
7645f7d
*:Fix MppInfo saving reference to temporary object. (#4638)
ywqzzy Apr 13, 2022
fdc25a7
Fix logger initialization of DTWorkload. (#4637)
JinheLin Apr 13, 2022
d30bf6c
move `output_field_types` and `output_offsets` to `DAGContext` (#4626)
SeaRise Apr 13, 2022
627a43e
Fix potential data race in DynamicThreadPool (#4648)
fuzhe1989 Apr 14, 2022
811a020
Fix create table error (#4630)
Lloyd-Pottiger Apr 14, 2022
3867638
Add a pagestorage v3 controller. (#4320)
jiaqizho Apr 14, 2022
aceeef1
update client-c to disable grpc client keepalive (#4655)
windtalker Apr 14, 2022
9f7b82f
fix typo (#4671)
nautaa Apr 15, 2022
45ce1f4
Add a fail point that can hold PS snapshot for five minute. (#4620)
jiaqizho Apr 15, 2022
b01b23c
Interpreter: Hand write tipb::Executor. (#4632)
ywqzzy Apr 15, 2022
a49a90d
refine `SubqueryForSet` (#4623)
SeaRise Apr 18, 2022
0e14757
Add gtests for MPPTunnel (#4553)
yibin87 Apr 18, 2022
454155b
Fix background_pool_size not take effect and BackgroundProcessingPool…
JinheLin Apr 18, 2022
6a284fb
fix: fix build issue `‘maybe_unused’ attribute ignored` on centos ser…
LittleFall Apr 18, 2022
d431a4f
*: fix tableScan incorrect trace. (#4699)
ywqzzy Apr 19, 2022
4a5266c
Make performance of TPCH q15 stable (#4570)
windtalker Apr 19, 2022
1fcd108
*: fix gettid compile error (#4704)
ywqzzy Apr 19, 2022
22cdd08
clear old range before apply snapshot (#4668)
lidezhu Apr 20, 2022
8797793
update client-c to latest version (#4681)
gengliqi Apr 20, 2022
9e3637e
reuse old index file format for empty dmfile and just ignore it's siz…
lidezhu Apr 20, 2022
fd970b3
Add some debug friendly utilities for dag request based test framewor…
yibin87 Apr 21, 2022
3f453e3
Add a background metrics for both v2 and v3 (#4593)
jiaqizho Apr 21, 2022
8bc4e71
Interpreter: Mock exchange executors. (#4706)
ywqzzy Apr 21, 2022
ce79cda
Fix pagestorage v3 ddl problem. (#4691)
jiaqizho Apr 21, 2022
3aab15a
support row_number, rank, dense_rank in tiflash (#4552)
mengxin9014 Apr 22, 2022
e3400ce
fix tiflash crash when setting join_concurrent_build = 0 (#4735)
SeaRise Apr 24, 2022
10f2f3b
Fxied some compile errors on mac in PS V3 (#4741)
flowbehappy Apr 25, 2022
cfb5cf6
Refine handle agg and exchange sender (#4721)
SeaRise Apr 25, 2022
2286551
update TiFlash proxy to raftstore-proxy-6.0 for up(down)grade test (#…
CalvinNeo Apr 25, 2022
f2f0244
ignore delmark when add minmax for pk column (#4746)
lidezhu Apr 25, 2022
3f21d54
Interpreter: Serialize executors (#4742)
ywqzzy Apr 26, 2022
6b382ba
fix: frame total size include header for indices (#4754)
SchrodingerZhu Apr 26, 2022
b4baa78
Add more debug info for gtest failure (#4769)
yibin87 Apr 27, 2022
c581632
Fix ingest dmfile won't rollback (#4785)
jiaqizho Apr 28, 2022
53c1473
fix: ut and more idx error (#4777)
SchrodingerZhu Apr 28, 2022
7a6f09c
refactor: Remove some non-const member methods in Context. (#4786)
junaire Apr 28, 2022
028d69d
doc: add sanitizer test related docs (#4750)
SchrodingerZhu Apr 28, 2022
0afd374
fix: Suppress the warning from boost to fix build failure. (#4796)
junaire Apr 28, 2022
f583ad9
fix some unsafe constructor and destructor (#4782)
windtalker Apr 28, 2022
ba5b999
Export PageStorage V3 config into global config (#4502)
jiaqizho Apr 29, 2022
2bb33df
Support weekofyear pushdown to tiflash (#4791)
gengliqi May 2, 2022
cf06903
Fix gtest mpp tunnel local write unstable issue (#4811)
yibin87 May 5, 2022
76a3d82
PageStorage APIs support throw_on_not_exist. (#4736)
jiaqizho May 5, 2022
2ece513
refactor: Remove unused FunctionIPv6NumToString from ClickHouse. (#4797)
junaire May 5, 2022
77583f2
Not moving local object in return statement (#4815)
ehds May 5, 2022
765510a
update TiFlash proxy to disable raft-engine (#4822)
CalvinNeo May 5, 2022
a18891a
Migrate table scan related code to `DAGStorageInterpreter` (#4783)
SeaRise May 5, 2022
d2a9639
*: remove useless BatchCommandsHandler. (#4828)
ywqzzy May 6, 2022
3a9a797
enable PageStorageV3 by default (#4831)
lidezhu May 6, 2022
19ecbde
fix ifnull bug (#4832)
windtalker May 6, 2022
6ca55ec
Fix a encryption problem in pagestorage v3. (#4177)
jiaqizho May 6, 2022
43fffe0
Add some tests to test PageStorageV3 IO limiter (#4094)
jiaqizho May 6, 2022
da9ebce
BlobStore:Add a block_alignment_bytes to make sure write have been al…
jiaqizho May 7, 2022
b05cdf9
Refine some comments about learner read (#4784)
JaySon-Huang May 7, 2022
9d3863d
Fix getMaxId won't get the right max id before GC (#4838)
jiaqizho May 7, 2022
ee1c8e4
Enable PageStorage V3 by default in dtworkload tool (#4817)
JinheLin May 7, 2022
793e89a
Disallow copy and move (#4578)
ZHANGWENTAI May 7, 2022
a6cb91e
avoid use MAX_NAMESPACE_ID as kvstore's ns_id (#4839)
lidezhu May 7, 2022
ac7ab7b
Fix dump snapshot encryption info not right after rename. (#4840)
jiaqizho May 7, 2022
5913002
Revert "Fix getMaxId won't get the right max id before GC" (#4845)
jiaqizho May 9, 2022
16fb597
Add a metric to statistics expansion or reused. (#4619)
jiaqizho May 9, 2022
938cc5c
MinMax Index Supports Nullable DataType (#4792)
hehechen May 10, 2022
3f5b302
Interpreter: Run interpreter test without exchange executors (#4788)
ywqzzy May 10, 2022
7107219
clear storage data before drop its metadata (#4673)
lidezhu May 11, 2022
d14a053
Fix wal may dump write diff type in same page_id (#4850)
jiaqizho May 11, 2022
85eb192
Fix bug for deseri type error that cause data error (#4854)
jiaqizho May 11, 2022
fd8bd3e
Interpreter: Remove Optimize query. (#4589)
ywqzzy May 11, 2022
f1d5991
Interpreter: Remove kill Query (#4564)
ywqzzy May 11, 2022
8bef777
Interpreter: Remove CheckQuery. (#4568)
ywqzzy May 11, 2022
8d5eff8
Interpreter: Remove Truncate query (#4587)
ywqzzy May 11, 2022
1fedb64
fix tispark query err: `for PushDownFilter, conditions and executor_i…
SeaRise May 11, 2022
f7324a0
PageStorage: Add mix mode (#4726)
jiaqizho May 12, 2022
ed7d6ed
Add fail point pause_after_copr_streams_acquired_once (#4833)
Lloyd-Pottiger May 12, 2022
6351eff
Interpreter: Remove Table properties query (#4586)
ywqzzy May 12, 2022
22594e9
Interpreter: Remove delete query (#4594)
ywqzzy May 12, 2022
bd0cd64
restore global storage pool in test cases (#4878)
hehechen May 12, 2022
ebb7673
Interpreter: Remove system query (#4590)
ywqzzy May 13, 2022
1a02963
misc: Add comments (#4863)
breezewish May 13, 2022
2ec5bac
update pprof for proxy (#4871)
CalvinNeo May 13, 2022
5a8b7be
add DB config max_threads by grpc context (#4795)
mengxin9014 May 13, 2022
80980ed
Add a force transfrom meta from v2 to v3 (#4873)
jiaqizho May 13, 2022
947dc73
Add to_seconds support for tiflash (#4866)
yibin87 May 13, 2022
df984f2
Support big page PUT into blobstore. (#4882)
jiaqizho May 13, 2022
c9a6079
Add a force transfrom kvstore from v2 to v3 (#4875)
jiaqizho May 13, 2022
e25d6d8
add metrics for storage pool run mode (#4881)
hehechen May 13, 2022
4abd22b
update proxy (#4894)
CalvinNeo May 16, 2022
ad0148c
Fix nullptr when mix mode change to only v3. (#4898)
jiaqizho May 16, 2022
849f81d
Revert "MinMax Index Supports Nullable DataType" (#4895)
hehechen May 16, 2022
20a0fa2
Delete config 'disable_bg_flush ' (#4869)
hzh0425 May 16, 2022
3013243
Interpreter: Run interpreter test with exchange executors (#4858)
ywqzzy May 17, 2022
2f2d0a4
update TiFlash proxy (#4907)
CalvinNeo May 17, 2022
64d7afe
Add option to support clang LTO(Link Time Optimization) and time-trac…
solotzg May 17, 2022
ab7f449
Fix the bug for restoring data with duplicated page id (#4889)
JaySon-Huang May 17, 2022
2380583
doc: add pagestorage v3 design doc (#4804)
jiaqizho May 18, 2022
df1ee1d
Fix some bugs in grafana panels (#4919)
JaySon-Huang May 18, 2022
bacf092
add design doc for mpp partition table support (#4913)
windtalker May 18, 2022
5a8f9c9
update tiflash proxy for unsafe recovery related PRs (#4921)
CalvinNeo May 18, 2022
ae9e54b
add window function doc (#4877)
mengxin9014 May 18, 2022
57fae94
feat: support the new compact RPC request (#4885)
breezewish May 18, 2022
faf3362
feat: Support scalar function `from_days` and `to_days` (#4905)
LittleFall May 18, 2022
8ea1348
fix bug of weekofyear (#4927)
windtalker May 19, 2022
edf8a52
Fix pageutils will truncate the file if write failed (#4925)
jiaqizho May 19, 2022
d7141b7
update tiflash proxy for fix of pprof (#4931)
CalvinNeo May 19, 2022
6d3ef87
docs: add TiDB Cloud in README.md (#4935)
Oreoxmt May 19, 2022
7044e97
docs: git clone concurrently (#4930)
breezewish May 19, 2022
245a875
Fix ref a del id failed in mix mode. (#4933)
jiaqizho May 19, 2022
70a2770
expression: check Overflow int128 (#4762)
fzhedu May 20, 2022
759d022
Add a fail point named force_change_all_blobs_to_read_only (#4920)
jiaqizho May 20, 2022
120d0a8
Fix StorageDeltaMerge constructor not match in gtest_manual_compact (…
jiaqizho May 20, 2022
ebf12fa
Add note about how to handle build oom or hang (#4951)
zanmato1984 May 20, 2022
b9289cd
flush cache before segment merge (#4955)
lidezhu May 23, 2022
4044129
Add a global max_id to fix reuse page_id problem (#4948)
jiaqizho May 23, 2022
26b11f9
Schedule: track the waiting tasks with task ID, and deleted the sched…
fzhedu May 23, 2022
f9eb43b
Enable LTO in release build process (#4924)
solotzg May 23, 2022
2948e03
Revert the retry applyRecord when loadEdit (#4938)
hehechen May 24, 2022
caaff2a
Remove useless template instantiations (#4978)
solotzg May 24, 2022
1a79d29
fix clang tidy err for `ExpressionAnalyzer`, `Join`, `StorageJoin`, `…
SeaRise May 24, 2022
5e55fa8
refine `handleJoin` (#4722)
SeaRise May 24, 2022
2f845ef
Change BlobFileId to uint64. (#4984)
jiaqizho May 24, 2022
11224ce
*: Rename the function with “withCollation” postfix (#3790)
ywqzzy May 24, 2022
5e781b2
PageStorage: Fix the WAL snapshot dumped into file may contains inval…
JaySon-Huang May 24, 2022
1bf77fb
Respect the cpu quota limit of cgroup
JaySon-Huang Dec 31, 2021
4ce3d48
Respect the memory quota limit of cgroup
JaySon-Huang Dec 31, 2021
ffd1a15
Record the cpu/memory quota in metrics
JaySon-Huang Dec 31, 2021
dea7699
add cpu conuts and memory amount pannel
jiaqizho Jan 12, 2022
aeee1fe
update
jiaqizho Jan 12, 2022
a8aa277
change the memory cgroup
jiaqizho Jan 12, 2022
7f0f88d
memory inside docker
jiaqizho Jan 13, 2022
6965142
update cpu
jiaqizho Jan 13, 2022
fa97fcb
update grafana
jiaqizho Jan 13, 2022
fbb5454
fix build
jiaqizho Jan 13, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Add a global max_id to fix reuse page_id problem (#4948)
close #4939
jiaqizho authored May 23, 2022
commit 4044129aeb12bbbf827b64f4a138b07e5619df41
13 changes: 11 additions & 2 deletions dbms/src/Storages/DeltaMerge/DeltaMergeStore.cpp
Original file line number Diff line number Diff line change
@@ -244,10 +244,19 @@ DeltaMergeStore::DeltaMergeStore(Context & db_context,
if (const auto first_segment_entry = storage_pool->metaReader()->getPageEntry(DELTA_MERGE_FIRST_SEGMENT_ID);
!first_segment_entry.isValid())
{
// Create the first segment.
auto segment_id = storage_pool->newMetaPageId();
if (segment_id != DELTA_MERGE_FIRST_SEGMENT_ID)
throw Exception(fmt::format("The first segment id should be {}", DELTA_MERGE_FIRST_SEGMENT_ID), ErrorCodes::LOGICAL_ERROR);
{
if (page_storage_run_mode == PageStorageRunMode::ONLY_V2)
{
throw Exception(fmt::format("The first segment id should be {}", DELTA_MERGE_FIRST_SEGMENT_ID), ErrorCodes::LOGICAL_ERROR);
}

// In ONLY_V3 or MIX_MODE, If create a new DeltaMergeStore
// Should used fixed DELTA_MERGE_FIRST_SEGMENT_ID to create first segment
segment_id = DELTA_MERGE_FIRST_SEGMENT_ID;
}

auto first_segment
= Segment::newSegment(*dm_context, store_columns, RowKeyRange::newAll(is_common_handle, rowkey_column_size), segment_id, 0);
segments.emplace(first_segment->getRowKeyRange().getEnd(), first_segment);
24 changes: 12 additions & 12 deletions dbms/src/Storages/DeltaMerge/StoragePool.cpp
Original file line number Diff line number Diff line change
@@ -378,18 +378,18 @@ PageStorageRunMode StoragePool::restore()
data_storage_v2->restore();
meta_storage_v2->restore();

max_log_page_id = log_storage_v2->getMaxId(ns_id);
max_data_page_id = data_storage_v2->getMaxId(ns_id);
max_meta_page_id = meta_storage_v2->getMaxId(ns_id);
max_log_page_id = log_storage_v2->getMaxId();
max_data_page_id = data_storage_v2->getMaxId();
max_meta_page_id = meta_storage_v2->getMaxId();

storage_pool_metrics = CurrentMetrics::Increment{CurrentMetrics::StoragePoolV2Only};
break;
}
case PageStorageRunMode::ONLY_V3:
{
max_log_page_id = log_storage_v3->getMaxId(ns_id);
max_data_page_id = data_storage_v3->getMaxId(ns_id);
max_meta_page_id = meta_storage_v3->getMaxId(ns_id);
max_log_page_id = log_storage_v3->getMaxId();
max_data_page_id = data_storage_v3->getMaxId();
max_meta_page_id = meta_storage_v3->getMaxId();

storage_pool_metrics = CurrentMetrics::Increment{CurrentMetrics::StoragePoolV3Only};
break;
@@ -456,18 +456,18 @@ PageStorageRunMode StoragePool::restore()
data_storage_writer = std::make_shared<PageWriter>(PageStorageRunMode::ONLY_V3, /*storage_v2_*/ nullptr, data_storage_v3);
meta_storage_writer = std::make_shared<PageWriter>(PageStorageRunMode::ONLY_V3, /*storage_v2_*/ nullptr, meta_storage_v3);

max_log_page_id = log_storage_v3->getMaxId(ns_id);
max_data_page_id = data_storage_v3->getMaxId(ns_id);
max_meta_page_id = meta_storage_v3->getMaxId(ns_id);
max_log_page_id = log_storage_v3->getMaxId();
max_data_page_id = data_storage_v3->getMaxId();
max_meta_page_id = meta_storage_v3->getMaxId();

run_mode = PageStorageRunMode::ONLY_V3;
storage_pool_metrics = CurrentMetrics::Increment{CurrentMetrics::StoragePoolV3Only};
}
else // Still running Mix Mode
{
max_log_page_id = std::max(log_storage_v2->getMaxId(ns_id), log_storage_v3->getMaxId(ns_id));
max_data_page_id = std::max(data_storage_v2->getMaxId(ns_id), data_storage_v3->getMaxId(ns_id));
max_meta_page_id = std::max(meta_storage_v2->getMaxId(ns_id), meta_storage_v3->getMaxId(ns_id));
max_log_page_id = std::max(log_storage_v2->getMaxId(), log_storage_v3->getMaxId());
max_data_page_id = std::max(data_storage_v2->getMaxId(), data_storage_v3->getMaxId());
max_meta_page_id = std::max(meta_storage_v2->getMaxId(), meta_storage_v3->getMaxId());
storage_pool_metrics = CurrentMetrics::Increment{CurrentMetrics::StoragePoolMixMode};
}
break;
11 changes: 9 additions & 2 deletions dbms/src/Storages/DeltaMerge/StoragePool.h
Original file line number Diff line number Diff line change
@@ -149,11 +149,18 @@ class StoragePool : private boost::noncopyable
// Caller must cancel gc tasks before drop
void drop();

PageId newDataPageIdForDTFile(StableDiskDelegator & delegator, const char * who);
// For function `newLogPageId`,`newMetaPageId`,`newDataPageIdForDTFile`:
// For PageStorageRunMode::ONLY_V2, every table have its own three PageStorage (meta/data/log).
// So these functions return the Page id starts from 1 and is continuously incremented.
// For PageStorageRunMode::ONLY_V3/MIX_MODE, PageStorage is global(distinguish by ns_id for different table).
// In order to avoid Page id from being reused (and cause troubles while restoring WAL from disk),
// StoragePool will assign the max_log_page_id/max_meta_page_id/max_data_page_id by the global max id
// regardless of ns_id while being restored. This causes the ids in a table to not be continuously incremented.

PageId maxMetaPageId() { return max_meta_page_id; }
PageId newDataPageIdForDTFile(StableDiskDelegator & delegator, const char * who);
PageId newLogPageId() { return ++max_log_page_id; }
PageId newMetaPageId() { return ++max_meta_page_id; }

#ifndef DBMS_PUBLIC_GTEST
private:
#endif
12 changes: 11 additions & 1 deletion dbms/src/Storages/Page/PageStorage.h
Original file line number Diff line number Diff line change
@@ -233,7 +233,17 @@ class PageStorage : private boost::noncopyable

virtual void drop() = 0;

virtual PageId getMaxId(NamespaceId ns_id) = 0;
// Get the max id from PageStorage.
//
// For V2, every table have its own three PageStorage (meta/data/log).
// So this function return the Page id starts from 0 and is continuously incremented to
// new pages.
// For V3, PageStorage is global(distinguish by ns_id for different table).
// In order to avoid Page id from being reused (and cause troubles while restoring WAL from disk),
// this function returns the global max id regardless of ns_id. This causes the ids in a table
// to not be continuously incremented.
// Note that Page id 1 in each ns_id is special.
virtual PageId getMaxId() = 0;

virtual SnapshotPtr getSnapshot(const String & tracing_id) = 0;

6 changes: 3 additions & 3 deletions dbms/src/Storages/Page/V2/PageStorage.cpp
Original file line number Diff line number Diff line change
@@ -355,7 +355,7 @@ void PageStorage::restore()
LOG_FMT_INFO(log, "{} restore {} pages, write batch sequence: {}, {}", storage_name, num_pages, write_batch_seq, statistics.toString());
}

PageId PageStorage::getMaxId(NamespaceId /*ns_id*/)
PageId PageStorage::getMaxId()
{
std::lock_guard write_lock(write_mutex);
return versioned_page_entries.getSnapshot("")->version()->maxId();
@@ -893,9 +893,9 @@ void PageStorage::drop()
struct GcContext
{
PageFileIdAndLevel min_file_id;
PageFile::Type min_file_type;
PageFile::Type min_file_type = PageFile::Type::Invalid;
PageFileIdAndLevel max_file_id;
PageFile::Type max_file_type;
PageFile::Type max_file_type = PageFile::Type::Invalid;
size_t num_page_files = 0;
size_t num_legacy_files = 0;

2 changes: 1 addition & 1 deletion dbms/src/Storages/Page/V2/PageStorage.h
Original file line number Diff line number Diff line change
@@ -95,7 +95,7 @@ class PageStorage : public DB::PageStorage

void drop() override;

PageId getMaxId(NamespaceId ns_id) override;
PageId getMaxId() override;

PageId getNormalPageIdImpl(NamespaceId ns_id, PageId page_id, SnapshotPtr snapshot, bool throw_on_not_exist) override;

49 changes: 7 additions & 42 deletions dbms/src/Storages/Page/V3/PageDirectory.cpp
Original file line number Diff line number Diff line change
@@ -685,7 +685,8 @@ void VersionedPageEntries::collapseTo(const UInt64 seq, const PageIdV3Internal p
*************************/

PageDirectory::PageDirectory(String storage_name, WALStorePtr && wal_, UInt64 max_persisted_log_files_)
: sequence(0)
: max_page_id(0)
, sequence(0)
, wal(std::move(wal_))
, max_persisted_log_files(max_persisted_log_files_)
, log(Logger::get("PageDirectory", std::move(storage_name)))
@@ -923,49 +924,10 @@ PageIdV3Internal PageDirectory::getNormalPageId(PageIdV3Internal page_id, const
}
}

PageId PageDirectory::getMaxId(NamespaceId ns_id) const
PageId PageDirectory::getMaxId() const
{
std::shared_lock read_lock(table_rw_mutex);
PageIdV3Internal upper_bound = buildV3Id(ns_id, UINT64_MAX);

auto iter = mvcc_table_directory.upper_bound(upper_bound);
if (iter == mvcc_table_directory.begin())
{
// The smallest page id is greater than the target page id or mvcc_table_directory is empty,
// and it means no page id is less than or equal to the target page id, return 0.
return 0;
}
else
{
// iter is not at the beginning and mvcc_table_directory is not empty,
// so iter-- must be a valid iterator, and it's the largest page id which is smaller than the target page id.
iter--;

do
{
// Can't find any entries in current ns_id
if (iter->first.high != ns_id)
{
break;
}

// Check and return whether this id is visible, otherwise continue to check the previous one.
if (iter->second->isVisible(UINT64_MAX - 1))
{
return iter->first.low;
}

// Current entry/ref/external is deleted and there are no entries before it.
if (iter == mvcc_table_directory.begin())
{
break;
}

iter--;
} while (true);

return 0;
}
return max_page_id;
}

std::set<PageIdV3Internal> PageDirectory::getAllPageIds()
@@ -1069,6 +1031,9 @@ void PageDirectory::apply(PageEntriesEdit && edit, const WriteLimiterPtr & write
// stage 2, create entry version list for page_id.
for (const auto & r : edit.getRecords())
{
// Protected in write_lock
max_page_id = std::max(max_page_id, r.page_id.low);

auto [iter, created] = mvcc_table_directory.insert(std::make_pair(r.page_id, nullptr));
if (created)
{
3 changes: 2 additions & 1 deletion dbms/src/Storages/Page/V3/PageDirectory.h
Original file line number Diff line number Diff line change
@@ -347,7 +347,7 @@ class PageDirectory
}
#endif

PageId getMaxId(NamespaceId ns_id) const;
PageId getMaxId() const;

std::set<PageIdV3Internal> getAllPageIds();

@@ -397,6 +397,7 @@ class PageDirectory
}

private:
PageId max_page_id;
std::atomic<UInt64> sequence;
mutable std::shared_mutex table_rw_mutex;
MVCCMapType mvcc_table_directory;
4 changes: 3 additions & 1 deletion dbms/src/Storages/Page/V3/PageDirectoryFactory.cpp
Original file line number Diff line number Diff line change
@@ -40,6 +40,7 @@ PageDirectoryPtr PageDirectoryFactory::create(String storage_name, FileProviderP
// After restoring from the disk, we need cleanup all invalid entries in memory, or it will
// try to run GC again on some entries that are already marked as invalid in BlobStore.
dir->gcInMemEntries();
LOG_FMT_INFO(DB::Logger::get("PageDirectoryFactory"), "PageDirectory restored [max_page_id={}] [max_applied_ver={}]", dir->getMaxId(), dir->sequence);

if (blob_stats)
{
@@ -111,7 +112,6 @@ void PageDirectoryFactory::loadEdit(const PageDirectoryPtr & dir, const PageEntr
{
if (max_applied_ver < r.version)
max_applied_ver = r.version;
max_applied_page_id = std::max(r.page_id, max_applied_page_id);

// We can not avoid page id from being reused under some corner situation. Try to do gcInMemEntries
// and apply again to resolve the error.
@@ -135,6 +135,8 @@ bool PageDirectoryFactory::applyRecord(
iter->second = std::make_shared<VersionedPageEntries>();
}

dir->max_page_id = std::max(dir->max_page_id, r.page_id.low);

const auto & version_list = iter->second;
const auto & restored_version = r.version;
try
1 change: 0 additions & 1 deletion dbms/src/Storages/Page/V3/PageDirectoryFactory.h
Original file line number Diff line number Diff line change
@@ -38,7 +38,6 @@ class PageDirectoryFactory
{
public:
PageVersion max_applied_ver;
PageIdV3Internal max_applied_page_id;

PageDirectoryFactory & setBlobStore(BlobStore & blob_store)
{
4 changes: 2 additions & 2 deletions dbms/src/Storages/Page/V3/PageStorageImpl.cpp
Original file line number Diff line number Diff line change
@@ -55,9 +55,9 @@ void PageStorageImpl::restore()
.create(storage_name, file_provider, delegator, parseWALConfig(config));
}

PageId PageStorageImpl::getMaxId(NamespaceId ns_id)
PageId PageStorageImpl::getMaxId()
{
return page_directory->getMaxId(ns_id);
return page_directory->getMaxId();
}

void PageStorageImpl::drop()
2 changes: 1 addition & 1 deletion dbms/src/Storages/Page/V3/PageStorageImpl.h
Original file line number Diff line number Diff line change
@@ -64,7 +64,7 @@ class PageStorageImpl : public DB::PageStorage

void drop() override;

PageId getMaxId(NamespaceId ns_id) override;
PageId getMaxId() override;

PageId getNormalPageIdImpl(NamespaceId ns_id, PageId page_id, SnapshotPtr snapshot, bool throw_on_not_exist) override;

4 changes: 2 additions & 2 deletions dbms/src/Storages/Page/V3/WAL/serialize.cpp
Original file line number Diff line number Diff line change
@@ -218,7 +218,7 @@ void deserializeFrom(ReadBuffer & buf, PageEntriesEdit & edit)
break;
}
default:
throw Exception(fmt::format("Unknown record type: {}", record_type));
throw Exception(fmt::format("Unknown record type: {}", record_type), ErrorCodes::LOGICAL_ERROR);
}
}
}
@@ -261,7 +261,7 @@ PageEntriesEdit deserializeFrom(std::string_view record)
UInt32 version = 0;
readIntBinary(version, buf);
if (version != 1)
throw Exception("");
throw Exception(fmt::format("Unknown version for PageEntriesEdit deser [version={}]", version), ErrorCodes::LOGICAL_ERROR);

deserializeFrom(buf, edit);
return edit;
Loading