Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge branch 'selectdb-cloud-release-2.0' (20221201 986a3d5e7) into s…
…electdb-cloud-dev (20221130 23a144c) (apache#1199) * [feature](selectdb-cloud) Fix file cache metrics nullptr error (apache#1060) * [feature](selectdb-cloud) Fix abort copy when -235 (apache#1039) * [feature](selectdb-cloud) Replace libfdb_c.so to make it compatible with different OS (apache#925) * [feature](selectdb-cloud) Optimize RPC retry in cloud_meta_mgr (apache#1027) * Optimize RETRY_RPC in cloud_meta_mgr * Add random sleep for RETRY_RPC * Add a simple backoff strategy for rpc retry * [feature](selectdb-cloud) Copy into support select by column name (apache#1055) * Copy into support select by column name * Fix broker load core dump due to mis-match of number of columns between remote and schema * [feature](selectdb-cloud) Fix test_dup_mv_schema_change case (apache#1022) * [feature](selectdb-cloud) Make the broker execute on the specified cluster (apache#1043) * Make the broker execute on the specified cluster * Pass the cluster parameter * [feature](selectdb-cloud) Support concurrent BaseCompaction and CumuCompaction on a tablet (apache#1059) * [feature](selectdb-cloud) Reduce meta-service log (apache#1067) * Quote string in the tagged log * Add template to enable customized log for RPC requests * [feature](selectdb-cloud) Use read-only txn + read-write txn for `commit_txn` (apache#1065) * [feature](selectdb-cloud) Pick "[fix](load) fix that load channel failed to be released in time (apache#14119)" commit 3690c4d Author: Xin Liao <[email protected]> Date: Wed Nov 9 22:38:08 2022 +0800 [fix](load) fix that load channel failed to be released in time (apache#14119) * [feature](selectdb-cloud) Add compaction profile log (apache#1072) * [feature](selectdb-cloud) Fix abort txn fail when copy job `getAllFileStatus` exception (apache#1066) * Revert "[feature](selectdb-cloud) Copy into support select by column name (apache#1055)" This reverts commit f1a543e. * [feature](selectdb-cloud) Pick"[fix](metric) fix the bug of not updating the query latency metric apache#14172 (apache#1076)" * [feature](selectdb-cloud) Distinguish KV_TXN_COMMIT_ERR or KV_TXN_CONFLICT while commit failed (apache#1082) * [feature](selectdb-cloud) Support configuring base compaction concurrency (apache#1080) * [feature](selectdb-cloud) Enhance start.sh/stop.sh for selectdb_cloud (apache#1079) * [feature](selectdb-cloud) Add smoke testing (apache#1056) Add smoke test, 1. upload,query http data api. 2. internal, external stage. 3. select,insert * [feature](selectdb-cloud) Disable admin stmt in cloud mode (apache#1064) Disable the following stmt. * AdminRebalanceDiskStmt/AdminCancelRebalanceDiskStmt * AdminRepairTableStmt/AdminCancelRepairTableStmt * AdminCheckTabletsStmt * AdminCleanTrashStmt * AdminCompactTableStmt * AdminCopyTabletStmt * AdminDiagnoseTabletStmt * AdminSetConfigStmt * AdminSetReplicaStatusStmt * AdminShowConfigStmt * AdminShowReplicaDistributionStmt * AdminShowReplicaStatusStmt * AdminShowTabletStorageFormatStmt Leaving a backdoor for the user root: * AdminSetConfigStmt * AdminShowConfigStmt * AdminShowReplicaDistributionStmt * AdminShowReplicaStatusStmt * AdminDiagnoseTabletStmt * [feature](selectdb-cloud) Update copy into doc (apache#1063) * [feature](selectdb-cloud) Fix AdminSetConfigStmt cannot work with root (apache#1085) * [feature](selectdb-cloud) Fix userid null lead to checkpoint error (apache#1083) * [feature](selectdb-cloud) Support controling the space used for upload (apache#1091) * [feature](selectdb-cloud) Pick "[fix](sequence) fix that update table core dump with sequence column (apache#13847)" (apache#1092) * [Fix](memory-leak) Fix boost::stacktrace memory leak (1097) * [Fix](selectdb-cloud) Several picks to fix memtracker (apache#1087) * [enhancement](memtracker) Add independent and unique scanner mem tracker for each query (apache#13262) * [enhancement](memory) Print memory usage log when memory allocation fails (apache#13301) * [enhancement](memtracker) Print query memory usage log every second when `memory_verbose_track` is enabled (apache#13302) * [fix](memory) Fix USE_JEMALLOC=true UBSAN compilation error apache#13398 * [enhancement](memtracker) Fix bthread local consume mem tracker (apache#13368) Previously, bthread_getspecific was called every time bthread local was used. In the test at apache#10823, it was found that frequent calls to bthread_getspecific had performance problems. So a cache is implemented on pthread local based on the btls key, but the btls key cannot correctly sense bthread switching. So, based on bthread_self to get the bthread id to implement the cache. * [enhancement](memtracker) Fix brpc causing query mem tracker to be inaccurate apache#13401 * [fix](memtracker) Fix transmit_tracker null pointer because phamp is not thread safe apache#13528 * [enhancement](memtracker) Fix Brpc mem count and refactored thread context macro (apache#13469) * [fix](memtracker) Fix the usage of bthread mem tracker (apache#13708) bthead context init has performance loss, temporarily delete it first, it will be completely refactored in apache#13585. * [enhancement](memtracker) Refactor load channel + memtable mem tracker (apache#13795) * [fix](load) Fix load channel mgr lock (apache#13960) hot fix load channel mgr lock * [fix](memtracker) Fix DCHECK !std::count(_consumer_tracker_stack.begin(), _consumer_tracker_stack.end(), tracker) * [tempfix][memtracker] wait pick 0b945fe Co-authored-by: Xinyi Zou <[email protected]> * [feature](selectdb-cloud) Add more recycler case (apache#1094) * [feature](selectdb-cloud) Pick "[improvement](load) some simple optimization for reduce load memory policy (apache#14215)" (apache#1096) * [feature](selectdb-cloud) Reduce unnecessary get rowset rpc when prepare compaction (apache#1099) * [feature](selectdb-cloud) Pick "[improvement](load) reduce memory in batch for small load channels (apache#14214)" (apache#1100) * [feature](selectdb-cloud) Pick "[improvement](load) release load channel actively when error occurs (apache#14218)" (apache#1102) * [feature](selectdb-cloud) Print build info of ms/recycler to stdout when launch (apache#1105) * [feature](selectdb-cloud) copy into support select by column name and load with partial columns (apache#1104) e.g. ``` COPY INTO test_table FROM (SELECT col1, col2, col3 FROM @ext_stage('1.parquet')) COPY INTO test_table (id, name) FROM (SELECT col1, col2 FROM @ext_stage('1.parquet')) ``` * [fix](selectdb-cloud) Pick "[Fix](array-type) bugfix for array column with delete condition (apache#13361)" (apache#1109) Fix for SQL with array column: delete from tbl where c_array is null; more info please refer to apache#13360 Co-authored-by: camby <[email protected]> Co-authored-by: cambyzju <[email protected]> * [feature](selectdb-cloud) Copy into support force (apache#1081) * [feature](selectdb-cloud) Add abort txn, abort tablet job http api (apache#1101) Abort load txn by txn_id: ``` curl "{meta_sevice_ip}:{brpc_port}/MetaService/http/abort_txn?token=greedisgood9999" -d '{ "cloud_unique_id": string, "txn_id": int64 }' ``` Abort load txn by db_id and label: ``` curl "{meta_sevice_ip}:{brpc_port}/MetaService/http/abort_txn?token=greedisgood9999" -d '{ "cloud_unique_id": string, "db_id": int64, "label": string }' ``` Only support abort compaction job currently: ``` curl "{meta_sevice_ip}:{brpc_port}/MetaService/http/abort_tablet_job?token=greedisgood9999" -d '{ "cloud_unique_id": string, "job" : { "idx": {"tablet_id": int64}, "compaction": [{"id": string}] } }' ``` * [feature](selectdb-cloud) Fix external stage data for smoke test and retry to create stage (apache#1119) * [feature](selectdb-cloud) Fix data leaks when truncating table (apache#1114) * Drop cloud partition when truncating table * Add retry strategy for dropCloudMaterializedIndex * [feature](selectdb-cloud) Fix missing library when compiling unit test (apache#1128) * [feature](selectdb-cloud) Validate the object storage when create stage (apache#1115) * [feature](selectdb-cloud) Fix incorrectly setting cumulative point when committing base compaction (apache#1127) * [feature](selectdb-cloud) Fix missing lease when preparing cumulative compaction (apache#1131) * [feature](selectdb-cloud) Fix unbalanced tablet distribution (apache#1121) * Fix the bug of unbalanced tablet distribution * Use replica index hash to BE * [feature](selectdb-cloud) Fix core dump when get tablets info by BE web page (apache#1113) * [feature](selectdb-cloud) Fix start_fe.sh --version (apache#1106) * [feature](selectdb-cloud) Print tablet stats before and after compaction (apache#1132) * Log num rowsets before and after compaction * Print tablet stats after committing compaction * [feature](selectdb-cloud) Allow root user execute AlterSystemStmt (apache#1143) * [feature](selectdb-cloud) Fix BE UT (apache#1141) * [feature](selectdb-cloud) Select BE for the first bucket of every partition randomly (apache#1136) * [feature](selectdb-cloud) Fix query_limit int -> int64 (apache#1154) * [feature](selectdb-cloud) Add more cloud recycler case (apache#1116) * add more cloud recycler case * modify cloud recycler case dateset from sf0.1 to sf1 * [feature](selectdb-cloud) Fix misuse of aws transfer which may delete tmp file prematurely (apache#1160) * [feature](selectdb-cloud) Add test for copy into http data api and userId (apache#1044) * Add test for copy into http data api and userId * Add external and internal stage cross use regression case. * [feature](selectdb-cloud) Pass the cloud compaction regression test (apache#1173) * [feature](selectdb-cloud) Modify max_bytes_per_broker_scanner default value to 150G (apache#1184) * [feature](selectdb-cloud) Fix missing lock when calling Tablet::delete_predicates (apache#1182) * [improvement](config)change default remote_fragment_exec_timeout_ms to 30 seconds * [improvement](config) change default value of broker_load_default_timeout_second to 12 hours * [feature](selectdb-cloud) Fix replay copy into (apache#1167) * Add stage ddl regression * fix replay copy into * remove unused log * fix user name * [feature](selectdb-cloud) Fix FE --version option not work after fe started (apache#1161) * [feature](selectdb-cloud) BE accesses object store using HTTP (apache#1111) * [feature](selectdb-cloud) Refactor recycle copy jobs (apache#1062) * [fix](FE) Pick fix from doris master (apache#1177) (apache#1178) Commit: 53e5f39 Author: starocean999 <[email protected]> Committer: GitHub <[email protected]> Date: Mon Oct 31 2022 10:19:32 GMT+0800 (China Standard Time) fix result exprs should be substituted in the same way as agg exprs (apache#13744) Commit: a4a9912 Author: starocean999 <[email protected]> Committer: GitHub <[email protected]> Date: Thu Nov 03 2022 10:26:59 GMT+0800 (China Standard Time) fix group by constant value bug (apache#13827) Commit: 84b969a Author: starocean999 <[email protected]> Committer: GitHub <[email protected]> Date: Thu Nov 10 2022 11:10:42 GMT+0800 (China Standard Time) fix the grouping expr should check col name from base table first, then alias (apache#14077) Commit: ae4f4b9 Author: starocean999 <[email protected]> Committer: GitHub <[email protected]> Date: Thu Nov 24 2022 10:31:58 GMT+0800 (China Standard Time) fix having clause should use column name first then alias (apache#14408) * [feature](selectdb-cloud) Deal with getNextTransactionId rpc exception (apache#1181) Before fixing, getNextTransactionId will return -1 if there is RPC exception, it will cause schema change and the previous load task execute in parallel unexpectedly. * [feature](selectdb-cloud) Throw exception for unsupported operations in CloudGlobalTransactionMgr (apache#1180) * [improvement](load) Add more log on RPC error (apache#1183) * [feature](selectdb-cloud) Add copy_into case(json, parquet, orc) and tpch_sf1 to smoke test (apache#1140) * [feature](selectdb-cloud) Recycle dropped stage (apache#1071) * log s3 response code * add log in S3Accessor::delete_objects_by_prefix * Fix show copy * remove empty line * [feature](selectdb-cloud) Support bthread for new scanner (apache#1117) * Support bthread for new scanner * Keep the number of remote threads same as local threads * [feature](selectdb-cloud) Implement self-explained cloud unique id for instance id searching (apache#1089) 1. Implement self-explained cloud unique id for instance id searching 2. Fix register core when metaservice start error 3. Fix drop_instance not set mtime 4. Add HTTP API to get instance info ``` curl "127.0.0.1:5008/MetaService/http/get_instance?token=greedisgood9999&cloud_unique_id=regression-cloud-unique-id-fe-1" curl "127.0.0.1:5008/MetaService/http/get_instance?token=greedisgood9999&cloud_unique_id=1:regression_instance0:regression-cloud-unique-id-fe-1" curl "127.0.0.1:5008/MetaService/http/get_instance?token=greedisgood9999&instance_id=regression_instance0" ``` * [improvement](memory) simplify memory config related to tcmalloc and add gc (apache#1191) * [improvement](memory) simplify memory config related to tcmalloc There are several configs related to tcmalloc, users do know how to config them. Actually users just want two modes, performance or compact, in performance mode, users want doris run query and load quickly while in compact mode, users want doris run with less memory usage. If we want to config tcmalloc individually, we can use env variables which are supported by tcmalloc. * [improvement](tcmalloc) add moderate mode and avoid oom with a lot of cache (apache#14374) ReleaseToSystem aggressively when there are little free memory. * [feature](selectdb-cloud) Pick "[fix](hashjoin) fix coredump of hash join in ubsan build apache#13479" (apache#1190) commit b5cd167 Author: TengJianPing <[email protected]> Date: Thu Oct 20 10:16:19 2022 +0800 [fix](hashjoin) fix coredump of hash join in ubsan build (apache#13479) * [feature](selectdb-cloud) Support close FileWriter without forcing sync data to storage medium (apache#1134) * Trace accumulated time * Support close FileWriter without forcing sync data to storage medium * Avoid trace overhead when disable trace * [feature](selectdb-cloud) Pick "[BugFix](function) fix reverse function dynamic buffer overflow due to illegal character apache#13671" (apache#1146) * pick [opt](exec) Replace get_utf8_byte_length function by array (apache#13664) * pick [BugFix](function) fix reverse function dynamic buffer overflow due to illegal character apache#13671 Co-authored-by: HappenLee <[email protected]> * [feature](selectdb-cloud) Pick "[fix](fe) Inconsistent behavior for string comparison in FE and BE (apache#13604)" (apache#1150) Co-authored-by: xueweizhang <[email protected]> * [feature](selectdb-cloud) Copy into support delete_on condition (apache#1148) * [feature](selectdb-cloud) Pick "[fix](agg)fix group by constant value bug (apache#13827)" (apache#1152) * [fix](agg)fix group by constant value bug * keep only one const grouping exprs if no agg exprs Co-authored-by: starocean999 <[email protected]> * [feature](selectdb-cloud) Pick "[fix](join)the build and probe expr should be calculated before converting input block to nullable (apache#13436)" (apache#1155) * [fix](join)the build and probe expr should be calculated before converting input block to nullable * remove_nullable can be called on const column Co-authored-by: starocean999 <[email protected]> * [feature](selectdb-cloud) Pick "[Bug](predicate) fix core dump on bool type runtime filter (apache#13417)" (apache#1156) fix core dump on bool type runtime filter Co-authored-by: Pxl <[email protected]> * [feature](selectdb-cloud) Pick "[Fix](agg) fix bitmap agg core dump when phmap pointer assert alignment (apache#13381)" (apache#1157) Co-authored-by: zhangstar333 <[email protected]> * [feature](selectdb-cloud) Pick "[Bug](function) fix core dump on case when have 1000 condition apache#13315" (apache#1158) Co-authored-by: Pxl <[email protected]> * [feature](selectdb-cloud) Pick "[fix](sort)the sort expr nullable info is wrong in some case (apache#12003)" * [feature](selectdb-cloud) Pick "[Improvement](decimal) print decimal according to the real precision and scale (apache#13437)" * [feature](selectdb-cloud) Pick "[bugfix](VecDateTimeValue) eat the value of microsecond in function from_date_format_str (apache#13446)" * [bugfix](VecDateTimeValue) eat the value of microsecond in function from_date_format_str * add sql based regression test Co-authored-by: xiaojunjie <[email protected]> Co-authored-by: Lightman <[email protected]> Co-authored-by: meiyi <[email protected]> Co-authored-by: Xiaocc <[email protected]> Co-authored-by: Lei Zhang <[email protected]> Co-authored-by: Xin Liao <[email protected]> Co-authored-by: Luwei <[email protected]> Co-authored-by: plat1ko <[email protected]> Co-authored-by: deardeng <[email protected]> Co-authored-by: Kidd <[email protected]> Co-authored-by: Xinyi Zou <[email protected]> Co-authored-by: zhannngchen <[email protected]> Co-authored-by: camby <[email protected]> Co-authored-by: cambyzju <[email protected]> Co-authored-by: Yongqiang YANG <[email protected]> Co-authored-by: starocean999 <[email protected]> Co-authored-by: Gabriel <[email protected]> Co-authored-by: AlexYue <[email protected]> Co-authored-by: xueweizhang <[email protected]> Co-authored-by: Pxl <[email protected]> Co-authored-by: zhangstar333 <[email protected]> Co-authored-by: xiaojunjie <[email protected]> Co-authored-by: xiaojunjie <[email protected]>
- Loading branch information
23 people
authored
Dec 1, 2022
1 parent
a021bb4
commit 96c230b