Release 2.5.0

[empiredan]

Release Notes

https://github.com/apache/incubator-pegasus/blob/50563ea33b33c5ae1a6d9b8e4cf74a7d1d9133b0/HISTORY.md

Change List

All of the changes that are made in version 2.5.0 are list here.

Behavior changes

• feat(thirdparty): use official rocksdb #1048
• refactor(log): adjust log level and macro names #1200
• feat(slog): flush and remove all shared logs for garbage collection (cherry-pick #1594) #1638
• chore: remove the supporting of EOL OS ubuntu 16.04 #1553
• chore: remove the supporting of EOL OS CentOS 6 #1557

New Features

• feat: Support Build on M1 MacOS #1094
• feat(server): disable UDP service by config #1132
• feat(jemalloc): support to dump stats #1133
• feat: expose parameter success_if_exist for interface create_app #1148
• feat: add rename_app interface  #1272
• build: Support Ubuntu 22.04 and clang 14 #1350
• feat(java-client): add listApps interface #1471
• Feature:Online Query and Dynamic Modification of Table-level RocksDB Options #1511
• build: add a devcontainer.json #1544

Some bigger features with multiple pull requests are listed as below:

Access control based on Apache Ranger

• feat(security): Use Apache Ranger for access control(1/n) #1360
• feat(security): Use Apache Ranger for access control(2/n) #1375
• feat(Ranger): Support encode and decode Ranger policies #1378
• feat(Ranger): Compatible with the old ACL and the new ACL #1379
• feat(Ranger): Pull policies from the Ranger Service and update using resources policies #1388
• feat(Ranger): Use Apache Ranger for meta access controller #1402
• feat(Ranger): Use Apache Ranger for replica access controller #1433
• feat(Ranger): Use Apache Ranger for ACL when replica master-slave learn #1445
• feat(Ranger): Use Apache Ranger for ACL when replica performs nfs copy #1452
• feat(Ranger): remove ACL of RPC_CM_LIST_APPS from global resource #1500
• feat(Ranger): add a default database policy name for legacy table #1507
• feat(Ranger): refactor the logic when ranger performs ACL #1518
• Pegasus-Shell supports executing kinit outside of the program and completing authentication within the program #1569

Support FQDN

• feat(FQDN): Implement of struct rpc_host_port #1430
• feat(FQDN): Implement of struct host_port_group #1436
• feat(FQDN): Implemention of struct dns_resolver #1464
• feat(FQDN): serialization format rpc_host_port #1496

Build Feature

• chore(script): Script downgrade_node.sh supports using specified config file when using shell client #1369
• chore(script): Script migrate_node.sh supports using specified config file when using shell config #1370
• chore(build): make build dir standalone and speedup re-build #1401
• fix(slog): add an option to exit the process when find slog error #1574

Feature enhancement

Count data

• feat: improve performance of count_data #1091

Jemalloc

• feat(jemalloc): support building third-parties images for both production and test with jemalloc #1143

Metrics

• feat(new_metrics): use rwlock instead of mutex to protect a metric entity or registry against few writes and numerous reads #1196
• feat(new_metrics): take snapshot of each metric as json format #1219
• feat(new_metrics): support filters for selected metric fields in response #1237
• feat(new_metrics): take snapshot of each entity as json format #1258
• feat(new_metrics): traverse the whole registry to choose entities and metrics according to the filters #1285
• feat(new_metrics): provide http service to process client query for metrics #1298
• feat(new_metrics): support frequently-used combinations of metric fields in the query string #1302
• feat(new_metrics): retire stale metric entities that are not used by any other object #1304
• feat(util): add cancel() for class defer #1458

Building

• chore: update rocksdb install url on thirdparty #1516
• build(thirdparty): bump fmtlib to 10.1.1 (cherry-pick #1605) #1640

Other

• feat: overload stream insertion operator(<<) to support formatting by {fmt} #1225
• feat: improve checking if C string is empty #1229
• feat(config): Make the configs take effect for the FLAGS_* only used once #1198
• feat: decide whether two C strings are equal and support CHECK_STREQ* and CHECK_STRNE* #1300
• feat(utils): add an utility function to find string prefix by separator #1338

Code Refactor

RDSN

• refactor(build): remove 'rdsn' from build and test commands #1099
• refactor(rdsn): remove 'include' directory in rdsn #1189
• refactor(rdsn): flatten source code directory by moving files from 'src/rdsn/src/*' to 'src/*' #1190

Java Client

• refactor(java-client): optimize code style #1087
• refactor(java-client): add more detail message for error log #1104
• refactor(java-client): optimize the code style of PegasusScanner #1130
• refactor(client-java): Use properties for depend #1137
• feat: make java client interface closable #1152
• build(pom): Use --release when building java client with java11+ #1256
• build(pom): Inherit apache pom #1260
• refactor: Remove assembly and output debug output to console when running UTs for java client #1266
• refactor: remove zkclient dependency #1264
• refactor: Use spotless plugin to format pom file #1271
• refactor: Remove usage of commons-configuration #1273
• build: Bump netty's version and shade it #1274
• build: Use maven flatten plugin to generate resolved pom #1277
• build: Support publish artifacts to repository.a.o #1278
• fix: jaas conf; make thread daemon #1393
• fix(java-client): Fix java spotbug #1416

Test

• refactor(function_test): use test_util to reduce duplicate code #1169
• refactor(test): remove useless upgrade_test #1171
• refactor: retry function test when run failed #1179

Logging

• refactor(log): adjust log level and rename for d*_f macros #1201
• refactor(log): adjust log level and rename for d*_replica/dup/rocksdb macros #1202
• fix: fix log header not corresponding to level #1246
• refactor(log): adjust default log level from DEBUG to INFO #1250
• refactor(log): use LOG_DEBUG_F instead of LOG_DEBUG #1306
• refactor(log): use LOG_INFO_F instead of LOG_INFO (1/3) #1307
• refactor(log): use LOG_INFO_F instead of LOG_INFO (2/3) #1310
• refactor(log): use LOG_INFO_F instead of LOG_INFO (3/3) #1311
• refactor(log): use LOG_FATAL_F instead of LOG_FATAL #1312
• refactor(log): use LOG_WARNING_F instead of LOG_WARNING (1/3) #1315
• refactor(log): use LOG_WARNING_F instead of LOG_WARNING (2/3) #1316
• refactor(log): use LOG_ERROR_F instead of LOG_ERROR (1/2) #1318
• refactor(log): use LOG_WARNING_F instead of LOG_WARNING (3/3) #1319
• refactor(log): use LOG_ERROR_F instead of LOG_ERROR (2/2) #1320
• refactor(log): rename LOG_*_F to LOG_* for all log macros #1322
• refactor(log): make macros LOG_AND_RETURN_NOT_* more general #1457
• fix(log): fix using of macro LOG_AND_RETURN_NOT_* print duplicate prefixes #1465

Macro

• refactor(macro): Use CHECK to replace dassert_f/dassert #1205
• refactor(macro): use CHECK to replace dassert_f/dassert (part2) #1211
• refactor(macro): use CHECK_EQ to replace dassert_eq* #1212
• refactor(macro): use CHECK_* to replace dassert_* #1214
• refactor(macro): use CHECK_* to replace dassert_* (part 2) #1217
• refactor(macro): use CHECK_* to replace dassert_* (part 3) #1226
• refactor(macro): use DCHECK* to replace dbg_dassert and self defined ASSERT_* #1228
• refactor(macro): use the more general CHECK_PREFIX* to replace dassert_replica #1230
• feat: support showing true expression in message for CHECK* macro #1253

IDL

• refactor(idl): unify the *.thrift files used by cpp and go-client #1284
• refactor(idl): unify the *.thrift files used by cpp and java-client #1290
• refactor(idl): unify the *.thrift files used by cpp and python-client #1317
• refactor(idl): unify the *.thrift files used by cpp and nodejs-client #1335

Flag/Config

• refactor(FLAG): remove the quotes of section field of FLAG #1308
• refactor(conf): use DSN_DEFINE_int32 to load int32 type of configs #1324
• refactor(conf): use DSN_DEFINE_int32 to load int32 type of configs (part 2) #1346
• refactor(conf): use DSN_DEFINE_uint32 to load uint32 type of configs #1352
• refactor(conf): use DSN_DEFINE_int64 to load int64 type of configs #1357
• refactor(conf): use DSN_DEFINE_uint64 to load uint64 type of configs #1359
• refactor(conf): use DSN_DEFINE_double to load double type of configs #1362
• refactor(conf): use DSN_DEFINE_bool to load bool type of configs #1363
• refactor(conf): use DSN_DEFINE_string to load string type of configs #1371

IWYU

• refactor(IWYU): introduce IWYU and fix issues #1354
• fix(IWYU): fix include errors for rpc_host_port reported by IWYU #1434
• chore(build): separate the build directory of IWYU version #1519

dir_node status

• refactor: update replica's dir_node status (part1) #1487
• refactor: update replica's dir_node status (part2) #1489

Metrics

• refactor(metrics): drop template specialization for gauge #1192

Other

• fix: fix wrong comments for down_cast #1194
• refactor: use linux_fd_t instead of dsn_handle_t for file descriptor #1238
• refactor: remove api_common.h and dlib.h header files #1235
• refactor: use state machine to improve split_args for strings #1240
• refactor: use command_deregister to auto deregister from command_manager #1241
• refactor: enable warning on unused result #1248
• refactor(rpc_address): simple refactor on rpc_address #1251
• refactor: let cmds deregister automatically #1287
• refactor: Move some functions from 'replica' to 'replica_stub' #1384
• refactor: Refactor the method 'check_leader' make it return a enum class type #1406
• refactor: minor refactor on replica module #1423
• feat: add new macro definitions & make the dockerfile compliant #1429
• doc(go-client): add deprecated marker to TableConnector.Close() #1448
• refactor: remove useless function 'last_flushed_decree()' #1456
• refactor: minor refactor on class fs_manager #1476
• refactor: remove some useless code #1480
• refactor: improve the single-responsibility of class fs_manager #1477
• refactor: remove counter_info and counter_info_ptr #1509

Bug Fix

• fix(go-client): scan will not recover when encounter error_invalid_state #1106
• fix: rocksdb options not changed even if update in Pegasus config file #1108
• fix(config): create stat table as required by cpp collector on bootstrap #1155
• fix: fix rollback by mistake for mutation_log::reset_from #1208
• fix: fix judgment to disk for partition split #1289
• fix: fix rocksdb build type #1340
• chore(cmake): fix project load failure by CLion #1376
• fix: resolve undeclared names that are reported while building alloc.h in debug mode #1392
• fix(java-client): add missing elements that will lead to misalignment and IndexOutOfBoundsException for resultMapList in batchGetByPartitions #1411
• refactor: return Status::code instead of meanless integer #1417
• fix: Fix the corruption RocksDB instance will be reused bug #1422
• fix: log error but not crash if found an imcomplete replica path #1428
• fix(ut): fix a flaky test integration_test.write_corrupt_db #1442
• fix(go-client): loopForRequest not return and retry forever #1444
• fix: fix a crash bug when open an incomplete created RocksDB instance #1451
• fix: Fault-tolerant storage engine errors for read operations #1447
• fix(ddl_client): sleep for a while before retry once meta server is busy #1453
• fix: fix garble words of time filed in logs #1468
• Fix write global seqno #1563
• fix(logs): fix logs in FATAL level not take effect #1573
• fix: support core dump for fatal logging in simple_logger::dsn_logv and call fatal logging exactly once for the assertion #1580
• fix: fix errors in format string for logging and macro #1588

Replica failed again and again due to corruption

• feat: skip IO_ERROR dir_node when assign replicas #1512
• refactor: improve the single-responsibility of class fs_manager (2/n) #1522
• feat(replica): close the replica and mark the dir_node as IO_ERROR when encounter read/write IO error #1473

Building

• fix: fix invaild build type #1521
• fix: fix build failure on older Ubuntu&GCC versions #1555
• fix(docker): fix pegasus-build-env of ubuntu:18.04 Dockerfile #1558
• fix(build): fix compilation error on MacOS for C++14 #1648
• fix(CI): Fix the "Lint Dockerfile" action failure (cherry-pick #1656) #1657

Scripts

• fix(shell): fix the additional options not applied bug of command "./run.sh test" #1526
• fix: fix start_onebox failed because of libjvm.so not found #1494

Unit Test

• feat(test): add bench type multi set/get #1373

CI/Workflows

• fix(actions): fix docker images conflict on different branches #1114
• fix(github): fix the conditions for file changes that trigger workflows #1123
• fix: update pegasus regular ci docker version base branch #1124
• fix: update pegasus clone branch when regular ci docker #1127
• fix(actions): build unit tests with portable rocksdb #1136
• fix(actinos): use portable thirdparties-bin-test-* as the base image to build and run unit tests for CI on github #1138
• chore(CI): Lint and build all projects regularlly #1142
• refactor(CI): enable and parallelize function test #1149
• fix: fix bad links and bad image names #1154
• fix(workflow): drop --on_travis option for unit tests of jemalloc #1161
• build(images): parallelize images building for all binary third-parties and only build ubuntu1804 for test images #1163
• build(images): download hadoop-2.8.4.tar.gz during third-parties image building for unit tests #1167
• ci: download hadoop-2.8.4.tar.gz during release building for unit tests once third-parties have changed #1168
• fix: fix workflow labeler not work #1182
• build(images): download apache-zookeeper-3.7.0-bin.tar.gz during third-parties image building for unit tests #1178
• chore(CI): disable some flaky tests #1187
• ci: download apache-zookeeper-3.7.0-bin.tar.gz during release building for unit tests once third-parties have change #1186
• fix(script): move zookeeper-bin under INSTALL_DIR rather than being it #1210
• chore(action): support checking issue number prefixed with '#' in github actions for PRs #1270
• chore(ci): use ubuntu-20.04 and install thrift-0.13 specially #1292
• fix(ci): fix build failure on ubuntu-16.04 #1295
• chore(CI): Add unit test workflow for python client #1314
• chore(CI): Add unit test workflow for node.js client #1332
• fix: fix centos image build #1356
• chore(CI): Use Ubuntu 22.04 as the default test platform #1381
• chore(CI): include cmath to fix ambiguous std::abs on ubuntu1604 #1395
• chore(CI): fix regularly build failure #1420
• chore(CI): unify to use Ubuntu 22.04 for CI (cherry-pick #1609) #1641
• fix(ASAN): avoid reporting false 'odr-violation' issues (cherry-pick #1611) #1646
• chore(CI): remove the "Check issue" from Standardization Lint (cherry-pick #1652) #1653
• fix: generate thrift sources for scala client (cherry-pick #1662) #1669

Free disk space

• chore(github): remove directories and tarballs after packing server and tools for github actions in case of running out of disk space #1486
• chore(github): remove CMakeFiles during Tar files in case of running out of disk space #1498
• fix(github): throttle_test failed frequently due to insufficient disk space #1530
• chore(github): free disk space for github workflows #1577
• chore(github): free disk space after packaging with jemalloc in github workflows #1581
• chore(github): free disk space for daily building on github #1586
• chore(github): remove release tarballs to free more disk space for unit tests #1643

Security

• fix: update commons-configuration2 version to 2.8.0 #1222

Doc

• docs: Update nodejs-client/README.md #1180
• fix: fix some bad links of docs #1282

License

• chore: update NOTICE year to 2022 #1119
• chore: update file paths in LICENSE #1121
• chore(license): check and fix *.txt files license #1172
• chore(license): add ignore file/path configs to .rat-excludes #1175
• chore: update NOTICE year to 2023 (cherry-pick #1671) #1672
• chore(license): add license checker for copyright and fix license files (cherry-pick #1674) #1677

Configuration Changes

There's no configuration removed by 2.5.0. All of the following configurations are added:

[network]
+ enable_udp = true
[metrics]
+ entity_retirement_delay_ms = 600000
[security]
+ enable_ranger_acl = false
[ddl_client]
+ ddl_client_max_attempt_count = 3
+ ddl_client_retry_interval_ms = 10000
[replication]
+ ignore_broken_disk = true
[ranger]
+ legacy_table_database_mapping_policy_name = __default__
[pegasus.server]
+ rocksdb_write_global_seqno = false
[replication]
+ crash_on_slog_error = false

All of the pull requests that are related to the added configurations:

• feat(server): disable UDP service by config #1132
• feat(new_metrics): retire stale metric entities that are not used by any other object #1304
• feat(Ranger): Compatible with the old ACL and the new ACL #1379
• fix(ddl_client): sleep for a while before retry once meta server is busy #1453
• refactor: improve the single-responsibility of class fs_manager #1477
• feat(Ranger): add a default database policy name for legacy table #1507
• Fix write global seqno #1563
• fix(slog): add an option to exit the process when find slog error #1574

Contributors

Thanks all of the contributors!

acelyc111
AlexNodex
Apache9
empiredan
foreverneverer
GehaFearless
liangyuanpeng
littlepangdi
ninsmiracle
padmejin
Praying
ruojieranyishen
shalk
Smityz
totalo
WHBANG