Native partitioning tokudb draft 5.7 #1

vlad-lesin · 2018-04-12T06:51:53Z

This PR is for preliminary code review and implementation details discussion.

…n_base.*

…ct' functionality. The general issue is that native partitioning initialization sequence is not the same as for non-native partitioning. Some functions are invoked before partition info is filled in, and, as a result, we can't initialize the array of partition file handlers. That is why the array initialization was moved to "open/create" functions, they are invoked when all necessary information is gathered from statement or *.frm file. The another issue was using the correct mem_root object for file handlers allocation. The initial code uses thd->mem_root, what is wrong, because it will be deallocated after query execution, and it worked only because thd->mem_root object was replaced with share->mem_root just before file handlers initialization. The current code uses share->mem_root explicitly.

Fix misc UBSAN warnings in unit tests. To repeat: export UBSAN_OPTIONS="print_stacktrace=1" ./runtime_output_directory/merge_large_tests-t --gtest_filter='-*DeathTest*' > /dev/null unittest/gunit/gis_algos-t.cc:78:70: runtime error: downcast of address 0x000012dc0be8 which does not point to an object of type 'Gis_polygon_ring' include/sql_string.h:683:35: runtime error: null pointer passed as argument 2, which is declared to never be null #1 0x373e7af in histograms::Value_map<String>::add_values(String const&, unsigned long long) sql/histograms/value_map.cc:149 percona#2 0x294fcf2 in dd_column_statistics_unittest::add_values(histograms::Value_map<String>&) unittest/gunit/dd_column_statistics-t.cc:62 runtime_output_directory/merge_keyring_file_tests-t --gtest_filter='-*DeathTest*' > /dev/null plugin/keyring/common/keyring_key.cc:82:57: runtime error: null pointer passed as argument 2, which is declared to never be null Change-Id: I2651362e3373244b72e6893f0e22e67402b49a52 (cherry picked from commit 1fe3f72561994da1d912a257689e1b18106f8828)

…_ROW()' FAILED Analysis: When a window with buffering follows a equijoin on a unique index (JT_EQ_REF) , we can get into trouble because windowing modifies the input record, presuming that once the windowing has been handed the record, next time control passes back to the join code a (new) record will be read to the input record. However, this does not hold with JT_EQ_REF, cf. the caching done in join_read_key: From its Doxygen: "Since the eq_ref access method will always return the same row, it is not necessary to read the row more than once, regardless of how many times it is needed in execution. This cache element is used when a row is needed after it has been read once, unless a key conversion error has occurred, or the cache has been disabled." Fix: We solve this problem by reinstating the input record before handing control back from end_write_wf. We optimize: only do this if the window in question follows after such a JOIN, i.e. window #1, and it has actually clobbered the input record. This can only happen if the last qep_tab has type JT_EQ_REF. Another, perhaps better approach, is to refactor to never touch the input record but keep the copying between the out record and the frame table record instead. Left for future refactoring. Added some missing Window method "const"s, and folded a couple of one-liners into window.h (from .cc). Repro added. Change-Id: I33bc43cd99ff79303b17d181abc3805ce226fb85

…TABLE_UPGRADE_GUARD To repeat: cmake -DWITH_ASAN=1 -DWITH_ASAN_SCOPE=1 ./mtr --mem --sanitize main.dd_upgrade_error A few dd tests fail with: ==26861==ERROR: AddressSanitizer: stack-use-after-scope on address 0x7000063bf5e8 at pc 0x00010d4dbe8b bp 0x7000063bda40 sp 0x7000063bda38 READ of size 8 at 0x7000063bf5e8 thread T2 #0 0x10d4dbe8a in Prealloced_array<st_plugin_int**, 16ul>::empty() const prealloced_array.h:186 #1 0x10d406a8b in lex_end(LEX*) sql_lex.cc:560 percona#2 0x10dae4b6d in dd::upgrade::Table_upgrade_guard::~Table_upgrade_guard() (mysqld:x86_64+0x100f87b6d) percona#3 0x10dadc557 in dd::upgrade::migrate_table_to_dd(THD*, std::__1::basic_string<char, std::__1::char_traits<char>, Stateless_allocator<char, dd::String_type_alloc, My_free_functor> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, Stateless_allocator<char, dd::String_type_alloc, My_free_functor> > const&, bool) (mysqld:x86_64+0x100f7f557) percona#4 0x10dad7e85 in dd::upgrade::migrate_plugin_table_to_dd(THD*) (mysqld:x86_64+0x100f7ae85) percona#5 0x10daec6a1 in dd::upgrade::do_pre_checks_and_initialize_dd(THD*) upgrade.cc:1216 percona#6 0x10cd0a5c0 in bootstrap::handle_bootstrap(void*) bootstrap.cc:336 Change-Id: I265ec6dd97ee8076aaf03763840c0cdf9e20325b Fix: increase lifetime of 'LEX lex;' which is used by 'table_guard'

AUTH_COMMON.H Description:- Sever crashes due to a NULL pointer de-reference. Analysis:- Sever encounters a NUll pointer de-reference during "acl_load()". Fix:- A check is introduced to avoid the NULL pointer de-reference. This issue is already prevented in 8.0 through Bug#27225806 fix. Therefore, this patch is applicable only for 5.7.

… work correctly for nes… Upstream commit ID : fb-mysql-5.6.35/6ddedd8f1e0ddcbc24e8f9a005636c5463799ab7 PS-4476 : Merge prod201801 Summary: …ted statements Variant #1: When the statement fails, we should roll back to the latest savepoint taken at the top level. Closes facebook/mysql-5.6#804 Differential Revision: D7509380 Pulled By: hermanlee fbshipit-source-id: 9a6f414

Use void* for function arguments, and cast Item_field* in function body. compare_fields_by_table_order(Item_field*, Item_field*, void*) through pointer to incorrect function type 'int (*)(void *, void *, void *)' sql/sql_optimizer.cc:3904: note: compare_fields_by_table_order(Item_field*, Item_field*, void*) defined here #0 0x34ac57d in base_list::sort(int (*)(void*, void*, void*), void*) sql/sql_list.h:278:13 #1 0x49ad303 in substitute_for_best_equal_field(Item*, COND_EQUAL*, Change-Id: I4f5417304a24201682f32fc7631034de7aa62589

…ENERATED_READ_FIELDS It's a SELECT with WHERE "(-1) minus 0x4d". this operation has a result type of "unsigned" (because 0x4d is unsigned integer) and the result (-78) doesn't fit int an unsigned type. This WHERE is evaluated by InnoDB in index condition pushdown: #0 my_error #1 Item_func::raise_numeric_overflow ... percona#7 Item_cond_and::val_int percona#8 innobase_index_cond ... percona#12 handler::index_read_map ... percona#15 handler::multi_range_read_next ... percona#20 rr_quick percona#21 join_init_read_record As val_int() has no "error" return code, the execution continues until frame percona#12; there we call update_generated_read_fields(), which has an assertion about thd->is_error() which fails. Fix: it would be nice to detect error as soon as it happens, i.e. in innodb code right after it calls val_bool(). But innodb's index condition pushdown functions only have found / not found return codes so they cannot signal "error" to the upper layers. Same is true for MyISAM. Moreover, "thd" isn't easily accessible there. Adding a detection a bit above in the stack (handler::* functions which do index reads) is also possible but would require fixing ~20 functions. The chosen fix here is to change update_generated_*_fields() to return error if thd->is_error() is true. Note that the removed assertion was already one cause of bug 27041382.

Group Replication does implement conflict detection on multi-primary to avoid write errors on parallel operations. The conflict detection is also engaged in single-primary mode on the particular case of primary change and the new primary still has a backlog to apply. Until the backlog is flushed, conflict detection is enabled to prevent write errors between the backlog and incoming transactions. The conflict detection data, which we name certification info, is also used to detected dependencies between accepted transactions, dependencies which will rule the transactions schedule on the parallel applier. In order to avoid that the certification info grows forever, periodically all members exchange their GTID_EXECUTED set, which full intersection will provide the set of transactions that are applied on all members. Future transactions cannot conflict with this set since all members are operating on top of it, so we can safely remove all write-sets from the certification info that do belong to those transactions. More details at WL#6833: Group Replication: Read-set free Certification Module (DBSM Snapshot Isolation). Though a corner case was found on which the garbage collection was purging more data than it should. The scenario is: 1) Group with 2 members; 2) Member1 executes: CREATE TABLE t1(a INT, b INT, PRIMARY KEY(a)); INSERT INTO t1 VALUE(1, 1); Both members have a GTID_EXECUTED= UUID:1-4 Both members certification info has: Hash of item in Writeset snapshot version (Gtid_set) #1 UUID1:1-4 3) member1 executes TA UPDATE t1 SET b=10 WHERE a=1; and blocks immediately before send the transaction to the group. This transaction has snapshot_version: UUID:1-4 4) member2 executes TB UPDATE t1 SET b=10 WHERE a=1; This transaction has snapshot_version: UUID:1-4 It goes through the complete patch and it is committed. This transaction has GTID: UUID:1000002 Both members have a GTID_EXECUTED= UUID:1-4:1000002 Both members certification info has: Hash of item in Writeset snapshot version (Gtid_set) #1 UUID1:1-4:1000002 5) member2 becomes extremely slow in processing transactions, we simulate that by holding the transaction queue to the GR pipeline. Transaction delivery is still working, but the transaction will be block before certification. 6) member1 is able to send its TA transaction, lets recall that this transaction has snapshot_version: UUID:1-4. On conflict detection on member1, it will conflict with #1, since this snapshot_version does not contain the snapshot_version of #1, that is TA was executed on a previous version than TB. On member2 the transaction will be delivered and will be put on hold before conflict detection. 7) meanwhile the certification info garbage collection kicks in. Both members have a GTID_EXECUTED= UUID:1-4:1000002 Its intersection is UUID:1-4:1000002 Both members certification info has: Hash of item in Writeset snapshot version (Gtid_set) #1 UUID1:1-4:1000002 The condition to purge write-sets is: snapshot_version.is_subset(intersection) We have "UUID:1-4:1000002".is_subset("UUID:1-4:1000002) which is true, so we remove #1. Both members certification info has: Hash of item in Writeset snapshot version (Gtid_set) <empty> 8) member2 gets back to normal, we release transaction TA, lets recall that this transaction has snapshot_version: UUID:1-4. On conflict detection, since the certification info is empty, the transaction will be allowed to proceed, which is incorrect, it must rollback (like on member1) since it conflicts with TB. The problem it is on certification garbage collection, more precisely on the condition used to purge data, we cannot leave the certification info empty otherwise this situation can happen. The condition must be changed to snapshot_version.is_subset_not_equals(intersection) which will always leave a placeholder to detect delayed conflicting transaction. So a trace of the solution is (starting on step 7): 7) meanwhile the certification info garbage collection kicks in. Both members have a GTID_EXECUTED= UUID:1-4:1000002 Its intersection is UUID:1-4:1000002 Both members certification info has: Hash of item in Writeset snapshot version (Gtid_set) #1 UUID1:1-4:1000002 The condition to purge write-sets is: snapshot_version.is_subset_not_equals(intersection) We have "UUID:1-4:1000002".is_subset_not_equals("UUID:1-4:1000002) which is false, so we do not remove #1. Both members certification info has: Hash of item in Writeset snapshot version (Gtid_set) #1 UUID1:1-4:1000002 8) member2 gets back to normal, we release transaction TA, lets recall that this transaction has snapshot_version: UUID:1-4. On conflict detection on member2, it will conflict with #1, since this snapshot_version does not contain the snapshot_version of #1, that is TA was executed on a previous version than TB. This is the same scenario that we see on this bug, though here the pipeline is being blocked by the distributed recovery procedure, that is, while the joining member is applying the missing data through the recovery channel, the incoming data is being queued. Meanwhile the certification info garbage collection kicks in and purges more data that it should, the result it is that conflicts are not being detected.

Source tree changes - Remove include/my_global.h from .gitignore - it should be visible, especially when merging from 5.7 (where it's present) to 8.0 (where it's not). This partially fixes bug https://bugs.mysql.com/bug.php?id=91834 (Cleanup .gitignore, e.g. include/my_global.h) - Remove storage/innobase/.clang-format due to upstream reformatting CMake / preprocessor symbol changes - Make CMake option WITH_SYSTEM_LIBS configure -DWITH_READLINE=system instead of -DWITH_EDITLINE=system - Remove WITH_SSL CMake settings from build scripts, leaving the implicit default of -DWITH_SSL=system, because the alternative -DWITH_SSL=wolfssl requires more work and is broken by upstream in 8.0.11. - Use MYSQL_ADD_EXECUTABLE instead of ADD_EXECUTABLE in CMake scripts. - Remove HAVE_REPLICATION from diff to match upstream [14]. - Remove WITH_EMBEDDED_SERVER, EMBEDDED_LIBRARY, NO_EMBEDDED_ACCESS_CHECKS, mysql-test/include/not_embedded.inc mentions due to upstream [15], [16], [24] etc. - Remove WITH_PARTITION_STORAGE_ENGINE due to upstream [32]. - Replace MYSQL_CLIENT preprocessor macro uses with !MYSQL_SERVER due to [22]. - Add sql/binlog_crypt_data.cc and sql/event_crypt.cc as mysqlbinlog source files in CMake, to replace the former practice of including them from source files. - Remove setting C++98/03 and C++11 detection from CMake due to upstream converting to C++11 in WL#8896. This includes removing cmake/check_stdcxx11.cmake and its uses from TokuDB / MyRocks. - Make CMake define __STDC_WANT_LIB_EXT1__=1 on macOS to get memset_s. - Define LOG_COMPONENT_TAG for various plugins and unit tests, to accomodate WL#11150 [19]. - Remove uses of UNIV_NONINL due to [35], consequently fold buf0dblwr.ic to buf0dblwr.h. - Revmove UNIV_DEBUG_THREAD_CREATION due to WL#9359 [36] Compilation changes - Fix now-enabled-by-default unused parameter warnings, by removing the unused parameters, or by marking them unused. C to C++ conversion - Convert mysys/system_key.c to C++ mysys/system_key.cc to match upstream [1]. - Convert storage/heap/hp_record.c and storage/heap/hp_dspace.c to C++ hp_record.cc and hp_dspace.cc respectively to match upstream [2]. - Mechanical renames, changes, and partial conversions, incomplete list: - my_bool/FALSE/TRUE/Sys_var_mybool uses to bool/false/true/Sys_var_bool due to upstream my_bool removal at [6]. - #define to [static] const constexpr - Declare local variables at first use - Pointer arguments to const reference ones, where applicable - Convert typedef struct declaration to plain struct, removed redundant struct tags. - NULL/0 to nullptr - compile_time_assert to static_assert - various ad-hoc macros to std::min - various integer types/0/1 to bool/false/true where applicable - Convert mysqldump.cc patches to C++11, remove uses of HASH, make key_type_t an enum class instead of enum. At the same time replace uses of LIST with std::list and std::forward_list, although this was not strictly required. - Remove C_MODE_START/C_MODE_END and extern "C" blocks from C++-only headers. C++03 to C++11 conversion - Replace HASH with std::unordered_map in many patches, because upstream removed HASH in series of commits for "25997748". - Mechanical renames and changes, incomplete list: - ut_snprintf uses to snprintf due to upstream removal of the former at [12]. - my_snprintf to snprintf due to C++11 guaranteeing the presence of the latter. - Applied noexcept to matching functions and methods. - Applied final to matching classes. - boost::movelib::unique_ptr to std::unique_ptr. - insert to emplace for standard containers, where applicable - converted some of std::map uses to std::unordered_map, if ordering is not required - std::auto_ptr to std::unique_ptr - Assignments in constructors and constructor initializer lists to default member initializers. - UNIV_THREAD_LOCAL to thread_local - Replaced uses of compiler builtins with std::atomic for atomic variables. User statistics - Due to HASH replacement with modern C++ collections, cleanup userstat to store objects directly instead of heap-allocated pointers to them. For this, add constructors to TABLE_STATS and INDEX_STATS to replace init_user_stats and init_thread_stats functions. Binlog encryption - Adjust binlog encryption patches for MYSQL_BIN_LOG "append" to "write" rename in cass methods. Parser - Put Percona tokens at the end of the token list due to upstream attempt to ensure token digest value stability ([17], [18]). - Because of upstream bottom-up parser refactoring for CREATE TABLE [23], which includes column attributes, rewrite TokuDB clustered index and InnoDB compression dictionary support: - add apply_zip_dict method to PT_column_attr_base class; - add zip_dict_name arg to the constructor and m_zip_dict_name field to PT_column_format_column_attr class, implement apply_zip_dict; - add m_zip_dict field to PT_field_def_base, get its value in contextualize_attrs method, and pass it to Alter_info instance in PT_column_def::contextualize and PT_alter_table_change_column::contextualize - rename PT_unique_key_column_attr to PT_unique_combo_clustering_key_column_attr, make it handle both UNIQUE and CLUSTERING - move zip_dict_name from LEX to HA_CREATE_INFO in line with the upstream LEX reduction in parser refactoring. - Typedef new alter tablespace option PT_alter_tablespace_option_encryption for general tablespace encryption. Performance Schema - Adjust PFS object declarations for the new volatility and documentation fields, account for PSI_FLAG_GLOBAL rename to PSI_FLAG_SINGLETON. - Move server PSF memory key declarations to sql/psi_memory_key.h and sql/psi_memory_key.cc - Adjust inline_mysql_unix_connect PFS instrumentation in include/mysql/psi/mysql_file.h to modern PFS instrumentation pattern which was introduced back in [3] because the old one stopped compiling in 8.0. While at that, switch from HAVE_PSI_INTERFACE to HAVE_PSI_FILE_INTERFACE. Audit log plugin - Converted the plugin to C++. - Made a lot of declarations in audit_log.cc static and/or const. - Converted hash tables in filter.cc to collation_unordered_set<std::string>. At the same time convert the hash table value struct creator/initialization functions to constructors. Backup locks - Remove everything associated with LOCK BINLOG FOR BACKUP / UNLOCK BINLOG due to upstream performance_schema.log_status table. - Refer to Oracle backup lock as "instance backup lock" and Percona backup lock as "table backup lock" to try to reduce confusion between the two. Expanded fast index creation - Change the type of delayed_key_list container in Alter_info from List to Mem_root_array. Threadpool - Re-implement TP_STATISTICS::num_worker_threads, pool_timer_t::current_microtime and next_timeout_check using C++11 atomics. - Convert Worker_thread_context struct to class with constructor/destructor instead of save/restore methods, to be used as RAII. New data dictionary - Percona features that need to store information in the new data dictionary (CLUSTERED keys for TokuDB; compression dictionary information for InnoDB have not been re-implemented for the new DD yet). For now the legacy implementation is disabled, to be removed together with the new implementation, except for the reading bits, because they are needed for upgrades from 5.7. Specifically, leave SYSTEM_TABLE_NAME patch in, dict_process_sys_zip_dict function Changed page tracking - Move fil_name_process patch not to cause any tablespace in filesystem changes when called by redo log tracking to fil_tablespace_redo_delete and fil_tablespace_redo_create. - Introduce apply arg for fil_tablespace_redo_encryption for redo log parsing mode only. - log_get_tracked_lsn and log_set_tracked_lsn have been remvoed after the tracked LSN conversion to C++11 atomic variable. - Fold log_check_tracking_margin to log0write.cc:log_write_write_buffer. - Move redo log signalling from log_io_complete_checkpoint to log0chkp.cc:log_checkpoint. - Drop log_group_read_log_seg release_mutex arg (which would have been moved to recv_read_log_seg) because of log_sys->mutex removal. - innobase_logs_empty_and_mark_at_shutdown patches moved to srv_log_shutdown. - Due to os_file_dir_t and associated function removals in [13], rewrite their users in log0online.cc to use MY_DIR and my_dir from mysys. - Remove log_group_t mentions from log0online.cc - Add handling for MLOG_COMP_PAGE_CREATE_SDI, MLOG_PAGE_CREATE_SDI, and MLOG_TABLE_DYNAMIC_META MTR types. - Adjust redo log reading code for log_group_read_log_seg rename to recv_read_log_seg, convert the latter from static to public function. - Refactor out redo log tracker-waiting code in fil0fil.cc to a new private static method Fil_system::wait_for_changed_page_tracker. Other changes due to upstream - Remove YaSSL-related source code (mysys_ssl/yassl.cpp, HAVE_YASSL check, build-binary.sh --with-yassl option) - Replace my_global.h includes with the needed smaller ones (my_inttypes.h, mysql/service_mysql_alloc.h, my_macros.h) due to upstream patch series "Bug #24710065: SPLIT MY_GLOBAL.H", and move my_global.h patches to appropriate places: s6_addr32 define to violite.h. - Remove THD::thr_create_utime uses and replace THD::update_server_status calls with THD::update_slow_query_status due to [30]. - Remove Sql_alloc uses due to [26]. - Adjust for class Key rename to Key_spec with declaration moved to sql/key_spec.h in [4]. - Remove NORMAL_PART_NAME uses because of [25]. - Break up the dependency between sql_class.h and handler.h by moving opt_log_slow_admin_statements and opt_log_slow_sp_statements from mysqld.h/mysqld.cc to log.h/log.cc, and including log.h instead of mysqld.h (which then includes handler.h) in sql_class.h. Also move struct fragmentation_stats_t declaration to system_variables.h from sql_thd_internal_api.h, put a forward declaration there, and do not include the latter header from the former. This is done due to [21]. - Remove THD::KILL_BAD_DATA uses due to [28]. - Remove THR_THD uses due to [29]. - Convert uses of dict_table_is_temporary to dict_table_t::is_temporary() due to [7]. - Remove TABLE_SHARE::uniques uses from sql_base.cc Table_Node::Table_Node due to upstream removal at [27]. It is not clear whether the resulting code is correct though. - Replace uses of SRV_LOG_SPACE_FIRST_ID with dict_sys_t::log_space_first_id due to [8] and the n dict_sys_t::s_log_space_first_id due to [9]. - Move srv_parallel_doublewrite_path initialisation to innobase_init_params to reflect InnoDB startup refactoring - Adjust btr_copy_externally_stored_field callers to call lob::btr_copy_externally_stored_field (WL#9141, [34]), and pass index definition to it (WL#8960, [33]). - Move dict_index_get_nth_col_pos patches to dict_index_t::get_col_pos because of [11]. - Adjust get-space-by-space-id code for fil_system mutex removal, sharding, and fil_node_complete_io -> complete_io rename, introduced in WL#8619 [10]. - Remove innodb_use_global_flush_log_at_trx_commit and commit innodb_flush_log_at_trx_commit back to global variable, and remove associated srv_use_global_flush_log_at_trx_commit source code variable and thd_flush_log_at_trx_commit function - Replace struct st_mysql_sys_var occurences with SYS_VAR due to [20]. - Adjust warn_or_info class for InnoDB logging changes at [5], which remove the actual logging from logger::~logger, pass diagnostics level to its constructor, and conditionally disable actual logging using UNIV_NO_ERR_MSGS preprocessor define. - Removed /*=...==*/ comments and replace /**...**//**\n with /** in the InnoDB patch according to [37]. - Replace reserved_system_space_name with dict_sys_t::sys_space_name because of [8], and then to dict_sys_t::s_sys_space_name because of [9]. Misc cleanups - Split assert(condition1 && condition2) to assert(condition1); assert(condition2); - Create type aliases for all the added function pointers in the handlerton interface. - Remove hash table size information leftovers that should have been removed in the 5.7 porting branch - Replace some of the ulint uses with space_id_t where applicable - Remove our #define SRV_MAX_N_PURGE_THREADS in favor of upstream MAX_PURGE_THREADS. - Remove remnants of innodb_flush_method = ALL_O_DIRECT: defines DEFAULT_SRV_LOG_WRITE_AHEAD_SIZE and MAX_SRV_LOG_WRITE_AHEAD_SIZE, xb_simulate_all_o_direct_write_failure, xb_simulate_all_o_direct_read_failure debug points. - Introduce flush_changed_page_bitmaps_t and purge_changed_page_bitmaps_t types for the corresponding handlerton methods. - Remove .is_percona_server_bzr - Replace README.MySQL contents with upstream README - Update man pages from mysql-8.0.11.tar.gz Testsuite These are partial changes only, as there was no goal to run the testsuite at this stage. - Remove uses of have_dynamic_loading.inc, have_symlink.inc, have_innodb.inc, have_perfschema.inc include files due to [31]. TokuDB, MyRocks, PAM authentication These three plugins are not handled in this merge yet. [1]: commit c7a720b Author: Jon Olav Hauglid <[email protected]> Date: Thu Sep 24 12:46:44 2015 +0100 Bug#21881278: SWITCH MYSYS LIBRARY FROM C TO C++ This patch switches all mysys/ files from .c to .cc. By doing this we get stricter compile checks (e.g. type checks) and it makes it possible to use C++ features for future bug fixes and extensions inside mysys. [2]: commit 4133e44 Author: Steinar H. Gunderson <[email protected]> Date: Thu Sep 7 15:18:39 2017 +0200 Bug #26781567: REMOVE C LEGACY [noclose] Convert the remaining C files to C++, except for external source code (e.g. memcached) and a few .c files that are there to verify that our client library still builds from C code. The only real exception is libservices/, where ODR violations (services are defined as structs in some files and as void * in others) mean that linking with C++ linkage would break linking on Windows. Change-Id: I61f6a281ffcd5c0fca9f4b9976aeb580dd1b0b65 [3]: commit bd8c99e Author: Marc Alff <[email protected]> Date: Fri Jul 1 10:46:20 2011 +0200 Bug#12346211 PERF IMPACT OF PERFORMANCE-SCHEMA WITH DISABLED INSTRUMENTS/CONSUMERS Performance improvement #1: instrumentation interface. In the instrumentation interface in include/mysql/psi/psi.h, implemented the following changes. 1) Removal of "if (PSI_server != NULL)". Instead of having PSI_server == NULL when the performance schema is disabled, a dummy implementation of the instrumentation interface is provided in mysys/psi_noop.cc. Replacing a test by calling an empty function is similar for the disabled code path. For the instrumented code path (the performance schema is enabled), removing this test completely improves performances. 2) Push down of "if (that->m_psi != NULL)". Testing if an object is instrumented has been pushed down to the implementation of the instrumentation interface. In case of psi_noop.cc, this test is not needed, which improves performances for the disabled case. 3) Removal of 1 "if (locker != NULL)" test. Before the fix, the coding pattern was: locker= ... if (locker != NULL) PSI_server->start_event() <instrumented code> if (locker != NULL) PSI_server->end_event() After the fix, the coding pattern is: locker= ... if (locker != NULL) { PSI_server->start_event() <instrumented code> PSI_server->end_event() } else { <instrumented code> } This saves an extra if() in both the instrumented and non instrumented code path. 4) Packing of helper structures. Unused attributes such as m_src_file and m_src_line have been removed. Attributes have been reordered to: - avoid holes caused by padding - put attributes used very frequently first 5) Use PSI_CALL() Calls to the performance schema implementation have been changed from: PSI_server->function(); to PSI_CALL(function)(); This is cosmetic only, and can help later to make static calls to pfs_function() instead of using the PSI_server pointer. [4]: commit 3f3c259 Author: Tor Didriksen <[email protected]> Date: Thu Mar 26 16:56:31 2015 +0100 Bug#20756282 MOVE CLASS KEY AND FOREIGN_KEY OUT OF SQL_CLASS.H Move Key_part_spec, Key_spec (new name for Key) and Foreign_key_spec (new name for Foreign_key) to new files key_spec.h/cc [5]: commit 29e3723 Author: Sunny Bains <[email protected]> Date: Fri Feb 16 14:30:31 2018 +1100 WL#11009: Logging services: log writers: add error ID to traditional error log (InnoDB changes) Add a unique ID to all log InnoDB messages. Remove InnoDB: prefix from the pattern matching in the tests. RB#18486 Reviewed by: Ingo, Annamalai and Praveen Hulakund. [6]: commit 6c174bd Author: Steinar H. Gunderson <[email protected]> Date: Tue Feb 21 13:39:24 2017 +0100 Bug #25597667: REMOVE MY_BOOL Replace my_bool with bool everywhere. Change-Id: Ibfe9f9b4a05d8ee82d09ffdc4f1f038ec4555b89 [7]: commit 2238b41 Author: El<C5><BC>bieta Babij <[email protected]> Date: Tue Jul 5 12:22:26 2016 +0200 Bug#23748128 - Add some accessor methods to dict_table_t. dict_table_t::get_n_user_cols(): Get the number of user-specified columns, possibly including a hidden FTS_DOC_ID column. dict_table_t::get_n_sys_cols(): Get the number of internal system columns. dict_table_t::get_n_cols(): Get the number of all columns. dict_table_t::get_col(): Get a column. dict_table_t::get_col_name(): Get a column name. dict_table_t::get_sys_col(): Get an internal system column. [8]: commit cd7649d Author: Allen Lai <[email protected]> Date: Thu Apr 13 10:59:58 2017 +0800 wl#9525 InnoDB_New_DD: update InnoDB specific metadata into new DD for DDL wl#9534 InnoDB_New_DD: Instantiate InnoDB in-memory metadata with newDD objects wl#9530 InnoDB_New_DD: FTS index support for newDD wl#9531 InnoDB_New_DD: Enable table encryption and transparent compression wl#9532 InnoDB_New_DD: InnoDB_New_DD: Move data dictionary tables to a single tablespace wl#9559 InnoDB_New_DD: Support in-place ALTER PARTITION wl#9537 InnoDB_New_DD: Support InnoDB table import/export for new DD wl#9357<C2><A0>InnoDB: Upgrade steps for new data dictionary wl#9570<C2><A0>Extend SHOW statements to list hidden columns and index information wl#9461<C2><A0>Upgrade for Innodb Internal Dictionary Migration This is the InnoDB side of work to support an unified metadata system between MySQL server and Storage Engines. It removes the InnoDB dependency on InnoDB system tables, that is, all InnoDB metadata are now read from server objects, instead of reading from InnoDB system tables. In these wls, we replaced dict_table_open_* (on_id/on_name) with their newDD counterpart. It replaced metadata instantiation (dict_load_table etc.) with newDD counterpart too. And also, we migrated all related features, like partition, encryption, compression and import/export etc. Upgrade from mysql-5.7 is supported as well. [9]: 3452648 Author: Jimmy Yang <[email protected]> AuthorDate: Thu Aug 17 00:34:24 2017 +0200 Commit: Jimmy Yang <[email protected]> CommitDate: Thu Aug 17 00:46:42 2017 +0200 Parent: f8076b3 Bug#25727892: Refactor Item::const_item() as a non-virtual function Merged: 5.5 5.6 mysql-5.7.7 Containing: 8.0 mysql-8.0.11-percona-patches mysql-8.0.3-percona-patches mysql-8.0.4-percona-patches (1 more) Follows: mysql-8.0.2 (1261) Precedes: mysql-8.0.3 (144) InnoDB NewDD worklogs(WL#9535 and WL#9536) to support Atomic DDL and completely remove old InnoDB system tables: 1. WL#9535: InnoDB_New_DD: Remove InnoDB System Table and modify the view of their I_S counterparts This worklog obsoletes all InnoDB System tables, which are no longer used and could only be accessed by upgrade. All metadata is now persisted and read from DD system tables in "mysql tablespace" instead of "old" InnoDB System Tables. All corresponding information_schema.innodb_sys_* tables are replaced by new internal system views. These new views have the name similar to the names of old system tables, just removing the 'SYS_' from the middle of the names. 2. WL#9536: InnoDB_New_DD: Support crash-safe DDL This worklog starts to support crash-safe DDL for InnoDB. DDLs like CREATE TABLE, DROP TABLE, RENAME TABLE, TRUNCATE TABLE, ALTER TABLE, CREATE TABLESPACE and DROP TABLESPACE are covered by this worklog. DDLs operations including DD system table updates and corresponding file changes(create, delete, rename etc.) are atomic. Physical data files would be synchronized with the transaction for DDL, thus there should be no physical data files for temporary tables or FTS auxiliary tables, etc. left after recovery. [10]: commit cf63b24 Author: Sunny Bains <[email protected]> Date: Fri Dec 8 12:10:42 2017 +1100 WL#8619 - InnoDB: Provide offline database portability without ISL files Summary of changes. 1. Introduce --innodb-directories := "dir1;...;dirN". This is renamed from --innodb-scan-directories which was introduced in WL#9499. Currently this is not dynamic. 2. Revert to 5.6 behaviour where we scan directories and find .ibd files. If you move tablespaces to a new location, add the path to innodb-data-directories so that InnoDB can "discover" the files on startup. 3. Tablespaces can only be created under known directories. This eliminates the need for .isl files. 4. Upgrade redo log format, not backward compatible 5. Change redo log type IDs 6. Code cleanup 7. Shard the fil system data structures for performance 8. Remove WL#9499 files that were used to track open tablespaces. RB#16842 Approved by: Jimmy and Kevin. [11]: commit b3ba94c Author: El<C5><BC>bieta Babij <[email protected]> Date: Tue Jul 26 15:00:04 2016 +0200 Bug #24361023 - Add accessors to indexes. dict_index_t::add_field(): Replaces dict_mem_index_add_field(). dict_index_t::get_field(): Replaces dict_index_get_nth_field(). dict_index_t::get_col(): Replaces dict_index_get_nth_col(). dict_index_t::get_col_no(): Replaces dict_index_get_nth_col_no(). dict_index_t::get_col_pos(): Replaces dict_index_get_nth_col_pos() and dict_index_get_nth_col_or_prefix_pos(). dict_index_t::get_sys_col_pos(): Replaces dict_index_get_sys_col_pos(). RB#13487 Reviewed by Kevin Lewis [12]: commit 98bba73 Author: El<C5><BC>bieta Babij <[email protected]> Date: Mon May 23 10:21:20 2016 +0200 Bug #23329353 - Replace ut_snprint with the C++11 snprintf function. [13]: commit 31350e8 Author: Gopal Shankar <[email protected]> Date: Tue Nov 3 13:27:06 2015 +0530 WL#6378 New data dictionary (umbrella). (...) [14]: commit f98b7bc Author: Jon Olav Hauglid <[email protected]> Date: Wed Jan 18 10:39:10 2017 +0100 Bug#25413703: REMOVE #IFDEF HAVE_REPLICATION After the removal of embedded server, the HAVE_REPLICATION ifdef is always true and can therefore be removed. [15]: commit 18c6270 Author: Tor Didriksen <[email protected]> Date: Tue Nov 8 09:55:35 2016 +0100 WL#9816 Do not build/package/test embedded server - Remove/Cleanup all cmake related code pertaining to embedded server - Remove/Cleanup embedded server from RMP and Debian packages - Remove embedded tests from our mtr test collections - Remove all source code in the libmysqld directory - Remove embedded code from mysql-test-run Change-Id: Ib2432e50043f060f1da64328815cdb9d53981de4 [16]: commit 0a68d5d Author: Jon Olav Hauglid <[email protected]> Date: Mon Jan 16 12:35:47 2017 +0100 Bug#25304844: REMOVE EMBEDDED_LIBRARY #IFDEF Remove dead code related to embedded server. Part 5: Remove EMBEDDED_LIBRARY ifdef from client/ and storage/ This patch also removes the --server-arg option from 'mysql'. This option was embedded server specific and undocumented. --server-arg and --server-file are, for the same reason, removed from 'mysqltest'. [17]: commit db3fd68 Author: Gleb Shchepa <[email protected]> Date: Thu Dec 7 16:21:28 2017 +0400 WL#8989: Create an internal API for the MySQL parser to enumerate reserved and non-reserved words Part I. Grammar preparation. * Replace the existent type tag <symbol> with <keyword> for a better readability. * Add the new type tag <keyword> to currently untyped terminal and non-terminal symbols. Part II. Create a simple compile-time program for transforming sql_yacc.yy into a .h file that contains an array of reserved and non-reserved words. * Find %token declarations with the <keyword> type tag by a regexp. * Separate %type declarations are ignored for the simplicity. Part III. Introduce a new system view: INFORMATION_SCHEMA.keywords. * Hardcode the statical list of keywords and reserved words into a view on top of a long JSON_TABLE(...) expression. * Use a trick in the .test file to trace changes in keywords: notify the developer to update the target I_S version in the mysqld binary where necessary -- this will force DD to refresh I_S view declarations on the next run. [18]: commit a2f3a60 Author: Marc Alff <[email protected]> Date: Thu Mar 10 10:10:25 2016 +0100 Bug#22906606 IMPROVE DIGEST STABILITY Before this fix, adding new tokens in the parser caused query digests to change. The root cause was that tokens are typically added in the middle of the list, causing renumbering of tokens. With this fix, the process is now changed and documented: new tokens should be added at the end of the list. The tool gen_lex_token, used during the build, is improved to enforce this process. [19]: commit f9d1c83 Author: Tatiana Azundris Nuernberg <[email protected]> Date: Sat Mar 3 13:31:37 2018 +0000 WL#11150: Add <component> to error log messages by default Adds another key/value pair to log events. The new field "subsystem" will contain one of the values "InnoDB" (for errors thrown in the InnoDB storage engine proper), "Repl" (for those thrown by the replication subsystem), or "Server" (otherwise). This value is also added to the traditional log. This field will appear between '[' <error_number> ']' and '[' <error_text> ']'. [20]: commit 545b168 Author: Steinar H. Gunderson <[email protected]> Date: Wed Sep 13 12:57:13 2017 +0200 Bug #26781567: REMOVE C LEGACY Replace the final typedefs to st_* and *_st structs. Note in particular that st_mysql_show_var was called sometimes SHOW_VAR and sometimes STATUS_VAR (by way of duplicated forward-declaring typedefs). This patch changes it so that it's universally SHOW_VAR, which seems to match better with the original type name. Change-Id: Ide483ceac31284d909c42adb46496308dd4d4f85 [21]: commit f58cb36 Author: Steinar H. Gunderson <[email protected]> Date: Tue Oct 10 14:54:53 2017 +0200 Bug #26927386: REDUCE WEIGHT OF SQL_CLASS.H [noclose] Remove field.h, handler.h and sql_profile.h from sql_class.h, and add a simple test to make sure they don't get back. Takes sql_class.h down from approx 1.0 to 0.85 seconds (GCC 7.2.0, -O2 -g). Change-Id: I72618c977c7af330d1215899971e2f9e49ab7b88 [22]: commit be029db Author: Jon Olav Hauglid <[email protected]> Date: Thu Jan 19 15:14:58 2017 +0100 Bug#25428042: REPLACE MYSQL_CLIENT IFDEF WITH !MYSQL_SERVER Replace the MYSQL_CLIENT #ifdef with !MYSQL_SERVER to simplify conditional compilation. Also remove unnecessary usage of MYSQL_SERVER and move setting of MYSQL_SERVER from C++ to CMake. [23]: commit f5c32ea Author: Gleb Shchepa <[email protected]> Date: Thu Jul 14 21:48:55 2016 +0400 WL#8067: True bottom-up server parser: refactoring of the CREATE TABLE statement WL#8434: True bottom-up server parser: refactoring of partitioning-related stuff WL#8435: True bottom-up server parser: cleanup and refactoring column definition stuff WL#8433: Separate DD commands from regular SQL queries in the parser grammar WL#7840: Allow parsing a single expression [24]: commit ed007c8 Author: Jon Olav Hauglid <[email protected]> Date: Thu Dec 22 11:42:43 2016 +0100 Bug#25304844: REMOVE EMBEDDED_LIBRARY #IFDEF Remove dead code related to embedded server. Part 3: Remove NO_EMBEDDED_ACCESS_CHECKS ifdef [25]: commit 11d6946 Author: Dmitry Lenev <[email protected]> Date: Fri Apr 21 09:52:09 2017 +0300 Removed old partitioning DDL implementation which is no longer in use. After WL#8971 "Deprecate and remove partition engine" and WL#9559 "InnoDB_New_DD: Support in-place ALTER PARTITION" code implementing "fast" partitioning DDL (fast_alter_partition_table() and related functions) became unused. This patch removes this dead code (including DDL_LOG implementation used by it) as a follow-up to the above WLs. [26]: commit 5fecdfa Author: Steinar H. Gunderson <[email protected]> Date: Thu Sep 21 14:40:20 2017 +0200 Bug #26030132: REMOVE SQL_ALLOC Finally remove Sql_alloc, now that 8.0 has been branched off. This is a no-op change for all existing code; there is no code that does new Foo() inheriting from Sql_alloc, since that overload doesn't exist, and similarly no code that deletes such objects, since operator delete is private. After this, it is possible to write such code, and it will do the obvious thing (allocate on the regular heap, and destroy similarly). As usual, code that wants to allocate on a MEM_ROOT must say so explicitly by use of overloaded new. Change-Id: I2e98b39daa06d6dfd8b8c5b1d45a85b1ddc02b3b [27]: commit 4880f97 Author: Guilhem Bichot <[email protected]> Date: Tue Nov 29 15:46:27 2016 +0100 WL#883 non-recursive CTE WL#3634 recursive CTE (...) [28]: commit 597a259 Author: Jon Olav Hauglid <[email protected]> Date: Tue Oct 6 14:04:03 2015 +0100 Bug#21944866: REFACTORING: REMOVE KILL_BAD_DATA Remove KILL_BAD_DATA from the THD::killed_state enum. KILL_BAD_DATA was only set by the connection itself when a warning was upgraded to error by STRICT mode. It therefore had nothing to do with the connection being killed by other connections (e.g. KILL QUERY / KILL CONNECTION). This is a refactoring with no changes in behavior. [29]: commit 2bf852b Author: Steinar H. Gunderson <[email protected]> Date: Wed Apr 12 12:26:14 2017 +0200 Bug #25909342: USE C++11 THREAD-LOCAL STORAGE FOR CURRENT_THD Our current implementation of current_thd uses mysys' my_thread_* API, which maps onto pthreads (at least for Linux/UNIX). C++11 supports native TLS, which means we just need two simple MOV instructions to do a TLS fetch instead of a full shared library call; the interface is also much simpler. We probably don't have a lot of performance gain, since we don't pick out current_thd all that often, but it's a nice cleanup to do nevertheless. There are still some uses of the my_thread_* API, particularly in PFS; cleaning this up is for a later patch. Change-Id: Ia16e0c1ccfa52af815882bcddd1eb3be80f77c61 [30]: commit 162e016 Author: Jon Olav Hauglid <[email protected]> Date: Tue Mar 15 14:32:39 2016 +0100 Bug#22948828: REFACTOR USAGE OF MY_MICRO_TIME() Remove redundant calls to my_micro_time() to simplify code and reduce the usage of gettimeofdat(). Especially focused on code for handling new connections. sql/conn_handler/connection_handler_one_thread.cc - Removed my_micro_time() call. THD::start_utime has just been set by THD::THD() => THD::init(). THD::thr_create_utime is only needed by connection_handler_per_thread.cc sql/conn_handler/connection_handler_per_thread.cc - Removed my_micro_time() call. THD::start_utime has just been set by THD::THD() => THD::init(). Remove THD::thr_create_utime, we can use THD::start_utime instead. sql/event_queue.cc - Replaced usage of THD::set_current_time() with THD::set_time(). THD::set_time() does the same as long as THD::user_time is NULL which it is for events. Allows the removal of THD::set_current_time() sql/events.cc - Removed THD::set_time() call. THD::set_time() has just been called by THD::THD() => THD::init(). sql/log.cc - Removed default argument value for make_iso8601_timestamp() for clarity. - Reduced critical section of LOCK_logger. - Call my_micro_time() directly instead of the unnecessary THD::current_utime() wrapper. sql/log_event.cc - Use the existing THD::query_start() function instead of accessing THD::start_time directly. - Renamed THD::update_server_status() to THD::check_slow_query() since the old name was somewhat misleading. sql/sql_class.h - Removed THD::thr_create_utime, no longer needed. - Removed redundant inline keyword, added const where possible. - Removed THD::query_start_usec(), not used. - Removed THD::query_start_timeval(), no longer needed. - Removed THD::set_current_time(), no longer needed. - Removed THD::is_valid_time(), code moved to sql_parse.cc - Renamed THD::update_server_status() to THD::check_slow_query() since the old name was somewhat misleading. sql/sql_connect.cc - Use THD::start_utime instead of removed THD::thd_create_utime (had the same value) - Removed call to THD::set_time(). Called by THD::init_for_queries() right afterwards anyway. sql/sql_thd_internal_api.cc - Removed THD::set_time() call. THD::set_time() has just been called by THD::THD() => THD::init(). - Removed call to my_micro_time(). Already done by THD::set_time(). [31] commit 485545b Author: Mohit Joshi <[email protected]> Date: Tue Jan 17 11:56:43 2017 +0530 Bug#25166686::REMOVE ALL THE REDUNDANT .INC FILES FROM MTR Description: It is noticed that there are a lot of redundant inc files which are not being used by any of the MTR tests. Also, there are several inc files which are included in the test but are not needed anymore. Fix: Below inc files are not being used by any test and hence removed: 1. include/check_key_reads.inc 2. include/check_key_req.inc 3. include/commandline_option_test.inc 4. include/config_file_option_test.inc 5. include/explain.inc 6. include/finish_option_test.inc 7. include/have_binlog_checksum_off.inc 8. include/have_dynamic_loading.inc 9. include/have_shm.inc 10. include/init_option_test.inc 11. include/not_blackhole.inc 12. include/show_partition_checksums.inc 13. include/show_partition_contents.inc 14. include/test_outfile.inc Below are a few inc files which are used in MTR tests but are no more needed. Hence these inc files along with their references from all MTR tests have been removed. 1. have_symlink.inc 2. have_perfschema.inc 3. have_innodb.inc Note: include/not_valgrind_server.inc is not being used by any MTR test but still not removed as this was introduced as part of Bug#22447330 Reviewed by: Parveez Baig <[email protected]> Deepa Dixit <[email protected]> [32] commit a0d0cab Author: Sivert Sorumgard <[email protected]> Date: Thu Jun 30 16:00:31 2016 +0200 WL#8971: Deprecate and remove partitioning storage engine This worklog implements the following main changes: 1. Modify cmake files, do not build partitioning engine. 2. Remove files implementing the partitioning engine. 3. Remove some mtr tests using non-native partitioning. 4. Rewrite some mtr tests to use native partitioning. Various adjustments necessary, e.g. regarding EXPLAIN output. 5. Introduce error handling at the SQL layer to fail if using partitioning related DDL statements for engines not supporting it natively. 6. Remove reference to the DB_TYPE_PARTITIONING_DB internally, but keep the enum entry to avoid disturbing other values. 7. Add an mtr test testing the error handling. 8. Add an mtr test to check for regressions in support for partitioning in ndb. 9. Copy comments from storage/partition/ha_partition.* to sql/handler.*, include/my_base.h etc. [33] commit eda9201 Author: Annamalai Gurusami <[email protected]> Date: Thu Apr 28 12:29:00 2016 +0530 WL#9141 InnoDB: Refactor uncompressed BLOB code to facilitate partial fetch/update (...) [34] commit eda9201 Author: Annamalai Gurusami <[email protected]> Date: Thu Apr 28 12:29:00 2016 +0530 WL#9141 InnoDB: Refactor uncompressed BLOB code to facilitate partial fetch/update (...) [35] commit 2eec7f3 Author: Marko M<C3><A4>kel<C3><A4> <[email protected]> Date: Fri Apr 22 10:50:45 2016 +0200 Bug#23150562 REMOVE UNIV_MUST_NOT_INLINE AND UNIV_NONINL Problem: InnoDB used to support a compilation mode that allowed to choose whether the function definitions in .ic files are to be inlined or not. This stopped making sense when InnoDB moved to C++ in MySQL 5.6 (and ha_innodb.cc started to #include .ic files), and more so in MySQL 5.7 when inline methods and functions were introduced in .h files. Solution: Remove all references to UNIV_NONINL and UNIV_MUST_NOT_INLINE from all files, assuming that the symbols are never defined. Remove the files fut0fut.cc and ut0byte.cc which only mattered when UNIV_NONINL was defined. RB: 12470 Reviewed-by: Jimmy Yang <[email protected]> [36]: commit 644aafd Author: Sunny Bains <[email protected]> Date: Tue May 24 19:02:55 2016 +0530 WL#9359 - Code cleanup [37]: commit 59b560e Author: Sunny Bains <[email protected]> Date: Tue Feb 6 22:00:10 2018 +1100 Bug#27496251 REMOVE /*====*/ AND CHANGE /****/ /***/ TO /** .... */

A subset of binlog encryption tests was crashing with: * thread percona#39, stop reason = signal SIGSTOP frame #0: 0x00007fff56063b66 libsystem_kernel.dylib`__pthread_kill + 10 frame #1: 0x00007fff5622e080 libsystem_pthread.dylib`pthread_kill + 333 frame percona#2: 0x000000010657442b mysqld-debug`my_write_core(sig=11) at stacktrace.cc:278 frame percona#3: 0x0000000104d84334 mysqld-debug`::handle_fatal_signal(sig=11) at signal_handler.cc:254 frame percona#4: 0x00007fff56221f5a libsystem_platform.dylib`_sigtramp + 26 frame percona#5: 0x00007fff5622934d libsystem_pthread.dylib`pthread_mutex_lock + 1 frame percona#6: 0x0000000106578d05 mysqld-debug`native_mutex_lock(mutex=0x0000000000000000) at thr_mutex.h:93 frame percona#7: 0x0000000106578a57 mysqld-debug`safe_mutex_lock(mp=0x0000000000000000, try_lock=false, file="/Users/laurynas/percona/mysql-server/mysys/mf_iocache2.cc", line=113) at thr_mutex.cc:70 frame percona#8: 0x000000010653cd3a mysqld-debug`my_mutex_lock(mp=0x00007ffb6b215038, file="/Users/laurynas/percona/mysql-server/mysys/mf_iocache2.cc", line=113) at thr_mutex.h:180 frame percona#9: 0x000000010653b2cc mysqld-debug`inline_mysql_mutex_lock(that=0x00007ffb6b215038, src_file="/Users/laurynas/percona/mysql-server/mysys/mf_iocache2.cc", src_line=113) at mysql_mutex.h:267 * frame percona#10: 0x000000010653b0d8 mysqld-debug`my_b_append_tell(info=0x00007ffb6b214fd8) at mf_iocache2.cc:113 frame percona#11: 0x0000000105ed6a96 mysqld-debug`MYSQL_BIN_LOG::write_buffer(this=0x00007ffb6b214cb8, buf="", len=47, mi=0x00007ffb6b1f6a00) at binlog.cc:7128 frame percona#12: 0x0000000105f4d54b mysqld-debug`queue_event(mi=0x00007ffb6b1f6a00, buf="", event_len=47, do_flush_mi=true) at rpl_slave.cc:7756 frame percona#13: 0x0000000105f3a243 mysqld-debug`::handle_slave_io(arg=0x00007ffb6b1f6a00) at rpl_slave.cc:5382 frame percona#14: 0x00000001065b87a5 mysqld-debug`pfs_spawn_thread(arg=0x00007ffb6a543af0) at pfs.cc:2836 frame percona#15: 0x00007fff5622b661 libsystem_pthread.dylib`_pthread_body + 340 frame percona#16: 0x00007fff5622b50d libsystem_pthread.dylib`_pthread_start + 377 frame percona#17: 0x00007fff5622abf9 libsystem_pthread.dylib`thread_start + 13 This was caused by my_b_append_tell trying to lock a nullptr IO_CACHE::append_buffer_lock. The lock was nullptr, because it's only initialized for SEQ_READ_APPEND IO_CACHEs, whereas we have WRITE_CACHE. This mismatch was introduced by WL#8599 [1] changing the IO_CACHE type from the former to the latter. Fix by using the correct API for the new IO_CACHE type: my_b_tell instead of my_b_append_tell. [1]: commit dbd2ca2 Author: Joao Gramacho <[email protected]> Date: Tue Nov 1 06:45:39 2016 +0000 WL#8599: Reduce contention in IO and SQL threads (...)

create_table_info_t::create_table_def leaked memory in the case enable_encryption(table) call failed: worker[5] Sanitizer report from /tmp/results/PS/mysql-test/var/5/log/mysqld.2.err after tests: binlog_encryption.binlog_encryption_without_keyring group_replication.gr_change_master_hidden group_replication.gr_server_uuid_matches_group_name group_replication.gr_perfschema_connect_status group_replication.gr_single_primary_and_leader_election_on_error group_replication.gr_without_perfschema rpl.rpl_key_rotation -------------------------------------------------------------------------- ==14131==ERROR: LeakSanitizer: detected memory leaks Direct leak of 1136 byte(s) in 1 object(s) allocated from: #0 0x7fe9233f1602 in malloc (/usr/lib/x86_64-linux-gnu/libasan.so.2+0x98602) #1 0xc692483 in ut_allocator<unsigned char>::allocate(unsigned long, unsigned char const*, unsigned int, bool, bool) storage/innobase/include/ut0new.h:608 percona#2 0xc692483 in mem_heap_create_block_func(mem_block_info_t*, unsigned long, unsigned long) storage/innobase/mem/memory.cc:281 percona#3 0xb99ff96 in mem_heap_create_func storage/innobase/include/mem0mem.ic:464 percona#4 0xbae8604 in create_table_info_t::create_table_def(dd::Table const*) storage/innobase/handler/ha_innodb.cc:10349 percona#5 0xbaee018 in create_table_info_t::create_table(dd::Table const*) storage/innobase/handler/ha_innodb.cc:12420 percona#6 0xbaf1aba in int innobase_basic_ddl::create_impl<dd::Table>(THD*, char const*, TABLE*, HA_CREATE_INFO*, dd::Table*, bool, bool, bool, unsigned long, unsigned long) storage/innobase/handler/ha_innodb.cc:12805 percona#7 0xbaf7e6a in ha_innobase::create(char const*, TABLE*, HA_CREATE_INFO*, dd::Table*) storage/innobase/handler/ha_innodb.cc:13756 percona#8 0x2857f7a in ha_create_table(THD*, char const*, char const*, char const*, HA_CREATE_INFO*, List<Create_field> const*, bool, bool, dd::Table*) sql/handler.cc:5156 percona#9 0x19d0d9f in rea_create_base_table sql/sql_table.cc:991 percona#10 0x19d0d9f in create_table_impl sql/sql_table.cc:7118 percona#11 0x19d37cf in mysql_create_table_no_lock(THD*, char const*, char const*, HA_CREATE_INFO*, Alter_info*, unsigned int, bool, bool*, handlerton**) sql/sql_table.cc:7200 percona#12 0x19dffb2 in mysql_create_table(THD*, TABLE_LIST*, HA_CREATE_INFO*, Alter_info*) sql/sql_table.cc:7950 percona#13 0x3b58b9b in Sql_cmd_create_table::execute(THD*) sql/sql_cmd_ddl_table.cc:319 percona#14 0x15917c1 in mysql_execute_command(THD*, bool) sql/sql_parse.cc:4417 percona#15 0x15b086e in mysql_parse(THD*, Parser_state*, bool) sql/sql_parse.cc:5139 percona#16 0x8efc7fd in Query_log_event::do_apply_event(Relay_log_info const*, char const*, unsigned long) sql/log_event.cc:5295 percona#17 0x8f7ea48 in Log_event::apply_event(Relay_log_info*) sql/log_event.cc:3882 percona#18 0x91cb682 in apply_event_and_update_pos sql/rpl_slave.cc:4352 percona#19 0x9215e69 in exec_relay_log_event sql/rpl_slave.cc:4812 percona#20 0x9254685 in handle_slave_sql sql/rpl_slave.cc:6912 percona#21 0xb1913a3 in pfs_spawn_thread storage/perfschema/pfs.cc:2836 percona#22 0x7fe9231436b9 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76b9) Fix by adding the missing mem_heap_free(heap) call.

Avoid undefined behavior in audit_log_update_thd_local by avoiding passing NULL as source pointer to memcpy, even with zero length. The UBSan report fixed is /usr/include/x86_64-linux-gnu/bits/string3.h:53:71: runtime error: null pointer passed as argument 2, which is declared to never be null #0 0x7fe5aad56fb1 in memcpy /usr/include/x86_64-linux-gnu/bits/string3.h:53 #1 0x7fe5aad56fb1 in audit_log_update_thd_local plugin/audit_log/audit_log.cc:987 percona#2 0x7fe5aad56fb1 in audit_log_notify plugin/audit_log/audit_log.cc:1105 percona#3 0x1ecac37 in plugins_dispatch sql/sql_audit.cc:1284 percona#4 0x1ecac37 in event_class_dispatch sql/sql_audit.cc:1322 percona#5 0x1ecb311 in event_class_dispatch_error sql/sql_audit.cc:1340 percona#6 0x1ed21b1 in mysql_audit_notify(THD*, mysql_event_connection_subclass_t, char const*, int) sql/sql_audit.cc:438 percona#7 0x1350071 in check_connection sql/sql_connect.cc:868 percona#8 0x1350071 in login_connection sql/sql_connect.cc:929 percona#9 0x1357881 in thd_prepare_connection(THD*, bool) sql/sql_connect.cc:1084 percona#10 0x1e66347 in handle_connection sql/conn_handler/connection_handler_per_thread.cc:313 percona#11 0xb1913a3 in pfs_spawn_thread storage/perfschema/pfs.cc:2836 percona#12 0x7fe5d352f6b9 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76b9) percona#13 0x7fe5d0bd741c in clone (/lib/x86_64-linux-gnu/libc.so.6+0x10741c)

vlad-lesin added 3 commits April 12, 2018 02:20

Just copy storage/partition/ha_partition.* to storage/tokudb/partitio…

f67d5b2

…n_base.*

Rename classes in partition_base.*

7fe6588

vlad-lesin closed this Apr 12, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Native partitioning tokudb draft 5.7 #1

Native partitioning tokudb draft 5.7 #1

vlad-lesin commented Apr 12, 2018

Native partitioning tokudb draft 5.7 #1

Native partitioning tokudb draft 5.7 #1

Conversation

vlad-lesin commented Apr 12, 2018