[Feature] Add inlist2subquery rewrite rule#39
Open
TiNnNnnn wants to merge 602 commits into
Open
Conversation
…n result. Problem: ======== If the outer table in a nested loop join is a const table, the inner table will be selected as a parallel table (Issue #1021). But if the join type is Left Join, the inner table cannot be a parallel table, because when the inner table scans the slice without corresponding data, a join record will also be generated. Solution: ========= Don't select the inner table as parallel table in nested left join. In addition, nested anti join also has the same problem. Nested semi join is rarely used, and this rule is also used to treat Nested semi join temporarily. Only when the join type is Inner Join, the inner table can be select as the parallel table. TODO: If the join type is not Inner Join, although the Nested join itself cannot be parallelized, the inner branch can still be parallelized. But exchange injection not support this condition currently. MR!1057, reviewed by kenkkchen, maxmqiu.
…ix query_string) Problem: It is reasonable to consider the original query text as an optional part of the optimization context, because THD::m_query_string could be rewritten. However, the original query text was not intercepted correctly, as the context was actually set by THD::reset_for_next_command() at the beginning of mysql_parse(). Besides, the default db was implicit. Solution: Move the interception site after THD::reset_for_next_command(), and add an optional parameter for mysql_parse() to tell if the invocation is interceptable. Add the default db as an explicit part of the context. MR!1086 reviewed by maxmqiu, hcduan part3 set_query set_db
…tor dbug) Problem: DBUG session is thread local. The part3 commits had added it (Opt_dbug_session) as a part of the optimization context. However, if the optimization context was used as a command argument, the processing thread would not be able to set up its DBUG session properly before the command. Solution: Move the DBUG session to THD::opt_dbug_session, and make it referenced rather than owned by the recorder's optimization context. MR!1086 maxmqiu, hcduan, xingyuyang
1. Extract out px_explain_init(). 2. Extract out and replace px_partition(). 3. Extract out px_run_root() and px_run_task(). 4. Consider THD::px_trx as part of optimization context. 5. Use PX_ROLE_ macros instead of THD::m_is_worker. 6. Add PX_exchange_info::m_scan_ctx for senders. 7. Fix const modifiers: - TableRowIterator::get_qep_tab() - QEP_TAB::set_parallel_scan() - QEP_TAB::set_parallel_workers() MR!1089 reviewed by maxmqiu, franzcheng, casazhang
Problem: ======= Issue #430 introduces partitioning of parallel tables for parallel queries. Parallel table partitioning can be mainly divided into three processes: 1) Construct parallel table partition boundary tuple 2) First partition based on partition boundary tuple 3) Adaptive Repartitioning Abnormalities may occur in these three processes, so it is necessary to enhance the abnormal scenario testing during partitioning Solution: ======= Added 6 exception scenarios to the three processes of parallel table partitioning, and completed the testing and processing of exceptions for each process MR!1139 reviewed by kenkkchen, maxmqiu
…taken into account when comparing the columns Problem: ======== The plan will be divided at the aggregation, if a sorting operator serves as the key order enforcer of a stream aggregation (issue #760). However, the order of the group and sort columns is reversed, which is not taken into account when checking the equivalence of the columns. Solution: ========= Reverse the group fields before the equivalence check. MR!1098, Reviewed by kenkkchen, maxmqiu
Problem:
========
System variables are not uniformly named.
Solution:
=========
Unified naming of system variables:
1) pre sys vars:
- px_max_parallel_threads (global)
- px_parallel_degree
- px_parallel_table_record_threshold
- px_parallel_cost_threshold
- px_exchange_buffer_size
- px_fallback_in_execution (tencent var, for testing)
- innodb_px_partitions_per_worker
new sys vars:
- txsql_max_parallel_worker_threads (global)
- txsql_parallel_degree
- txsql_parallel_table_record_threshold
- txsql_parallel_cost_threshold
- txsql_parallel_exchange_buffer_size
- txsql_parallel_fallback_in_execution (tencent var, for testing)
- innodb_txsql_parallel_partitions_per_worker
2) pre status:
- PX_stmt_executed
- PX_stmt_fallback
- PX_stmt_error
- PX_thread_current
new status:
- Txsql_parallel_stmt_executed
- Txsql_parallel_stmt_fallback
- Txsql_parallel_stmt_error
- Txsql_parallel_threads_currently_used
MR!1103 reviewed by kenkkchen, maxmqiu
Problem: ======= Materialize and Append obtain data through items_to_copy and copy_fields of tmp_table_param. FixAccessPathForUnion does not clear copy_fields when fixing the connection to Materialize or Append. In this case, invalid read will occur when these original items in copy_fields read data from the freed temporary table. Solution: ======== Clear copy_fields before fixing the connection to Materialize and Append. MR!1101 reviewed by kenkkchen, maxmqiu
Problem: ======= Issue #445 introduces the implementation of the exchange operator. PX_sender is the base class of the exchange operator producer, and its main process is as follows: 1) PX_sender::Init, register the producer to PX_exchange_info, and then attach to the corresponding exchange channels 2) Read from data source 3) encode data 4) The data is sent to the corresponding exchange channels 5) After sending, release resources and detach from exchange channels Solution: ======= Add the test case parallel_scan_exception.test, and add 4 possible abnormal test scenarios for the above process MR!1102 reviewed by kenkkchen maxmqiu
Problem: ======= Issue #445 introduced the implementation of the exchange operator, including the exchange operator without merge sort and with merge sort. Among them, the exchange operator with merge sort is defined as class PX_receiver_merge, which implements merge sort through a binary heap. The submission of Issue#445 was earlier than the submission of parallel query error exception handling. Therefore, it is necessary to adapt new error handling logic and fix the exception and return value handling of PX_receiver_merge. Solution: ======= Issue#801 introduces PX_io_error as the result of exchange channel data exchange; at the same time, PX_receiver_merge is regarded as a standard RowIterator class, therefore, the return value of all functions should be output according to the standard RowIterator: 0 means success -1 for EOF 1 means an error occurred This patch processes the return values of all functions and completes the adaptation of this set of rules MR!1076 reviewed by kenkkchen maxmqiu
Problem: ======= Currently, if an error occurs during a parallel execution (including parallel optimization), it will fall back to serial execution. In the process of splitting Aggregate operators, some errors were not passed out in the way of return, which would lead to crash in some cases. Solution: ======== Adjust the structure of some functions to handle error returns and check whether the memory was allocated successfully. MR!1070 reviewed by kenkkchen, maxmqiu
…funcs Problem: ======= There are two problems in the implementation of aggregation splitting code: (1) There are a lot of temporary table creation behaviors in this process, but this part of the code is not standardized. (2) JOIN::order will affect the memory allocation in the function alloc_func_list_with_param, but the order parameter will be cleared in some cases, which will cause wrong memory allocation and lead to invalid reads. In addition, function alloc_func_list_with_param is only used during Aggregation splitting, so it makes more sense to implement it as a static function. Solution: ======== (1) Wrap the creation of temporary table with the function CreateTmpTableForAgg, and set the parameters needed to create the temporary table in this function. (2) Use static function AllocSumFuncList to alloc memory for sum_funcs and use JOIN::saved_order instead of JOIN::order to calculate group_parts. (3) Adjust the order of static functions in a top-down manner. MR!1070 reviewed by kenkkchen, maxmqiu
Problem: ======= Issue #445 introduced the implementation of the exchange operator. PX_receiver is the base class for the consumer side of the exchange operator. Its main process can be divided into: 1) PX_receiver::Init, register the consumer to PX_exchange_info, and then attach to the corresponding exchange channels 2) PX_receiver::receive, read data based on exchange channel handle 3) PX_compact_codec::decode, decode the received data 4) After the dfo is completed, call the End function to detach from the exchange channel and release the created exchange channel handles For the above processes, enhance test coverage for possible exceptions Solution: ======= Add the test case parallel_receiver_exception.test, and add 5 possible abnormal test scenarios for the above process MR!1107 reviewed by kenkkchen maxmqiu
Problem: ======== The Ref slice number after Aggregate is set to REF_SLICE_SAVED_BASE, because agg's ref_slice is considered always REF_SLICE_ORDERED_GROUP_BY or REF_SLICE_TMP1 (for temp table Aggregate). But queries with ROLLUP or COUNT DISTINCT will create extra temp tables, which will result in a different number of ref slices in Aggregate. Solution: ========= Compute the ref slice number acoording to the slice number in Aggregate. Ref slice number saved in split position was tested to always be equal to output_slice, possibly using split_pos in the future. MR!1108, reviewed by kenkkchen, maxmqiu.
…execution on the server (postfix) Problem: ======= The testcase need tencentroot configuration. Solution: ======= Add parallel_execution_enabled-master.opt to configure. MR!1078 reviewed by kenkkchen, maxmqiu, hcduan
Problem: ======== Currently, splitting aggregate under Materialize is not supported, because the fix of the Materialize operator after splitting is not supported. Solution: ======== The fix of the Materialize is mainly to fix the temporary table used, because the splitting of the aggregation may change the number and position of items, so it is necessary to create a new temporary table to fix the corresponding relationship between item and field. It is worth noting that the fix requires special attention to one variable, PROP_AGGREGATION, which changes during optimization and needs to save its original value. MR!1019 reviewed by kenkkchen maxmqiu
Problem: ======= Issue#445 introduced the exchange operator. An exchange operator mainly includes three interfaces: 1) Init interface, responsible for initializing resources and attaching to exchange channels 2) Read interface, responsible for data exchange 3) End interface, responsible for resource cleaning and detaching from exchange channels In some cases, the receiver operator's End interface is not explicitly called, resulting in exchange resource leaks and no detach. Solution: ======= Register the pointer of the exchange operator in the corresponding THD. After the task runs, the End interface of the exchange operator is displayed through the THD to ensure that the resource release and the detach call must be completed. MR!1109 reviewed by kenkkchen maxmqiu
Problem: ========= The root DFO is run in the coordinator thread so as to send results to the client. The coordinator thread already exits, and should be excluded from worker thread requirement. Otherwise, as in previous implementation, one additional thread will be allocated and always idle. Solution: ========= Exclude root DFO when analyzing worker thread requirements. MR!1121 reviewed by kenkkchen maxmqiu
Problem: ======= Issue#1152 fixed the problem that when there is a limit operator on the exchange receiver operator, the End interface of the receiver operator operator may not be called. On the one hand, this problem causes memory leaks, and on the other hand, the exchange operator does not detach from the exchange channel, which may lead to hang. A branch of the union query may have a limit operator. This time, the receiver operatorof this branch is not detached, and there is a risk of hang. Solution: ======= Make sure that the exchange operator must call the End interface in time. 1) The exchange operator can sense its own end (EOF or exception), and the exchange operator calls End by itself 2) It is not the last exchange operator of a task and cannot sense the end by itself (there is a limit operator above), then in the Init function of the next exchange operator of the task, explicitly call the End interface of THD::px_receiver/px_sender before attaching 3) The last exchange operator of the task explicitly calls the End interface of THD::px_receiver/px_sender at the end of the task execution to ensure detach MR!1125 reviewed by kenkkchen maxmqiu
…be wrong Problem: ======== Full-text searching is performed using MATCH() AGAINST() syntax, which use ft_prebuilt to search columns in Full-text index. Even in serial plan slice, ft_prebuilt->fts_doc_id does not get updated correctly in parallel execution, which leads to wrong results. Solution: ========= Don't allow querys with the match functions to be executed in parallel. MR!1127, reviewed by kenkkchen, maxmqiu.
… (tests) Problem: ======== Some testcases failed due to the change of status variable names. Solution: ========= For show status like xxxx: 1) '%px%' => '%txsql_parallel%' 2) '%px_stmt_%' => '%txsql_parallel_stmt_%' By the way, we add tmy.cnf for testcases which need tencentroot. All the result of the testcase is sorted by '-sort-select-no-orderby'. Besides, prefer 'temp' to 'temporary' as table alias in CreateTmpTableForAgg() and BuildFinalAggregateAccessPath(). MR!1123 reviewed by kenkkchen maxmqiu casazhang
Problem: ========= In previous implementation, the state of a worker (PX_worker) destructs while certain parts might still be used by others. For example, PX_proc is used to detect liveness. Solution: ========= All workers should be uniformly destructed when the query execution is completed. By the way, kill worker case is removed, one unstable case is also fixed. MR!1120 reviewed by kenkkchen maxmqiu
Problem: ========= A new tmp table will be created for materialize when splitting agg under materialize, so that table of table_path in materialize should be updated to the new one. Solution: ========= Update table of table_path with the new tmp table. MR!1132 reviewed by kenkkchen maxmqiu
Problem:
Conditional jump or move depends on uninitialised value
Opt_ctx_clint::m_optimizing, which was used to tell statiststics
requests from TABLE_LIST::fetch_number_of_rows().
Solution:
Make sure it is intialized.
Verified by
./mtr parallel_scan_exception --valgrind
MR!1136 reviewed maxmqiu hcduan franzcheng
Problem: ======== px_execute_init() is used by the coordinator and each worker to set up their private execution environments (THD::px_executor). These env objects should not be destroyed until all workers reach their ends and get reclaimed by the coordinator, because sync vars (PX_proc) are part of them. However, in previous implementation, px_execute_init() destroyed env on worker with errors such as consistent plan, which was not only unsafe for synchronization but also causing double-free in reclaimation by the coordinator. Solution: ======== Remove the destroying of worker env in px_execute_init(). MR!1138 reviewed by kenkkchen maxmqiu
Problem: ========= Worker threads currently executing in parallel are not displayed in the process list, this leaves the worker thread in a black box. The user can only see that the coordinator thread is waiting, but does not know what the background worker thread is doing. Solution: ========= Add SHOW PARALLEL PROCESSLIST to view the status of all parallel threads, while SHOW PROCESSLIST shows regular threads and coordinators which also serve the user. This way provides maximum user interface compatiblity, however, it is inevitable that coordinators might show parallel specific state and info. Some notes about the fields of SHOW PARALLEL PROCESSLIST: - Id: show the id of coordinator and worker; - User/Host/Db: empty in worker thread show; - Command: Task or Wait in worker thread show; - Info: show worker number and task number executing; As a product requirement, killing worker threads directly is not allowed. When a user KILL is issued on a worker, the user will get the warning "Killing worker is not allowed, please kill query/connection <id>". MR!1104 reviewed by kenkkchen maxmqiu
Problem: ======= Parallel execution does not support prepared statement yet. However, the state of prepared statement detection (LEX::is_from_ps) is set by Prepared_statement::prepare() and reset by LEX::reset(), thus a LEX::reset() call subverts the state, e.g. when rewriter plugin is involved. Solution: ======= Add a new state LEX::is_executing_ps which is managed by Statement_backup for all prepared statement commands. MR!1137 reviewed by kenkkchen maxmqiu
Problem: ======= PX_sender, as an Iterator class, also maintains a standard three-state return value: 0: Iteration succeeded -1: End of iteration 1: iteration failed PX_sender judges the end of iteration in any of the two cases: 1) Read data from child (source) node and get EOF 2) Send data to exchange channels and get EOF from all of them (detached by receivers) Previous code considered first channel EOF as sender EOF, thus incorrect return value and loss of data for multiple receivers. It happened to be OK because the return value was ignored and there was only one receiver so far. Solution: ======= For the second case, generate sender EOF only when all channels are detached. MR!1140 reviewed by kenkkchen maxmqiu
Problem: ======= It is no need to check the full plan of workers with that of the coordinator. Only the sub-tree under the exchange operator needs to check. Solution: ======== The patch skips the equality check of accesspaths and items, which are only be executing in the coordinator. By the way, when testing the equality of item_sum, the field pointer of window function is different between worker threads and the coordinator thread. But it is ok, while the window function under the exchange operator, the false returning disables the window function in parallel execution. Test: mtr of rpl_window_functions & main.union MR!1139 revieded by kenkkchen, maxmqiu, happychang
… count Problem: ======= MySQL WL#11720 introduced parallel query, which supports unconditional count(*) query, but forces the use of primary key index even if secondary index is selected by the optimizer. For some tables, the primary key index data is very large, and the secondary index data is small. At this time, the primary key index under the large amount of data leads to a large amount of disk IO, Even with parallel queries, execution will be slower than a single-threaded secondary index scan. Also, TXSQL parallel query does not support insert Exchange operator on UnqualifiedCountAccessPath, an AccessPath for unqualified COUNT(*) (ie., no WHERE, no join conditions, etc.) Therefore, support is needed to force UnqualifiedCount not to be used, so that the secondary index chosen by the optimizer can be used, and it can go into TXSQL Parallel query logic. Solution: ======= Add a system variable txsql_pread_count_enabled, the default value is true. When set txsql_pread_count_enabled to false, the creation of UnqualifiedCountAccessPath will be skipped in the JOIN::create_root_access_path_for_join function. This patch is not specific to parallel execution. It is included because it could be useful in certain user cases. MR!1142 reviewed by kenkkchen maxmqiu
Additional patch for 8.0:
use ${CRYPTO_LIBRARY} rather than OpenSSL::Crypto
use WITH_AUTHENTICATION_FIDO rather than WITH_AUTHENTICATION_WEBAUTHN
Change-Id: Ief9209178077472a87758f3a909d2c6cda6b79f6
Remove old source files. Change-Id: I8414ff82a112aa3774ea67fe06a07e7a565270d3 (cherry picked from commit cc09b13fc9f6323f40726e1ee84ddb161474cf98) (cherry picked from commit add34ca7cde91ff4d64b357d8a52dea8caa9f752)
Additional patch for 8.0: Disable NFC on el6. Change-Id: Ic245ab9ad4b4cca8fcd503ef6da5a296751ba6a7
Additional patch to upstream code: Do not enable LTO locally for libcbor. The decision to use LTO or not is done in our top level CMakeLists.txt (we have actually seen build breaks on win calling check_ipo_supported) Change-Id: I2447d0ad9d26b10bbf60a313467fb202cc718da4 (cherry picked from commit 7f8c6340444bfd77d483c759e47881bf14670242) (cherry picked from commit 580099f89e2cb18bd92d6148dc96bbbe0baa5f77)
Update cmake code in fastcov.cmake to the most recent fastcov.py Improve warning messages in case fastcov is not found, or gcc is too old. The 3rd party library duktape has been upgraded, change cmake code to reflect it. When using non-default gcc (e.g. from /opt/rh/devtoolset...) extend the PATH when executing fastcov. Generated symlinks differ for Ninja and Unix Makefiles generators. Similarly for the dummy/empty duktape source files generated. Change-Id: I6c625382b1b6aee46ade8f922e855b609c6d72e7
Description: ======== In 8.0.30, we reuse the official api to judge if an index offsert can be cached in parallel ddl. However, this api is not judge null cols and only judge n_unique_in_tree columns, which is not suitable for reading a primary index that contains all columns. Solution: ======== We traverse all columns for primary key to judge whether they are with fixed length and whether they are nulls.
…sql_enable_copy_free_snapshot
Problem
=======
KUNPENG has different cache line padding.
Solution
========
padding with 128 on KUNPENG processor
origin work from lucianli
Description: ============ Many follower thread wait for the global Lock commit_stage_manager.m_lock_done and cause performance degradation. Use per-thread Lock instead of the global Lock to reduce lock contention.
… stmt twice mysqld crashes when a PREPARE statement which includes a GROUP BY on a constant with rollup is executed twice. The first execution handles this case correctly - it detects that the GROUP BY is on a constant and does not attempt to create a temporary table. After the temporary table check completes, it wraps the constant in a rollup wrapper so that the HAVING clause (if present) handles the rolled-up rows correctly. So after the first EXECUTE, the field list contains a rollup wrapper instead of a constant in the group-by list. The second execution finds that the group-by list contains a non-constant value and attempts to construct a temporary table for it. During temporary table construction, the constant is looked up in the field list. This causes a crash because the constant cannot be found in the field list since it was overwritten by a rollup wrapper. This is fixed by cleaning up the rollup wrappers after every execution. All rollup wrappers are unwrapped to restore the field list to its original state. For hypergraph, a cleanup fix is added to restore the updated join fields instead of the original fields. Rollup wrapper creation for ORDER-BY items is postponed from the PREPARE phase to the EXECUTE phase. To handle comparisons of rollup wrappers with items, FIELD_ITEM and ITEM_FUNC comparisons are modified to compare unwrapped rollup items. Change-Id: I410b8799647dd8e059707e92a956d060f3cd75e5
Description: ======== Introduce a new feature in the database user login authentication phase to support using the SM3 hash algorithm, replacing existing algorithms such as SHA1 and SHA2, thereby enhancing security. Main things: ======== 1. When creating a user, specify the plugin and declare the use of the SM3 hash algorithm during login authentication. Below is an example: CREATE USER <username>[@<hostname>] IDENTIFIED WITH 'txsql_sm3_password' BY '<Your Password>'; 2. Two functions are provided: SM3(<data>) to generate digest values. SM3_PASSWORD(<data>) to generate password digests. 3. You need using TXSQL's client/library to connect with sm3 hash user's By Weng Haixing <wenghaixing@unionpay.com>
Description: ======== Introduce a new feature than user can disable or enable DDL operation, controled by a new globle variable 'txsql_disable_ddl', enhancing security. Main things: ======== 1. The new variable 'txsql_disable_dll' is globle, you can't set it in session. 2. When 'txsql_disable_ddl' is set ON, means all DDL operations are prohibited, like CREATE-TABLE/ALTER-TABLE command. IF you want to restore, please set it OFF. By Pang Tianze <pangtianze@unionpay.com>
Description: ======== Introduce a new feature that monitoring sensitive database operations, including modify table schemas, variables, or configuration items. Related database operations will be recorded in the error log. Main things: ======== 1. Two new global variables, "txsql_audit_alter_table_enable" and "txsql_audit_set_option_enable", decide whether monitor sensitive operations or not. 2. When txsql_audit_alter_table_enable is ON, means monitoring "modify table scheams" operation; and when txsql_audit_set_option_enable is ON, means monitoring "modify variables, config" oeration. 3. The monitor log record in error log file is on INFO level, you need set log_error_verbosity=3 to check them. By Pang Tianze <pangtianze@unionpay.com>
Description: ======== This feature introduces client-side load balancing, allowing the client to automatically select one of the configured instances when establishing a connection. Main things: ======== 1.This feature allows users to connect to the database using a command such as: mysql -u$user -hloadbalance://<ip1>:<port1>,<ip2>:<port2>... -p The client automatically selects one of the specified instances to establish the connection. 2.Alternatively, users can enable load balancing by exporting the environment variable TXSQL_LOAD_BALANCE_<ip>_<port> as follows: export TXSQL_LOAD_BALANCE_<ip>_<port>=loadbalance://<ip1>:<port1>,<ip2>:<port2>... When connecting to the database using the corresponding <ip>:<port>, the client automatically selects one of the configured instances to establish the connection. By Cheng Bohan <chengbohan@unionpay.com>
Description: ======== Support gb18030 2022 character set. Main things: ======== 1. Add parameter txsql_gb18030_charset_standard to specify the original gb18030 character set to 2005 or 2022 standard. 2. Implement gb18030 2022 character set standard. By An Rungong <anrungong@unionpay.com> Co-authored-by: willhan <526786050@qq.com>
… statistics srv_update_cpu_usage() (Linux path) doesn't look cpu affinity bits over 128. It causes ignoring cpu counts for processor id over 128. This fix changes to look all available cpu affinity bits by sched_getaffinity() FYI, Windows path is still limited to 64, because GetProcessAffinityMask() API can treat 64 bit only. Change-Id: I904938b19c5ddc68c2e9150eb42669b6b3e88a80
|
|
Collaborator
|
补充下相关测试用例 |
flyfox20
force-pushed
the
8.0.30
branch
2 times, most recently
from
120155a to
e2abf4f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
15 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Add inlist2subquery rewrite rule.
Control parameters
two parameters were added:
simple example:
After enable inlist2subquery rewrite rule:
Performance Test Under Different size of in-list