[improvement](storage) For debugging problems: add session variable to treat agg and unique data model as dup model

[jacktengg]

Proposed changes

Issue Number: close #xxx

Problem summary

For debug purpose:
Add session variable skip_storage_engine_merge, when set to true, tables of aggregate key model and unique key model will be read as duplicate key model.
Add session variable skip_delete_predicate, when set to true, rows deleted with delete statement will be selected.

Test result( withc auto compaction off):

create test table and insert data:

drop table if exists t1;
create table t1(a int, b int) engine = olap unique key (a) distributed by hash(a) buckets 16 properties("replication_allocation" = "tag.location.default:1");

insert into t1 values (1,1),(2,1);
insert into t1 values (1,11),(2,11);
insert into t1 values (3,1);

initial session variables and data:

mysql> show variables like "%skip%";
+---------------------------+-------+
| Variable_name             | Value |
+---------------------------+-------+
| skip_delete_predicate     | false |
| skip_storage_engine_merge | false  |
+---------------------------+-------+
2 rows in set (0.00 sec)

mysql> show variables like "%hidd%";
+---------------------+-------+
| Variable_name       | Value |
+---------------------+-------+
| show_hidden_columns | false |
+---------------------+-------+
1 row in set (0.00 sec)

mysql> select * from t1;
+------+------+
| a    | b    |
+------+------+
|    1 |   11 |
|    2 |   11 |
|    3 |    1 |
+------+------+
3 rows in set (0.50 sec)

enable skip_storage_engine_merge and check select result, not merged original rows are returned:

mysql> set skip_storage_engine_merge = true;
Query OK, 0 rows affected (0.01 sec)

mysql> select * from t1;
+------+------+-----------------------+
| a    | b    | __DORIS_DELETE_SIGN__ |
+------+------+-----------------------+
|    3 |    1 |                     0 |
|    2 |    1 |                     0 |
|    2 |   11 |                     0 |
|    1 |    1 |                     0 |
|    1 |   11 |                     0 |
+------+------+-----------------------+
5 rows in set (0.04 sec)

turn off skip_storage_engine_merge

mysql> set skip_storage_engine_merge = false;

load delete and select again:
csv:

1|111

curl --location-trusted -uroot: -H "column_separator:|" -H "columns:a, b" -H "merge_type: delete" -T delete.csv http://127.0.0.1:8030/api/test_skip/t1/_stream_load

mysql> select * from t1;
+------+------+
| a    | b    |
+------+------+
|    3 |    1 |
|    2 |   11 |
+------+------+
2 rows in set (0.05 sec)

delete rows with a = 2:

mysql> delete from t1 where a = 2;
Query OK, 0 rows affected (0.09 sec)
{'label':'delete_3b7fc2c3-becb-4586-bc46-dc225274186d', 'status':'VISIBLE', 'txnId':'30049'}

mysql> select * from t1;
+------+------+
| a    | b    |
+------+------+
|    3 |    1 |
+------+------+
1 row in set (0.04 sec)

enable skip_storage_engine_merge and select, rows deleted with delete statement is not returned:

mysql> set skip_storage_engine_merge = true;
Query OK, 0 rows affected (0.00 sec)

mysql> select * from t1;
+------+------+-----------------------+
| a    | b    | __DORIS_DELETE_SIGN__ |
+------+------+-----------------------+
|    3 |    1 |                     0 |
|    1 |    1 |                     0 |
|    1 |   11 |                     0 |
|    1 |  111 |                     1 |
+------+------+-----------------------+
4 rows in set (0.04 sec)

enable skip_delete_predicate, rows deleted with delete statement is also returned:

mysql> set skip_delete_predicate = true;
Query OK, 0 rows affected (0.00 sec)

mysql> select * from t1;
+------+------+-----------------------+
| a    | b    | __DORIS_DELETE_SIGN__ |
+------+------+-----------------------+
|    3 |    1 |                     0 |
|    2 |    1 |                     0 |
|    2 |   11 |                     0 |
|    1 |    1 |                     0 |
|    1 |   11 |                     0 |
|    1 |  111 |                     1 |
+------+------+-----------------------+
6 rows in set (0.03 sec)

Checklist(Required)

1. Does it affect the original behavior:
   • Yes
   • No
   • I don't know
2. Has unit tests been added:
   • Yes
   • No
   • No Need
3. Has document been added or modified:
   • Yes
   • No
   • No Need
4. Does it need to update dependencies:
   • Yes
   • No
5. Are there any changes that cannot be rolled back:
   • Yes (If Yes, please explain WHY)
   • No

Further comments

If this is a relatively large or complex change, kick off the discussion at dev@doris.apache.org by explaining why you chose the solution you did and what alternatives you considered, etc...

[yiguolei]

the variable name： skip_storage_engine_merge

[jacktengg]

done

[yiguolei]

also add another variable: skip_delete_predicate

[jacktengg]

done

[yiguolei]

This line is useless?

[yiguolei]

LGTM

[github-actions]

PR approved by at least one committer and no changes requested.

[github-actions]

PR approved by anyone and no changes requested.

[morningman]

use public, or it may cause some serde probem.

[jacktengg]

updated

[morningman]

If this is true, can we select DELETE_SIGN directly?

[jacktengg]

updated

[yiguolei]

LGTM

[github-actions]

PR approved by at least one committer and no changes requested.