-
Notifications
You must be signed in to change notification settings - Fork 3.7k
Closed
Description
Motivation
At present, Doris do not push down limit operation to Exteranl Table Engine.It may serious performance problems in some scenarios. For example, this is a Exteranl Table of Mysql:
CREATE EXTERNAL TABLE `baseall_mysql` (
`k1` tinyint(4) NULL COMMENT "",
`k2` smallint(6) NULL COMMENT "",
`k3` int(11) NULL COMMENT "",
`k4` bigint(20) NULL COMMENT "",
`k5` decimal(9, 3) NULL COMMENT "",
`k6` char(5) NULL COMMENT "",
`k10` date NULL COMMENT "",
`k11` datetime NULL COMMENT "",
`k7` varchar(20) NULL COMMENT "",
`k8` double NULL COMMENT "",
`k9` float NULL COMMENT ""
) ENGINE=MYSQL
COMMENT "MYSQL"
PROPERTIES (
"host" = "192.168.1.1",
"port" = "8086",
"user" = "root",
"password" = "123456",
"database" = "test",
"table" = "base"
);
This table have 60000 row in external table.
We do query like:
select * from baseall_mysql limit 10;
Now Doris do not push down limit clause of mysql table, so Doris fetch all row in baseall_mysql and get top 10 of the table.it is cost unnecessary network and mem in MySQL Scan Node / ODBC Scan Node.
In my test:
this query cost 2.1s but push down the limit clause, it only take 0.21s.
So we need to support limit clause push down for ODBC Table and MySQL Table.
Metadata
Metadata
Assignees
Labels
No labels