Enhancement
LOAD DATA can load csv/parquet/sql type data files, with some compression, from s3/gcs...
work breakdown:
- data reader
- support read one file from MySQL client connection, cloud
- support read multiple file and filter from cloud
- data parser
- integrate lightning's parser to support CSV/parquet/...
- data processor
- KV writer
- lightning ingest writer
- transaction KV writer based on insert executor family
syntax
For CSV-like data source files, we support loading it from cloud storage when no LOCAL keyword, and we add NULL DEFINED BY clause to better support different NULL patterns.
LOAD DATA
[LOCAL]
INFILE 'file_or_url'
[FORMAT 'string']
[REPLACE | IGNORE]
INTO TABLE tbl_name
[PARTITION (partition_name [, partition_name] ...)]
[CHARACTER SET charset_name]
[{FIELDS | COLUMNS}
[TERMINATED BY 'string']
[[OPTIONALLY] ENCLOSED BY 'char']
[ESCAPED BY 'char']
[DEFINED NULL BY 'string' [OPTIONALLY ENCLOSED]]
]
[LINES
[STARTING BY 'string']
[TERMINATED BY 'string']
]
[IGNORE number {LINES | ROWS}]
[(col_name_or_user_var
[, col_name_or_user_var] ...)]
[SET col_name={expr | DEFAULT}
[, col_name={expr | DEFAULT}] ...]
[load_data_options]
format can be delimited data(when format is omitted, default to this format), sql file, parquet. delimited data is normal MySQL format. when format is sql file, parquet, there cannot be fields... lines... ignoreLines... clause
load data option clause
WITH (detached | option_name=literal)...
allowed option include
- import_mode: logical or physical, see lightning
local/tidb mode
- disk_quota: see lightning
disk-quota, only works on physical mode
- checksum_table: see lightning
checksum, only works on physical mode
- add_index: whether import should add secondary index for the target table, only works on physical mode
- analyze_table: see lightning
analyze, only works on physical mode
- thread: see lightning
region-concurrency
- batch_size: import transaction batch size
- max_write_speed: control how fast we write
- split_file: see lightning
strict-format
- record_errors: see lightning
max-errors
- detached: When an import runs in DETACHED mode, it will execute asynchronously and the job ID will be returned immediately without waiting for the job to finish.
more details will be posted later
Enhancement
LOAD DATA can load csv/parquet/sql type data files, with some compression, from s3/gcs...
work breakdown:
syntax
For CSV-like data source files, we support loading it from cloud storage when no LOCAL keyword, and we add NULL DEFINED BY clause to better support different NULL patterns.
format can be
delimited data(when format is omitted, default to this format),sql file,parquet.delimited datais normal MySQL format. when format issql file,parquet, there cannot befields...lines...ignoreLines...clauseload data option clause
allowed option include
local/tidbmodedisk-quota, only works on physical modechecksum, only works on physical modeanalyze, only works on physical moderegion-concurrencystrict-formatmax-errorsmore details will be posted later