Add sequence numbers to metadata

[rdblue]

Row-level deletes require a way to determine whether a given delete file applies to a data file.

The scope of a delete file will be limited in 2 ways: by time and by partition. Sequence numbers will be used to implement the first. All data and delete files will be added with a sequence number and deletes apply to any data file with a sequence number less than or equal to the delete's sequence number.

[waterlx]

Hi @rdblue , regarding

deletes apply to any data file with a sequence number less than or equal to the delete's sequence number

I think the delete could not be applied to the data file with a sequence number equal to the delete's sequence number, am I correct?

[rdblue]

Yes, a delete should not be applied to a file with the same sequence number. An upsert would add a delete file and a data file with the same sequence number, one containing the row-level delete and one containing the upserted row content. Not applying deletes with the same sequence number ensures that the upsert doesn't delete itself.

[chenjunjiedada]

I would like to restart this job by myself upon #588.

Next step are adding sequence number for manifest file in the manifest list and modifying the manifest reader.

[rdblue]

I think this is complete enough to close.