DeltaWriter's FlushStatistic needs a metric

[vagetablechicken]

More dev-friendly way to debug, instead of grep logs.
DeltaWriter can flush many times, and there are too much writers opened, so we can get the max flush time in 15s. Or avg?

[wangbo]

I think the main pain points here is that the flush log isn't associated with a transaction id or query id, so it's hard to know how much time cost on DeltaWriter.close when we tracking a slow stream load using transaction id or query id.

[vagetablechicken]

I think the main pain points here is that the flush log isn't associated with a transaction id or query id, so it's hard to know how much time cost on DeltaWriter.close when we tracking a slow stream load using transaction id or query id.

Adding info in DeltaWriter is useful for tracking one slow stream load, also make sense.
But if we don't have the slow load id, e.g. users may be less sensitive to the execution time, or none of load is significantly slower than others.
Just think, for some reason the flush time or count in one BE is always longer/higher than others. But it's relative, we need the global statistics of BEs. If we have metric about it, it'll be more intuitive to us.

[wangbo]

I see.
I am curious about detail of some reason the flush time or count in one BE is always longer/higher than others.
Under what circumstances will this problem occur and what is the corresponding processing strategy?

[vagetablechicken]

I see.
I am curious about detail of some reason the flush time or count in one BE is always longer/higher than others.
Under what circumstances will this problem occur and what is the corresponding processing strategy?

If flush more times, it's probably because the loading data are only sent to a few BE. Maybe key cols are improper, or partition bucket num is too small(bucket num can be set in each parition, maybe a few partitions has improper bucket num, hard to find out).
If flush for too long, maybe the hard disk has some errors.

Actually, if we have tablet monitoring data shown on BE web or other place, we can easily find the problem.
e.g. it's slow read or write? which tablet is abnormal? blabla.
The tablet monitoring data could contain the flush time&count.

[wangbo]

Thanks for the answer

[vagetablechicken]

MemTable flush count & time already have BE level stats, sorry, I didn't see it before create this issue. But no tablet level. It's better to add tablet monitoring than just adding a tablet flush metric. I'll close this issue.