FE Catalog should know the path info of each replica in BE

[morningman]

Problem Description

Currently, Catalog only saves 'which backend a replica belongs to' info, but does not save the path info of a replica.

Without the path info, We can not precisely control the MIGRATION or SUPPLEMENT when handling load balance or tablet repair. If a Backend has multi-disks, it may cause the situation that one disk is almost full while another disk is empty.

Solution

Report path info via tablet report from Backend, and save it in Catalog.

How to

Backend

Change the tablet report data structure.

Current TReportRequest of tablet is:

map<Types.TTabletId, TTablet>.

We change it to:

map<long, map<Types.TTabletId, TTablet>>.

The outer map key is the hash value of "root path". This modification just slightly increase the data size of tablet report pack.

We use hash value, not the path string as key, because 'long' compare faster than 'string'.

Frontend

Currently we only support full report of tablets. So on each tablet report, we need to compare and set the path of each replicas.

By testing, 500000 times compare and set operations of Long on MacBook Pro(2.3 GHz Intel Core i5) cost 3ms. So this is not a big deal cause handling tablet report is not a latency sensitive operation.

Path info will not persist in catalog, which means only Master FE knows this info. Since all scheduler work is done on Master FE, so non-master FE does not need this info.

Tablet report trigger

Currently, Backend reports tablet info in a fix interval, which means if FE restarts, it may wait at most a report interval to get the path info of replicas.

So we need an active trigger to notify the Backend that 'You need to report tablet info, immediately!' once Master FE restarted or a non-master FE transfer to Master.

Probably, this could be done simply by heartbeat. But the problem is: How to get a Backend realize that the FE restart?

The solution is to add a random integer in heartbeat. Once the Backend get the heartbeat and find nothing changed but this random integer, it should realize that the Master FE restart or is changed. At that moment, Backend reports tablet immediately.

Backend will report tablet once itself restarts. So nothing need to be modified in Backend restart logic.

[imay]

@morningman
I think there is no need to use a map in TReportRequest , it should be a list of TabletInfo. When we need add/drop some more information about tablet later, we only need to modify TabletInfo. In addition Map is not exist in all serialization protocol like protobuf2. So, I think it's better to be a list.

[morningman]

@imay
List is not easy to use. We need to do the diff between metadata and backend, by using some map's method such as 'containsKey'.
Rebuild map from list may cost more time, and some info, such as path, are redundant, which may cause a large report data pack.

[imay]

@imay
List is not easy to use. We need to do the diff between metadata and backend, by using some map's method such as 'containsKey'.

You can convert list to a map only contain what you want.

Rebuild map from list may cost more time, and some info, such as path, are redundant, which may cause a large report data pack.

I don't think this will cost more time, because RPC should do this either. Even more you can construct a map to content in the list, which may be a pointer or reference.

Anyway, I think we should separate our code logical from RPC protocol. Otherwise it will be a nightmare when want to change our protocol.

[morningman]

Anyway, I think we should separate our code logical from RPC protocol. Otherwise it will be a nightmare when want to change our protocol.

I agree. For this time, I will just add a path info member in TTabletInfo structure.
Because change the old map to the new one, may cause metadata lose, unless you update FE first.

[imay]

I agree. For this time, I will just add a path info member in TTabletInfo structure.
Because change the old map to the new one, may cause metadata lose, unless you update FE first.

Actually，you can add a list RPC struct and process logic in frontend code; then after next version, we can use list in backend code.

[morningman]

Yes I checked the code, it is OK to modify both Backend and Frontend this time.

[morningman]

For now, I just add the path info (path hash exactly) in TTabletInfo, nothing else change.

[morningman]

1. A path hash is calculated and saved in BE when BE start

2. BE will report this path hash along with disk report and tablet report, and FE will update this info when handling report.

3. Path info in FE catalog is not persist, thus only Master FE knows it, and it will be lost if FE crashed.

4. Add a trigger that when Master FE restart or Master is changed, BE will report disk and tablet info immediately.

   • For now, this trigger is not implemented well, cause we don't have a uniform framework to trigger a report from different place in different situation.

5. Add path hash exhibition in Backend Proc and Tablet Proc.