API to drop and reload data by interval

[surekhasaharan]

Motivation

The process to delete data (mark segments unused) is not simple currently, look at the steps in tutorial http://druid.io/docs/latest/tutorials/tutorial-delete-data.html.
It would be useful to provide coordinator APIs which can be used to delete(mark unused) or reload(mark used) segments for a datasource by interval. The API to reload data should mark any non-overshadowed segments as used.

Proposed changes

Propose to add two new API's to DataSourcesResource.java

POST /druid/coordinator/v1/datasources/{dataSourceName}/markUnused
POST /druid/coordinator/v1/datasources/{dataSourceName}/markUsed

The payload would contain the interval or segment ids which needs to be marked used/unused. Either interval or segment ids should be provided, if both are provided in the payload , the API would throw an error.
Interval specifies the start and end times as IS0 8601 strings.
interval = [start, end] where start and end are both inclusive.
Only the segments which are contained in the interval will be affected, i.e. segments which partially fall in the start or end will not be affected by these API's.

For example, interval payload json can be:

{
     "interval" : "2012-01-01T00:00:00.000/2012-01-03T00:00:00.000"
}

sample payload with segment ids :

{
	"segmentIds": 
	[
		"wikipedia_2019-01-03T19:00:00.000Z_2019-01-03T20:00:00.000Z_2019-01-03T19:00:05.964Z",
		"wikipedia_2019-01-03T22:00:00.000Z_2019-01-03T23:00:00.000Z_2019-01-03T19:00:05.964Z_2"
	]
}

Following http response code would be returned by both APIs:

200 : if data is successfully dropped or reloaded
204 : if no content is found for given interval or segment ids not found in db
400 : if the payload is incorrect, either none or both (interval and segments) are provided
500 : if there was an exception trying carry out the action

Handling overshadowed segments while reloading data

The API to reload data would only mark used=true for segments which are not overshadowed by other segments to avoid the needless reload and drop of segments on historicals. This can be done in SQLMetadataSegmentManager by filtering segments which are overshadowed before updating in the metadata store.

Rationale

Considered DELETE /druid/coordinator/v1/datasources/{dataSourceName}/intervals/{interval} and POST /druid/coordinator/v1/datasources/{dataSourceName}/intervals/{interval}, but the former is already being used to kill segments. So decided to use PUT with the datasource and action, with the payload.

Operational impact

New api, so no operational impact.

Future work

DELETE /druid/coordinator/v1/datasources/{dataSourceName}/intervals/{interval} kills the segments, but DELETE /druid/coordinator/v1/datasources/{dataSourceName}/segments/{segmentId} disables segments. It would be useful to make the behavior of these two APIs coherent or change the URL to make it more intuitive.

[drcrallen]

Please be aware of the impact of re-using segments on the coordinator's timeline logic. When you manually flag all segments that umbrella a time interval as "used" then the coordinator has to consider them ALL when doing the "what should be loaded?" logic. And I don't remember if it was fixed where those segments had to be loaded before it was able to make that distinction. This can cause a lot of churn on the coordinator during such an operation for datasources with a lot of segments.

[surekhasaharan]

thanks @drcrallen for heads up. The proposal mentions "The API to reload data should mark any non-overshadowed segments as used." So I think that would avoid the needless churn. I also added some more details to the "Proposed Changes" section and added a sub-section "Handling overshadowed segments while reloading data" to clarify reload API behavior.

[surekhasaharan]

Updated proposal to change PUT to POST as PUT is supposed to be idempotent and these API's may not be idempotent if invoked multiple times. Specified the exclusivity behavior of the interval start/end values.

[jon-wei]

In case of overshadowed segments found in the interval or segment-ids payload, the reload API would return response code 200 and response body would contain the number of non-overshadowed segments that were successfully reloaded.

Returning a count of affected segments is good for the default case, but I think a non-default option on the API to report per-segment errors/failures in the response would be useful, in case the user wants to know info at such granularity. I think this can be done in a follow-on patch to the PRs that implement this proposal, if you agree that it's useful.

[leventov]

For new APIs, it's better to use PUT for endpoints like .../markUsed and .../markUnused. It's semantically more correct than POST.

[leventov]

Updated proposal to change PUT to POST as PUT is supposed to be idempotent and these API's may not be idempotent if invoked multiple times.

@surekhasaharan how the API may not be idempotent?

[surekhasaharan]

@leventov the drop API may not be idempotent between calls, if more segments get added to the same interval. And the reload API can have different outcomes based on the number of segments overshadowed in the interval at a time, so for same interval, reload API can mark different subset of segments as used=true. See the discussion here