Skip to content

Add service to decide if a DagInstance creation can be skipped #690

@kevinwallimann

Description

@kevinwallimann

Background
The goal is to prevent Hyperdrive job instance creations when there are no messages available on a kafka topic (or only a few). To that end, the offsets from the checkpoint folder should be compared with the latest offsets from the kafka topic.

This issue is about deciding whether a dag instance should be created or not based on the latest checkpoint offset and kafka offset.

Task
In the DagInstanceService a new dag instance is created. The creation of this dag instance can be skipped, if the latest kafka offset is equal to the latest checkpoint offset, if the latest checkpoint offset has been committed. If the latest checkpoint offset has not been committed, a dag instance has to be created.

Related
#687, #688

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions