Skip to content

Add service to read checkpoint folder #687

@kevinwallimann

Description

@kevinwallimann

Background
The goal is to prevent Hyperdrive job instance creations when there are no messages available on a kafka topic (or only a few). To that end, the offsets from the checkpoint folder should be compared with the latest offsets from the kafka topic.

This issue is about getting the offsets from the checkpoint folder.

All configurations (including keytab location) needed to access the checkpoint folder should be available from the resolved job definition.

Task
A service method should be written, that takes a ResolvedJobDefinition and returns the latest offset from the checkpoint. The method should also return whether or not the offset has been committed.

Related
#688, #690

New application properties
kafka.consumers.cache.size Optional, default 50

Metadata

Metadata

Labels

No labels
No labels

Type

No type

Projects

No projects

Milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions