Skip to content

Skip dag instance creation if no new message is available in kafka #700

@kevinwallimann

Description

@kevinwallimann

Feature

  • Add logic in Executors (see around lines 100+)
  • For Hyperdrive jobs, determine if there are messages to be consumed, using the service created in Add service to decide if a DagInstance creation can be skipped #690 HyperdriveOffsetComparisonService.isNewJobInstanceRequired
  • Create a HyperdriveExecutor, copy execute method from SparkExecutor, job status must be updated always (cannot be skipped)
  • If no message is available, finish stop with status No Data
  • If message is available, use the SparkExecutor

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions