Skip to content

Conversation

@mingjerli
Copy link
Owner

Summary

  • Add KestraOrchestrator class for generating Kestra YAML flows
  • Add to_kestra_flow() method to Pipeline class
  • Support cron schedules, custom connection config, and labels
  • Generate YAML flows with automatic task dependencies using dependsOn

Changes

New Files

  • src/clgraph/orchestrators/kestra.py - KestraOrchestrator class
  • tests/test_kestra_integration.py - 24 comprehensive tests

Modified Files

  • src/clgraph/orchestrators/__init__.py - Export KestraOrchestrator
  • src/clgraph/pipeline.py - Add to_kestra_flow() method

Features

  • YAML output - Generates human-readable YAML workflow definitions
  • Automatic dependencies - Tasks wired based on table lineage using dependsOn
  • Scheduling - Cron schedules via triggers
  • Retries - Configurable retry attempts with ISO 8601 duration
  • Labels - Custom key-value labels for flow organization

Example

from clgraph import Pipeline

pipeline = Pipeline.from_sql_files("sql/", dialect="clickhouse")

yaml_content = pipeline.to_kestra_flow(
    flow_id="analytics_pipeline",
    namespace="clgraph.production",
    cron="0 6 * * *",
)

with open("flows/analytics.yml", "w") as f:
    f.write(yaml_content)

Test plan

  • All 24 tests pass
  • Pre-commit checks pass
  • Tested with Kestra Docker container

🤖 Generated with Claude Code

mingjerli and others added 2 commits January 12, 2026 18:03
- Add KestraOrchestrator class in orchestrators/kestra.py
- Add to_kestra_flow() method to Pipeline class
- Generate YAML flows with automatic task dependencies
- Support cron schedules, custom connection config, and labels
- Add comprehensive test suite (24 tests)

The Kestra orchestrator generates YAML workflow definitions
compatible with Kestra's declarative orchestration platform.

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Kestra's io.kestra.plugin.jdbc.clickhouse.Query task type doesn't support
the dependsOn field. Instead, we rely on topological ordering of tasks,
since Kestra executes tasks sequentially in the order they're defined.

Changes:
- Remove dependsOn field generation from KestraOrchestrator
- Update tests to verify topological ordering instead of dependsOn
- Update docstrings to reflect actual behavior

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
@mingjerli mingjerli merged commit b88fb0d into main Jan 13, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants