Skip to content

Conversation

@CYarros10
Copy link
Contributor

This PR adds support for Google Cloud Vertex AI Feature Store synchronization operations in Apache Airflow.

Vertex AI Feature Store is a managed, cloud-native feature store service that's integral to Vertex AI. It streamlines your ML feature management and online serving processes by letting you manage your feature data in a BigQuery table or view. You can then serve features online directly from the BigQuery data source.

Components Added

  1. SyncFeatureViewOperator: Triggers synchronization of a feature view, updating online serving data from the batch source
  2. GetFeatureViewSyncOperator: Retrieves status and details of a feature view sync operation
  3. FeatureViewSyncSensor: Monitors the progress of feature view sync operations
  4. FeatureStoreHook: Core integration with the Vertex AI Feature Store API

Usage Example

    sync_task = SyncFeatureViewOperator(
        task_id="sync_task",
        project_id=PROJECT_ID,
        location=REGION,
        feature_online_store_id=FEATURE_ONLINE_STORE_ID,
        feature_view_id=FEATURE_VIEW_ID,
    )
    wait_for_sync = FeatureViewSyncSensor(
        task_id="wait_for_sync",
        location=REGION,
        feature_view_sync_name="{{ task_instance.xcom_pull(task_ids='sync_task', key='return_value')}}",
        poke_interval=60,  # Check every minute
        timeout=600,  # Timeout after 10 minutes
        mode="reschedule",
    )
    get_task = GetFeatureViewSyncOperator(
        task_id="get_task",
        location=REGION,
        feature_view_sync_name="{{ task_instance.xcom_pull(task_ids='sync_task', key='return_value')}}",
    )
    sync_task >> wait_for_sync >> get_task

@boring-cyborg boring-cyborg bot added area:providers kind:documentation provider:google Google (including GCP) related issues labels Dec 12, 2024
Copy link
Contributor

@shahar1 shahar1 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice! Got a small suggestion :-)
Could be merged IMO after resolving the above + @MaksYermak 's comments.

Copy link
Contributor

@MaksYermak MaksYermak left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@potiuk potiuk merged commit 16022e0 into apache:main Dec 16, 2024
138 checks passed
got686-yandex pushed a commit to got686-yandex/airflow that referenced this pull request Jan 30, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

area:providers kind:documentation provider:google Google (including GCP) related issues

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants