Skip to content

Ability for data handlers to stream data#12018

Merged
StpMax merged 104 commits into
releases/26.1.0from
feat/FQE-1618
Mar 5, 2026
Merged

Ability for data handlers to stream data#12018
StpMax merged 104 commits into
releases/26.1.0from
feat/FQE-1618

Conversation

@StpMax
Copy link
Copy Markdown
Collaborator

@StpMax StpMax commented Dec 16, 2025

Description

This PR introduces changes that enable data handlers to return data in chunks rather than a single large block.

Originally, data handlers would always return data as one large block in response to a user's request (when using the native_query method). This approach caused issues when requesting large amounts of data, as it could exceed memory capacity and lead to 'out of memory' errors.

Now, with specific modifications, data handlers can send data to the user in parts without storing all the data in memory at once. Currently, this is supported for:

  • The HTTP protocol
  • "Simple" queries that don't require data processing on the MindsDB's side

In this PR, the following handlers have been modified to support chunked data transfer:

  • PostgreSQL (+inherited)
  • Oracle
  • Snowflake
  • MySQL (+inherited)

Changes to the HTTP API:
A new parameter, response_format, has been added to the /query/sql endpoint with the following possible values:

  • (missed): Default behavior, the response will be regular application/json with all data in one piece.
  • sse: The response will be streamed as text/event-stream.
  • jsonlines: The response will be streamed as application/jsonlines.

Fixes #FQE-1618

Type of change

(Please delete options that are not relevant)

  • 🐛 Bug fix (non-breaking change which fixes an issue)
  • ⚡ New feature (non-breaking change which adds functionality)
  • 📢 Breaking change (fix or feature that would cause existing functionality not to work as expected)
  • 📄 This change requires a documentation update

Verification Process

To ensure the changes are working as expected:

  • Test Location: Specify the URL or path for testing.
  • Verification Steps: Outline the steps or queries needed to validate the change. Include any data, configurations, or actions required to reproduce or see the new functionality.

Additional Media:

  • I have attached a brief loom video or screenshots showcasing the new functionality or change.

Checklist:

  • My code follows the style guidelines(PEP 8) of MindsDB.
  • I have appropriately commented on my code, especially in complex areas.
  • Necessary documentation updates are either made or tracked in issues.
  • Relevant unit and integration tests are updated or added.

Comment thread mindsdb/api/executor/data_types/sql_answer.py Outdated
Comment thread mindsdb/api/executor/sql_query/result_set.py Outdated
Comment thread mindsdb/integrations/libs/response.py
Comment thread mindsdb/integrations/libs/response.py Outdated
Comment thread mindsdb/integrations/handlers/postgres_handler/postgres_handler.py Outdated
Comment thread mindsdb/integrations/handlers/mysql_handler/mysql_handler.py Outdated
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants