DataLakeCatalog namespace filter #1337

ianton-ru · 2026-01-23T17:55:52Z

Changelog category (leave one):

New Feature

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

DataLakeCatalog namespace filter

Documentation entry for user-facing changes

New setting namespaces for DataLakeCatalog with comma-separated list of namespaces.
Supports rest, glue and unity types.

namepsaces='foo,bar'

rest type can have nested namespaces, supports next rules:

foo - tables from namespace foo, but not from nested namespaces
foo.bar - tables from nested namespace, but not from base namespace
foo.* - tables from al; nested namespaces, but not from base namespace
When tables from both namespaces (base and nested) are required, need to use both: foo,foo.*

CI/CD Options

Exclude tests:

Regression jobs to run:

…pace

ianton-ru · 2026-01-23T17:57:16Z

@codex review

github-actions · 2026-01-23T17:57:47Z

Workflow [PR], commit [51cbf62]

chatgpt-codex-connector

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 8b92a48e5b

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

chatgpt-codex-connector · 2026-01-23T18:05:05Z

src/Databases/DataLake/RestCatalog.cpp

 DB::Names RestCatalog::getTables(const std::string & base_namespace, size_t limit) const
 {
+    if (!allowed_namespaces.isNamespaceAllowed(base_namespace, /*nested*/ false))
+        throw DB::Exception(DB::ErrorCodes::OUT_OF_SCOPE,
+            "Namespace {} is filtered by `namespaces` database parameter", base_namespace);


Skip filtered namespaces when probing emptiness

This new OUT_OF_SCOPE throw means RestCatalog::getTables() now fails for any namespace excluded by the namespaces filter. RestCatalog::empty() still iterates all namespaces returned by the catalog and calls getTables() in its stop condition without checking the filter, so any filtered namespace will now throw and bubble up. That makes DatabaseDataLake::empty() (used by DatabaseCatalog::detachDatabase when dropping/detaching DBs) fail whenever the catalog contains namespaces outside the filter (e.g., namespaces='dev' but prod exists). Consider skipping filtered namespaces in empty() or swallowing OUT_OF_SCOPE there.

Useful? React with 👍 / 👎.

ianton-ru · 2026-01-23T19:54:13Z

@codex review

chatgpt-codex-connector · 2026-01-23T20:08:35Z

Codex Review: Didn't find any major issues. 🎉

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

Open a pull request for review
Mark a draft as ready
Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

ilejn · 2026-01-23T22:00:14Z

src/Storages/ObjectStorage/DataLakes/DataLakeConfiguration.h

+        DataLakeStorageSettingsPtr settings_,
+        std::optional<std::string> catalog_namespaces_ = std::nullopt)
+        : settings(settings_)
+        , catalog_namespaces(catalog_namespaces_ ? *catalog_namespaces_ : std::string("*")) {}


consider std::optional::value_or()

ilejn · 2026-01-23T22:07:20Z

src/Databases/DataLake/GlueCatalog.cpp

+{
+    return allowed_namespaces.contains("*") || allowed_namespaces.contains(namespace_);
+}
+


Not a big deal, but may be make this method virtual and move this implementation to ICatalog to avoid duplication?
Feel free to ignore if you've considered this option and decided that it is better to keep it simple.

Implementation in ICatalog requires to make allowed_namespaces a ICatalog member. But allowed_namespaces has a simple type for 'flat' case and more complex type for case with nested namespaces.

Implementation in ICatalog requires to make allowed_namespaces a ICatalog member. But allowed_namespaces has a simple type for 'flat' case and more complex type for case with nested namespaces.

More complex type can live (under a different name) near more complex implementation in a derived class.
Member of a simple type can be std::optional in ICatalog.

Up to you.

ilejn · 2026-01-23T22:28:05Z

src/Databases/DataLake/GlueCatalog.cpp

        glue_client = std::make_unique<Aws::Glue::GlueClient>(chain, endpoint_provider, client_configuration);
    }

+    boost::split(allowed_namespaces, settings.namespaces, [](char c){ return c == ','; });


Consider boost::is_any_of instead of lambda.

BTW, am I right that 'aaa, bbb' would not work because of space after comma?
Is it ok?
To fix it one can
...split ... is_any_of(", "), boost::token_compress_on)

ilejn · 2026-01-23T22:32:32Z

src/Common/ErrorCodes.cpp

    M(754, UDF_EXECUTION_FAILED) \
    M(755, TOO_LARGE_LIGHTWEIGHT_UPDATES) \
    M(756, CANNOT_PARSE_PROMQL_QUERY) \
+    M(757, OUT_OF_SCOPE) \


Not sure, though may be it is better to either use existing error code (e.g. DATALAKE_DATABASE_ERROR) or make up a more specific name, e.g. CATALOG_NAMESPACE_DISABLED ?

ilejn · 2026-01-24T16:05:26Z

docs/en/engines/database-engines/datalake.md

+| `aws_access_key_id`     | AWS access key ID for S3/Glue access (if not using vended credentials)          |
+| `aws_secret_access_key` | AWS secret access key for S3/Glue access (if not using vended credentials)      |
+| `region`                | AWS region for the service (e.g., `us-east-1`)                                  |
+| `namespaces`            | Comma-separated list of namespaces, supported types: `rest`, `glue` and `unity` |


Initially I thought that rest, glue and unity are supported namespaces. Probably target audience of this feature would not have this problem, but may be '... implemented for catalog types: ..'

ilejn · 2026-01-24T16:26:53Z

src/Databases/DataLake/RestCatalog.cpp

+        boost::split(list_of_nested_namespaces, ns, [](char c){ return c == '.'; });
+
+        size_t len = list_of_nested_namespaces.size();
+        if (!len)


Not sure that I understand what it actually means.
That 'ns' is an empty string? Is it possible?

Yes. Some kind of useless value.
You mean. need to throw exception in this case?

I think that the only possible example of 'useless value'- a string that contains nothing except dots, right?
It can be processed in any way, e.g. as you currently do.

ianton-ru · 2026-01-27T14:51:41Z

Test test_keeper_memory_soft_limit/test.py::test_soft_limit_create failed by timeout, can't strat cluster by some reason. Current PR shoudln't affect it, looks like unrelated flap.
Test test_async_load_databases/test.py::test_materialized_views_cascaded_multiple also looks unrelated, may be some kind of race conditions in materialized views filling, it's a background process, and possible that query was made before changes in MV.
Test test_async_load_databases/test.py::test_materialized_views_replicated because previous failed test did not drop database test_mv. Unrelated to this PR too.

…pace

ianton-ru · 2026-01-28T14:48:21Z

Failed stateless tests:
amd_ubsan 03221_merge_profile_events - looks unrelated. Test for ProfileEvents['UserTimeMicroseconds'] > 0 and ProfileEvents['OSCPUVirtualTimeMicroseconds'] > 0 - is it possible that test computer too fast and execution took less than 1 microsecond?
amd_msan 00160_decode_xml_component, 01732_race_condition_storage_join_long, 03595_equality_deletes_simple - Reason: server died, killed by some timeout?

ianton-ru and others added 2 commits January 23, 2026 18:38

Connect catalog with specific namespaces

c2246f0

Merge branch 'antalya-25.8' into feature/antalya-25.8/data_lake_names…

8b92a48

…pace

chatgpt-codex-connector bot reviewed Jan 23, 2026

View reviewed changes

Filter in 'empty' method

7891aa4

ilejn reviewed Jan 23, 2026

View reviewed changes

ilejn reviewed Jan 24, 2026

View reviewed changes

Fixes after review

404ba0f

ilejn approved these changes Jan 26, 2026

View reviewed changes

Merge branch 'antalya-25.8' into feature/antalya-25.8/data_lake_names…

51cbf62

…pace

svb-alt added the antalya label Jan 28, 2026

DataLakeCatalog namespace filter #1337

Are you sure you want to change the base?

DataLakeCatalog namespace filter #1337

Uh oh!

Conversation

ianton-ru commented Jan 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Changelog category (leave one):

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

Documentation entry for user-facing changes

CI/CD Options

Exclude tests:

Regression jobs to run:

Uh oh!

ianton-ru commented Jan 23, 2026

Uh oh!

github-actions bot commented Jan 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

chatgpt-codex-connector bot left a comment

Choose a reason for hiding this comment

💡 Codex Review

Uh oh!

chatgpt-codex-connector bot Jan 23, 2026

Choose a reason for hiding this comment

Uh oh!

ianton-ru commented Jan 23, 2026

Uh oh!

chatgpt-codex-connector bot commented Jan 23, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ilejn Jan 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ilejn Jan 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ianton-ru commented Jan 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

ianton-ru commented Jan 28, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

ianton-ru commented Jan 23, 2026 •

edited

Loading

github-actions bot commented Jan 23, 2026 •

edited

Loading

ilejn Jan 26, 2026 •

edited

Loading

ilejn Jan 26, 2026 •

edited

Loading

ianton-ru commented Jan 27, 2026 •

edited

Loading