Table manager ignores prefixes for retention deletes

[nrjpoddar]

If retention is enabled on table-manager -table-manager.retention-deletes-enabled & -table-manager.retention-period=<>, table manager tries to delete all tables in DynamoDB if the retention criterion is met irrespective of the prefixes (set by -dynamodb.chunk-table.prefix=cortex_chunks_ & -dynamodb.periodic-table.prefix=cortex_weekly_).

Issue is the logic here will always create a v1 schema config (if you're using legacy schema based on CLI args) whose ChunkTables PeriodConfig prefix is never set.

Then the table manager code tries to prefix match for an empty string which always matches and all DynamoDB tables get added to the toDelete list.

[nrjpoddar]

@bboreham or @tomwilkie we should patch this immediately as it can cause unwanted DynamoDB deletes if you haven't locked down your IAM permissions.

[bboreham]

Is there a similar bug on the creation side?

[nrjpoddar]

I think so but i'm unclear on the fix. We can make sure that the default prefix is also set for ChunkTables here https://github.com/cortexproject/cortex/blob/master/pkg/chunk/schema_config.go#L105 like we do for IndexTables.

[nrjpoddar]

Update on this issue if you're having this problem, the table-manager will also skip creating tables as it will first try to delete https://github.com/cortexproject/cortex/blob/master/pkg/chunk/table_manager.go#L225 and skip creates if there's an error on deletes.