Link up row-based datasources to serving layer.#9503
Conversation
- Add SegmentWrangler interface that allows linking of DataSources to Segments. - Add LocalQuerySegmentWalker that uses SegmentWranglers to compute queries on data that is available locally. - Modify ClientQuerySegmentWalker to use LocalQuerySegmentWalker when the base datasource is concrete and not a table. - Add SegmentWranglerModule to the Broker so it has them available and can properly instantiate . LocalQuerySegmentWalkers. - Set InlineDataSource and LookupDataSource to concrete, since they can be directly queried now.
clintropolis
left a comment
There was a problem hiding this comment.
rad, this lgtm 🤘
It might be nice to have some additional test coverage with other query engines to make sure there isn't any funny stuff, but since this functionality isn't really documented yet I'm ok if this is filled out in one or more future PRs.
Also, there seems to be a legitimate test failure, you forgot to update InlineDataSourceTest.test_isConcrete to reflect the newly concrete status of InlineDataSource.
There should be a lot more in a future patch that connects this to the SQL layer. But for now, in this patch, there are integration tests.
Thanks, I'll fix that. I also got one of the integration tests wrong, which I'll fix: |
data that is available locally.
datasource is concrete and not a table.
properly instantiate . LocalQuerySegmentWalkers.
directly queried now.