Skip to content

Feature: Extractor class#1

Merged
avisionh merged 14 commits intomainfrom
feature/extractor-class
May 21, 2021
Merged

Feature: Extractor class#1
avisionh merged 14 commits intomainfrom
feature/extractor-class

Conversation

@avisionh
Copy link
Owner

@avisionh avisionh commented May 17, 2021

Summary

This branch extracts table names from SQL script names and their associated table dependencies.

Changes

The high-level changes made in this PR are:

  1. Create class and methods to extract table names and their dependencies from scripts
  2. Write tests for methods.
  3. Add automated testing in CI pipeline.

Please see the commit history for more detailed changes.


Check

  • The code is readable.
  • The test suite passes via running the below:
python -m pytest tests/
  • The GitHub Actions checks pass.

avisionh added 12 commits May 20, 2021 23:32
This is so we can start parsing and extracting dependencies from it.
This is so we have functionality to remove strings that make the query difficult to parse.
This is so we can get it into a format that we can extract table names from.
This is so we can get the equivalent value of a passed in key for a nested dictionary.
This is so people can replicate the work here.
This is so we can get it in a dictionary for visualising.
This is so we can have a layer of tables derived off another layer.
This is so we have a complicated query which uses ctes and subqueries, which will be good for testing.
This is to ensure it works.
This is to improve efficiency.

Also make documentation of reference_datasets argument clearer.
This is so we can ensure class method works on a directory of SQL scripts.
This is so we can check this works also.
@avisionh avisionh force-pushed the feature/extractor-class branch from abefeb7 to fb3e799 Compare May 20, 2021 22:37
This is so it can be run automatically.
@avisionh avisionh marked this pull request as ready for review May 20, 2021 22:40
@avisionh avisionh force-pushed the feature/extractor-class branch from 5a927f5 to 66f594f Compare May 20, 2021 23:23
This is to show what it builds on.

Include Python badge also and move ISSUES_TEMPLATE/ to .github/.
@avisionh avisionh force-pushed the feature/extractor-class branch from 66f594f to 0d4881d Compare May 20, 2021 23:25
@avisionh avisionh merged commit 494defd into main May 21, 2021
@avisionh avisionh deleted the feature/extractor-class branch May 21, 2021 06:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant