feat: v5 basics

[joloppo]

Jira Ticket

dynamicio/dynamicio
├── __init__.py
├── base.py # This is the MEAT <- More explanation below
├── handlers
│   ├── __init__.py
│   ├── keyed.py # This can be used for to contain multiple other resources, emulating 'environments
│   └── file.py # 
├── inject.py # functions for injecting [[ var ]] and {var} variables
├── utils.py # <- contains some stuff from old version
└── validators.py # <- we will put the validation functions that pandera is missing here

Usage example - plain

from dynamicio.handlers.file import ParquetFileResource
df = ParquetFileResource(path="input/bar.parquet", allow_no_schema).read()

Usage example - with Schema

from pandera import SchemaModel, Series
from dynamicio.handlers.file import ParquetFileResource

class BarSchema(SchemaModel):
    column_a: Series[int]

df = ParquetFileResource(path="input/bar.parquet").read(pa_schema=BarSchema)

# OR

df = ParquetFileResource(path="input/bar.parquet", pa_schema=BarSchema)

Info

base.py contains a BaseResource class, which is inherited by relevant handlers for different filetypes, (and coming, S3, postgres and kafka). Because BaseResource itself is a pydantic model we have typechecking on-use.

To inject variables into resources you can either use f-strings, or use the inject method.

resource = ParquetFileResource(path="input/[[ something ]]/{name}.parquet")
resource = resource.inject(something='foo', name='bar')
resource.read()

If you want to go with the 'chuck it all in' approach, I would recommend something like this resource.inject(os.environ)

Usage - Testing

For different environments during testing feel free to use the keyed resource. In upcoming releases testing will be handled differently and keyed resources should only be used when needing a different dev setup or similar shenanigans.

Note

This is a first sample, to raise any architecture/api concerns. Things to follow include All other IO methods, more tests, logging/metrics capabilities, extra validations and migration scripts and other cli features.