Consider a better implementation for source

[galamit86]

Currently, each data has a strict requirement to have a source specified. It is then used mainly (possibly only) for naming of files and folders - i.e. *cbs*.v3.83583NED_ColDescriptions.json in several places along the process:

• Locally
• In GCS
• IN BQ (as a prefix for datasets an names)

While informative and often relevant, there are times when it becomes an obstacle. For example, uploading CBS catalogs, currently results in a BQ dataset titled None_catalogs.

Another problem occurs if we want to load datasets from multiple sources at once - such as when loading the entirety of the IV3 catalog from statline.

I can imagine changing the flow parameters to take a tuple (or dict) of (source, id).

These might be two different issues - and neither of them is strictly necessary. The concept of source might be reconsidered.

[galamit86]

Another semi-related issue is that with our current setup, adding a new source (i.e. mkb) requires updating two places:

1. In the user_config.toml file:

[pahts]
cbs = "cbs"
mlz = "mlz"
rivm = "rivm"
mkb = "mkb"

2. The same within statline-bq/config.toml (this isn't strictly necessary if running from within nl-open-data, but practically there is no reason not to update it as well when a new source is added to the list.

3. In statline-bq/config.py, within the Paths class:

@deserialize
@serialize
@dataclass(frozen=True)
class Paths:
    cbs: str = None
    mlz: str = None
    rivm: str = None
    mkb: str = None

@dkapitan - this is what I was talking about re source the other day. It feels like there should be a better way, but I am not sure if this is "wrong enough" to worry about.

[dkapitan]

This is not that wrong 🙃. Points 2 and 3 are not completely DRY but I'll leave it for now.

…

On Wed, 17 Mar 2021, 22:21 Amit Gal, ***@***.***> wrote: Another semi-related issue is that with our current setup, adding a new source (i.e. mkb) requires updating two places: 1. In the user_config.toml file: [pahts] cbs = "cbs" mlz = "mlz" rivm = "rivm" mkb = "mkb" 1. The same within statline-bq/config.toml (this isn't strictly necessary if running from within nl-open-data, but practically there is no reason not to update it as well when a new source is added to the list. 2. In statline-bq/config.py, within the Paths class <https://github.com/dataverbinders/statline-bq/blob/ef48587cba6bf8f4a9e907d81920b5044621823c/statline_bq/config.py#L78> : @deserialize @serialize @DataClass(frozen=True) class Paths: cbs: str = None mlz: str = None rivm: str = None mkb: str = None @dkapitan <https://github.com/dkapitan> - this is what I was talking about re source the other day. It feels like there should be a better way, but I am not sure if this is "wrong enough" to worry about. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#66 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AA7GYDSKPUABNWC4V4RAUK3TEEMPFANCNFSM4ZB7KPGQ> .