[WIP] 3482 Develop components for the model package MVP

[Nic-Ma]

Fixes #3482 .

Description

Develop the complete prototype of model package in this branch as @wyli suggested: #3518 (comment).
Then split into smaller PRs for review.

Status

Work in progress

Types of changes

• Non-breaking change (fix or new feature that would not break existing functionality).
• Breaking change (fix or new feature that would cause existing functionality to change).
• New tests added to cover the changes.
• Integration tests passed locally by running ./runtests.sh -f -u --net --coverage.
• Quick tests passed locally by running ./runtests.sh --quick --unittests --disttests.
• In-line docstrings updated.
• Documentation updated, tested make html command in the docs/ folder.

[Nic-Ma]

/black

[Nic-Ma]

/build

[Nic-Ma]

/black

[Nic-Ma]

/build

[Nic-Ma]

/black

[Nic-Ma]

/build

[Nic-Ma]

/build

[ericspod]

The config parsing code looks good to me, submitting that separately to be integrated now separate from the other concepts here would work.

[Nic-Ma]

The config parsing code looks good to me, submitting that separately to be integrated now separate from the other concepts here would work.

Hi @ericspod ,

Thanks for your review.
I started to split the features into smaller and clearer PRs for easier review and update, the first one is: #3720.
It's not urgent to merge it, please review and discuss the solution within team.

Thanks in advance.

[Nic-Ma]

Hi @ericspod ,

BTW, do you know any other solutions in other projects that parse / construct python instances from JSON / YAML config?
I see something related:
https://github.com/cwacek/python-jsonschema-objects
https://pynative.com/python-convert-json-data-into-custom-python-object/
Maybe we can refer to them for unknown ideas or verify this current solution.

Thanks in advance.

[ericspod]

Hi @ericspod ,

BTW, do you know any other solutions in other projects that parse / construct python instances from JSON / YAML config? I see something related: https://github.com/cwacek/python-jsonschema-objects https://pynative.com/python-convert-json-data-into-custom-python-object/ Maybe we can refer to them for unknown ideas or verify this current solution.

Thanks in advance.

These solutions are about creating new types or objects without classes from JSON. What I think we need is just a way to instantiate existing classes based on a name to find the right class and the values for constructor arguments given as JSON. Given a schema like

{
    "object_type" : "foo.bar.baz",
    "arguments" : {
        "arg1":1,
        "arg2":[0,1,2]
    }
}

This would be interpreted as creating a new object of type baz taken from the module foo.bar. We can use importlib.import_module to load foo.bar, extract the member baz from it, then instantiate with baz(arg1=1,arg2=[0,1,2]). We can do this recursively if arguments are objects rather than literals or JSON structures, but we are still somewhat limited by the possible values JSON provides.

[Nic-Ma]

Hi @ericspod ,

Thanks for your explanation, I think your idea aligns with my proposal in this PR.
I will involve more guys to take a look and start to review the first separate PR: #3720.

Thanks.

[wyli]

closing this one as the major part is now merged into dev, the others are tracked here #3482