add example with python parameters file

[aandrusenko]

Related to treeverse/dvc#4456

[jorgeorpinel]

Thank you @aandrusenko !

Missing a couple files:

• \content\docs\start\experiments.md
• \content\docs\user-guide\basic-concepts\parameter.md

Would you mind including them? I can do it if you're unable for any reason, no problem at all.

[jorgeorpinel]

@shcheklein this makes me realize we only have params.yaml examples. Should we create a small Parameters files section with CSV, TOML, and Python examples in the Metafiles guide?

[shcheklein]

@jorgeorpinel I would not make it a priority, I hope users can figure the idea out from the YAML example.

• Python is definitely a different beast and it's good to see how does it actually look like.

[jorgeorpinel]

OK. @aandrusenko would you like to try adding a Python params file example in content\docs\command-reference\params\index.md ? Again, I can do it if not. Thanks

[aandrusenko]

@jorgeorpinel

• \content\docs\start\experiments.md

was done

• \content\docs\user-guide\basic-concepts\parameter.md

added

[aandrusenko]

OK. @aandrusenko would you like to try adding a Python params file example in content\docs\command-reference\params\index.md ? Again, I can do it if not. Thanks

I have already added an example

[jorgeorpinel]

Oh, I see. I didn't notice it the first time I looked. OK, checking again now ⌛

[jorgeorpinel]

What other data structures are supported as groups other than classes @aandrusenko @efiop ? Thanks

[aandrusenko]

Only classes and dicts

[jorgeorpinel]

Looks good. There's a couple minor questions left above which may need addressing but could be merged as-is too, probably.

Do you want to double check @shcheklein ? Thanks

[shcheklein]

I see that some checks failed? Also, please let's merge upstream master to run (new) link checks?

[shcheklein]

@aandrusenko qq - if we depend on the instance variable (self.layers), when do we extract value? What is the semantics here?

[aandrusenko]

From a class, you can extract either class constants or self variables defined in __init__

[shcheklein]

Yep, thanks! But what about the semantics? What if I have

self.layers = 10
self.layers = 12

or if I have

self.layers = 3+2

what are the expectations and limitations here?

[aandrusenko]

In the first case, TrainConfig.layers will be 12. In the second, TrainConfig.layers will not be found because the complex expression

[shcheklein]

let's make it part of the docs and/or example? @aandrusenko it's totally fine if you don't have enough capacity, let us know and we'll take it over, but I think we should specify some basic semantics for this. Modifying example (add a few lines with Python comments + some text ?) should be enough, I guess.

[aandrusenko]

@shcheklein, I added more comments

[shcheklein]

we still have some open questions we need to answer (and potentially add details to the docs)

@efiop @aandrusenko it would be great if you could help us here

[aandrusenko]

@shcheklein @jorgeorpinel I squashed changes and rebased

[shcheklein]

Thanks @aandrusenko ! Looks like there is a conflict. We can fix it on my end, but feel free to rebase if you want.

[shcheklein]

@aandrusenko perfect, thanks! 🙏

one last thing - something is happening with GH not attributing commits to you- could you please check your local and GH settings. Both should use the same email I think.

[aandrusenko]

one last thing - something is happening with GH not attributing commits to you- could you please check your local and GH settings. Both should use the same email I think.

Oh, sorry, fixed it

[shcheklein]

thanks @aandrusenko !

[jorgeorpinel]

Post-merge review that I'll address myself. Includes some questions though:

[jorgeorpinel]

Why can't DVC find vars that need evaluation though? Isn't the whole params file interpreted by Python first @aandrusenko @efiop? (Just curious) Thanks

[aandrusenko]

Nope, parameters are parsed using the ast module, it parses complex expressions into a complex structure that I did not support

[jorgeorpinel]

I see, thanks.