Programmatically add sections to model card

[merveenoyan]

Currently add only supports injecting values to already existing sections in a template, meanwhile add_plot can append new sections at the end of the model card and write plots. add should also support adding sections and subsections to the model card, without appending if possible. e.g. for default template, if user wants to add a subsection to training procedure (which is in the middle of the card) they should be able to do that without modifying the template themselves.

[adrinjalali]

We could also document how users can do this by adding markdown markers. As in, if the user adds this as the content of a section:

this is the main content.

## Subsection

this is the subsection content.

it will be rendered as a subsection.

[0xrushi]

Can anyone assign this to me

[BenjaminBossan]

@rushic24 Before working on this, let's first discuss how to actually implement this feature, as it's not trivial. If you already have an idea, please describe it here.

[E-Aho]

Is it neccesary for add to take multiple sections at a time?

If not, we could refactor the method to take in a single section at a time, and add an optional "parent" parameter, that would allow a workflow like:

1. If parent, add under parent
2. If section in template, add as normal
3. Otherwise, add at end

This would need a bit of a refactor for the _generate_card method and the way self._template_sections stores sections, but should be intuitive for the user

[BenjaminBossan]

Is it neccesary for add to take multiple sections at a time?

I wouldn't change it, honestly, as I believe it's quite useful as is.

Perhaps there is a way, however, to keep it and still add the feature. OTOH:

card = Card(...)
card.add({
    "Model description/Training Procedure/New section": "some text",
})

where "Model description" and "Training Procedure" are existing sections and "New section" is added as a subsection. This still leaves a few open questions, but it's maybe worth exploring.

Going from API to implementation, this is the bigger concern for me though. I don't think the current way we handle model cards lends itself well for this type of dynamic additions. Trying to insert sections with regex looks too brittle to me. I couldn't yet come up with a good solution that works with the current model card implementation.

[0xrushi]

Instead of string we can enable markdown rendering for card values
like

card = Card(...)
card.add({
    "Model description/Training Procedure/New section": """
    
    # Test
    sdjksdksdjsjks
    ## Subsection
    """,
})

[E-Aho]

Perhaps there is a way, however, to keep it and still add the feature. OTOH: ...

I think an API like that could be powerful, but it would need some thought on handling edge cases and how to keep it intuitive for the users.

I don't think the current way we handle model cards lends itself well for this type of dynamic additions. Trying to insert sections with regex looks too brittle to me.

I definitely agree, I don't think regex would be the right approach.

I think to have robust, programatic dynamic additions, it might need a bit of an overhaul, perhaps using a markdown util library to parse the template and keep a running .md structure in a python object we can work with?

[BenjaminBossan]

I think to have robust, programatic dynamic additions, it might need a bit of an overhaul, perhaps using a markdown util library to parse the template and keep a running .md structure in a python object we can work with?

Yes, that's something that I have also thought about. Having some container type in Python to hold the content of the model card instead of a template would give us a couple of advantages. It would be much easier to add, delete, or otherwise modify the model card.

Parsing the template didn't look trivial to me, maybe I just missed something but it looked like it would require quite some effort. Not using a template at all would be a solution, instead having aforementioned container be pre-filled by default with the same sections as the template, but the template gives users a nice thing to look at for guidance, so I'm not sure.

[BenjaminBossan]

Instead of string we can enable markdown rendering for card values

Maybe I don't quite understand your suggestion, but your example should already work right now.

[E-Aho]

Parsing the template didn't look trivial to me, maybe I just missed something but it looked like it would require quite some effort. Not using a template at all would be a solution, instead having aforementioned container be pre-filled by default with the same sections as the template, but the template gives users a nice thing to look at for guidance, so I'm not sure.

It wouldn't be trivial, but there are some nice libraries that parse .md into a Python Object, which you can extend and work with. Python-Markdown comes to mind, and python markdown comes with utilities to edit and add to the element tree. Unsure if adding a dependency to get this functionality is the right call, but I think writing a robust .md parser from scratch that converts to an element tree we can work with and then recompiles to md would be overkill.

I could play with this and try to get a proof of concept up and running if it's a direction that you think might be worth exploring @BenjaminBossan

[BenjaminBossan]

I could play with this and try to get a proof of concept up and running if it's a direction that you think might be worth exploring

Personally, I think having this feature could be useful, so having a POC would be nice, but before you spend too much time on it, I would like to hear if the other @skops-dev/maintainers agree.

[0xrushi]

Instead of string we can enable markdown rendering for card values like

card = Card(...)
card.add({
    "Model description/Training Procedure/New section": """
    
    # Test
    sdjksdksdjsjks
    ## Subsection
    """,
})

Sooo, this won't fix it ?

[E-Aho]

So I've played with it a bit more, and I think there are two main ways forward if we want the functionality to add sections to specific areas of existing Markdown:

1. Use a md parser that makes use of an etree under the hood, like Python-Markdown, then after adding to it recompile back to .md from HTML/XML with something like markdownify, which feels a little heavy handed but would work.

2. Write our own methods to parse the .md template into an object we can work with and then recompile in a similar way as is being done now.

I think either method would be fine, but looking at the implementation of how Python-Markdown converts .md to an etree, writing a well specced parser from scratch would likely be a big task. However, option 1 would need a lot of testing make sure that everything works after the the multiple conversion steps.

There is still the option of just inserting into the markdown string with some regex logic, which having seen what the alternatives would look like, doesn't sound as bad to me