added sdg api library design

[oindrillac]

No description provided.

[russellb]

Some tips for the linter issues

• make md-lint-fix will fix most of the md-lint issues. I think there is one type of issue to fix manually after that.
• and then for make spellcheck you have to add entries to .spellcheck-en-custom.txt

[russellb]

I think there's a need to configure the model name on a per-stage basis. We may have an endpoint configured with more than one model and we want to hit different ones depending on the stage.

[russellb]

This was marked as resolved, but I don't think it got moved.

[oindrillac]

@russellb made updates to this to accept a client, model-name and default model prompt

[russellb]

@oindrillac but it's still not specified on a per-stage basis?

[oindrillac]

the value will be a class variable that gets passed on when an object is initialized. we have also decided to keep the run config in a .py file as per your suggestion let me update the doc

[oindrillac]

PTAL @russellb

[oindrillac]

cc: @shivchander @aakankshaduggal

[russellb]

Thanks for the updates! There are just two minor comments left to resolve.

• moving model_name to be per-step
• noting that model_template and stop_token can be optional and defaulted to the right thing for the subset of models ilab supports right now.
• I suggested an edit to describe two stages of CLI integration:
  1. changing generate_data() without impact to the CLI code
  2. adding new options to the CLI to allow custom pipeline extensions

[russellb]

This was marked as resolved, but I don't think it got moved.

[russellb]

just checking back in on this -- there's still at least one unresolved comment.

You should also squash all the commits back down to one. That will resolve the DCO error in CI. It looks like something else needs to get added to the spellcheck dictionary, as well.

[oindrillac]

its breaking on customizations @russellb how can I update the spellcheck library?

[aakankshaduggal]

its breaking on customizations @russellb how can I update the spellcheck library?
You might want to update here --
https://github.com/instructlab/dev-docs/blob/main/.spellcheck-en-custom.txt

[oindrillac]

#103 <- so that Spell Check passes on this PR

[russellb]

#103 <- so that Spell Check passes on this PR

for this PR to pass CI, that change needs to be included in this PR instead of a separate one.

[oindrillac]

done, added this here

[russellb]

please see the open comment thread about the config format and model name

[russellb]

thank you!