feat: add "stop" keywords as alternative to eot token

[longregen]

Rewrite of #365 (addresses #57) by @joshmackwilliams, updated to the new folder/file structure as the PR might have been abandoned.

From the original author:

Implements #57.
Stop keywords can be specified using the "--stop" parameter. Upon seeing one of these keywords in the generated output, the model will terminate generation immediately. Like reverse prompts, multiple stop keywords can be specified by specifying the --stop argument multiple times.
The implementation is heavily based on the reverse prompt implementation to keep things simple. Tested using 7B (quantized) in both interactive and non-interactive modes.

[howard0su]

we should check the token id instead of the string for stop.

[joshmackwilliams]

For context, the precedent set by #330 is to check for the string (in reverse prompts). I think checking for tokens caused a bug, or at least unintuitive behavior (#292).

[longregen]

Indeed, this should work in the same way as "antiprompts" to provide a better UX to users. Users should be able to add to the CLI parameters --stop "### Assistant" (for example, in the spirit of the trained vicuna model) or --stop PAUSE (for example, to implement Simon Willinson's ReAct Python example), even though these are multi-token markers.

[howard0su]

do we know why? It will be a good learning for me to understand why token will not work? Because one stop string can be generated by the different tokens??

[j-f1]

Yes, see #292 (comment)

[KevinColemanInc]

I also noticed that the stop words weren't always consistently formatted. Sometimes it would do STOP instead of stop. I'd appreciate a normalize function to down case before comparing.

[joshmackwilliams]

Yes, I got busy and no longer have time to maintain that PR. Thanks for rewriting!

[ggerganov]

Haven't looked in details.
Merge it if you have confirmed to be working as expected

[longregen]

While #863 appears to be more polished, I'd merge these changes since it's uncertain when that one will be reviewed and integrated. Plus, it's likely that any significant use of this code would involve utilizing the library rather than relying on this example binary.

[ggerganov]

@longregen
#863 is now ready to merge. Is there anything that this PR adds that is not covered by #863 ?
If yes, maybe we can create a separate PR after the merge?

[longregen]

@longregen #863 is now ready to merge. Is there anything that this PR adds that is not covered by #863 ? If yes, maybe we can create a separate PR after the merge?

#863 is a superset of this PR. Closed this one as that other got merged. Thank you for your awesome work!

[KevinColemanInc]

@longregen The PR you mentioned has been reverted (c85e03d). Is the plan to wait for that to be tested more or should we reopen this PR?