This repository was archived by the owner on Jun 24, 2024. It is now read-only.
refactor(llama): remove bincode#123
Merged
philpax merged 3 commits intorustformers:mainfrom Apr 12, 2023
philpax:remove-bincode
Merged
refactor(llama): remove bincode#123philpax merged 3 commits intorustformers:mainfrom philpax:remove-bincode
philpax merged 3 commits intorustformers:mainfrom
philpax:remove-bincode
Conversation
philpax
added
the
meta:maintenance
setzer22
approved these changes
Apr 12, 2023
Collaborator
setzer22
left a comment
setzer22
left a comment
There was a problem hiding this comment.
LGTM! 👍 Not much to add, the change makes sense, we don't need bincode and any llama-rs user can pick the serialization library that best fits them.
But I think it would be worth it to add a comment on InferenceSession to make it clear to users to always pick a binary format (i.e. no JSON or RON) because serializing large number matrices in this format could easily increase the size (of the already quite large snapshots) by several orders of magnitude.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
While writing up #122, I realised it doesn't really make sense to be prescriptive about how to read or write a snapshot to disk. This PR moves all read/write logic to the CLI, and leaves the snapshot/ref as Serde-compatible so that users can make their own decisions on how to snapshot.
The main benefit of this is that it removes the
bincodedependency from the library.