Different storage backend

[x-tabdeveloping]

Hey!
I think using orjson for storing items was probably a good choice from many aspects, but I'm a bit afraid that the cost of it is going to be that saving and loading vector stores is relatively slow, and it takes much more storage than some binary format would have.

Would you mind adding some other storage option for the items in a vector store? Like parquet, MessagePack or something of the sorts?
((msgspec is an awesome library for very fast serialization btw))

[x-tabdeveloping]

Awesome library btw, very well made and structurally sensible. Probably the best vector store I have used to date :D

[Pringled]

Hey @x-tabdeveloping, thanks for the kind words and suggestion! I'll have a look at it; I think we can add an optional dependency to support different storage backends. I'm personally a fan of parquet, but if you have another suggestion, do let me know.

[stephantul]

Hello @x-tabdeveloping ,

We implemented msgpack using msgspec. However, I anticipated this to be a bit faster and/or smaller on disk than JSON, but apparently this is not the case. This is contrary to my expectations. Any ideas whether just serializing a list of strings should be faster and or smaller using msgpack? Maybe I'm doing something wrong.

Stéphan

[Pringled]

Closing due to inactivity; feel free to re-open if this is still needed!