new service: Add native ceph rados support

[Xuanwo]

Inspired by https://ceph.io/en/news/blog/2024/ceph-a-journey-to-1tibps/

It must be interesting for opendal to have native ceph rados support

[Xuanwo]

We have a Rust interface for Ceph at https://github.com/ceph/ceph-rust, but it's currently not well-maintained.

[Lancern]

Wanna give it a try. I don't know if anyone is working on this, but I have some progress in my own testing fork.

The ceph-rust crate is almost unusable for several reasons:

• It is not actively maintained for several years;
• Its Rust interface lacks several key features that are indeed required by OpenDAL:
  • Configurations cannot be set until you have connected to the remote cluster;
  • No asynchronous support.

However its raw C library bindgen seems usable. I have built a new binding for testing upon its raw bindgen.

[Xuanwo]

Wow, that's perfect. Thanks for your work!

[pcmid]

I discovered this project and issue a few months ago, and very interested in it. Recently I wrote a wrapper librados library with asynchronization, which is still a very early version.

Later I will try to support native ceph rados based on this library.

[guojidan]

Maybe we don't need to maintain a library by ourselves. We just modify the official library by contributing to the official repository. 🤔

[Xuanwo]

Maybe we don't need to maintain a library by ourselves. We just modify the official library by contributing to the official repository. 🤔

Hi, @guojidan, That would be perfect. Do you know anyone from the Ceph community interested in helping maintain or mentor this project?

[guojidan]

I am a contributor of Ceph and now I need to use the official library to write my own project, so maybe I will implement it. but I have been quite busy recently with moving to a new city. 😄

[Xuanwo]

but I have been quite busy recently with moving to a new city. 😄

Congratulations! Wishing you happiness in your new city.

I am a contributor of Ceph and now I need to use the official library to write my own project, so maybe I will implement it.

Happy to know that. Could you find someone interested in maintaining ceph-rust? I believe both @Lancern and @pcmid are interested in joining the development.

Additionally, if ceph-rust lacks a maintainer, I am willing to help by removing contribution blocks and reviewing PRs.

[Lancern]

I'm definitely willing to help here :) Half a year ago I wrote some experimental implementation and I can share my code if anyone is interested (but it may be a bit out of date since I haven't updated it for half a year).

[Xuanwo]

Hi @Lancern, I'm not familiar with the underlying details of Ceph. Could you tell me how complex it would be to develop pure native Rust bindings for the Ceph RADOS API (without the need for librados.so)?

[pcmid]

I am also very interested in developing librados implemented in pure rust. But maybe this is a long-term project.

[Lancern]

Could you tell me how complex it would be to develop pure native Rust bindings for the Ceph RADOS API (without the need for librados.so)?

@Xuanwo I'm not very familiar with the inner workings of librados but I believe it would require a lot of effort to have a pure Rust implementation. librados is a very fundamental library and some part of this library is implemented in the very core of ceph to deal with protocol details such as message exchanges, authentication, and encryption. I couldn't find a thorough document on the protocol itself besides the official one at https://docs.ceph.com/en/latest/dev/rados-client-protocol/, which is quite incomplete. We may need a guy who knows this better if we decide to implement librados in pure Rust.