other databases

[pm100]

I know sqlite very well (and love it), but it is not intended for large scale backends.

I have not looked at the code base to see how married it is to sqlite but it seems like a no brainer that it should support other databases (via diesel or some other abstraction layer)

[ignatz]

I know sqlite very well (and love it), but it is not intended for large scale backends.

If you were more specific on the issues you see and what "large" means in your context, we could have discussion about it. For example: high-availability, write locks, ...

I have not looked at the code base to see how married it is to sqlite but it seems like a no brainer that it should support other databases (via diesel or some other abstraction layer)

abstraction layers over a set of technologies usually restrict you to the least-common-denominator. Specifically, databases often have a lot more functionality that goes beyond ISO SQL, e.g. listening to changes, retrieving structured schema information.

Maybe you're looking for a specific DB or feature? If you're looking for API serving, you could have a look at postgrest et al. If you're looking for a more tightly integrated solution, there's SupaBase for Postgres and AppWrite for MySQL/MariaDB.

[pm100]

Fully aware of all the DB issues, abstraction layer issues .... Former chief architect and CTO of large SAAS company. Multi-tenant, multi-platform, multi-db.

The main issue with the architecture of this product at the moment is that it will never be used for big production applications. It lacks scalability and reliability. Maybe thats not your target audience tho.

I came here becuase I finished work on a couple of other rust projects and saw this highlighted (this week in rust, or maybe rustaceans).

[ignatz]

I came here becuase I finished work on a couple of other rust projects and saw this highlighted (this week in rust, or maybe rustaceans).

Welcome! It's great to have technical discussions.

The main issue with the architecture of this product at the moment is that it will never be used for big production applications. It lacks scalability and reliability. Maybe thats not your target audience tho.

Sounds like reliability and scalability are your main concerns? What specifically? I asked you before because there are a lot of misconceptions and obsolete limitations. I want to make sure we're on the same page.
The misconceptions alone may validate your hypothesis regarding adoption. Postgres has severe limitations when it comes to horizontal scalability, yet folks have taken it very far.

I tried to point out to you that ORMs, query builders, ... don't address functionality that goes beyond ISO SQL that is needed for features that go beyond CRUD apis.

If you were more specific on what scale means, what your workflows look like, what limitations you see, what reference you're striving for, we could have a constructive conversation.

Fully aware of all the DB issues, abstraction layer issues ....

Look, I'm sure I'm not aware of all the issues in all the contexts, but I'm eager to hear.

Former chief architect and CTO of large SAAS company. Multi-tenant, multi-platform, multi-db.

That's wonderful. Unfortunately, it says nothing about the problem space, yet it says a lot about you. Most experienced folks don't appreciate authority arguments but you'll find them very intrigued by sound technical arguments.

[vootooguru]

I've reviewed some of the code for this project,and it appears to be deeply integrated with SQLite.To add support for other databases,significant modifications would be required.As mentioned,supporting multiple databases often means being limited to more basic functionalities.I use Trailbase for storing account and payment data,while business data storage is handled by Postgres.The volume of business data can be quite substantial,making SQLite unsuitable.For a startup product,a user base in the millions is a significant number,and SQLite is sufficient for that scale.

By the way,I noticed the roadmap on the website mentions plans to implement proxy functionality.I've been studying this area recently for developing a resource download tool.I'm not quite clear on what problem the proxy feature would solve for this project.

[ignatz]

Thanks for chiming in and sharing your perspective.

I use Trailbase for storing account and payment data,while business data storage is handled by Postgres.The volume of business data can be quite substantial,making SQLite unsuitable.

Could you elaborate a bit more on your concerns? I'm genuinely curious and would like to understand where folks are coming from. IMHO, the scaling behavior of SQLite and Postgres is quite similar with the biggest difference being row-level locks.

---

By the way,I noticed the roadmap on the website mentions plans to implement proxy functionality.I've been studying this area recently for developing a resource download tool.I'm not quite clear on what problem the proxy feature would solve for this project.

It's a bit of tangent but happy to answer. If you have more questions or concern, we should probably open a separate discussion to stay on brand.

The thought process starts with TLS. Right now using TB's auth requires a reverse proxy for TLS termination, which is fine for many and probably even a feature. However, if you really want a minimal stack and TB to be completely standalone, it needs builtin termination. If that's the setup you choose and TB sits at the root, you might grow into a situation where you might want to terminate some other API that's where some basic routing might come in handy. TB already supports proxying requests to javascript endpoints already so proxying externally isn't too much of a leap. That said, I would expect any such functionality to be fairly basic. There are phenomenal reverse proxies out there and I don't see TB competing with them. On the other hand there's great rust tooling (e.g. https://github.com/cloudflare/pingora) and if you use TB as a framework you could probably get creative. Time will tell. Does that make sense?

[vootooguru]

Could you elaborate a bit more on your concerns? I'm genuinely curious and would like to understand where folks are coming from. IMHO, the scaling behavior of SQLite and Postgres is quite similar with the biggest difference being row-level locks.

Based on my past experience, there is a need to elastically increase storage capacity. Previously, I used MySQL to configure clusters to address this issue, which basically met the requirements. I am relatively concerned about SQLite in this regard. Therefore, I generally use TB to store data that can be anticipated to remain within a controllable range over the next ten or twenty years, such as user accounts or subscription records. However, for business data, I would consider using other more elastic solutions.

I generally use a reverse proxy as a front-end for my application servers.I don't think TB needs to become a reverse proxy server—that's a different domain with its own set of solutions,and there are already great products out there(I use Caddy).If TB can serve as an ultra-lightweight alternative to Firebase,that would be more than sufficient.The main reason I'm here is that Firebase has some issues,such as restricted access in certain regions.Therefore,I need a self-deployable solution to reduce the redundancy of similar infrastructure across different applications.Both PocketBase and TrailBase have this potential,and I have used PocketBase.However,I'm not familiar with the Go language.As an independent developer,I prefer Rust,a language where you write it once and it's mostly good to go.I also don't use JavaScript much(TypeScript is much better and serves as my auxiliary language),and it might take me a lot of time to debug runtime issues.

[ignatz]

Based on my past experience, there is a need to elastically increase storage capacity. Previously, I used MySQL to configure clusters to address this issue, which basically met the requirements. I am relatively concerned about SQLite in this regard. Therefore, I generally use TB to store data that can be anticipated to remain within a controllable range over the next ten or twenty years, such as user accounts or subscription records. However, for business data, I would consider using other more elastic solutions.

That clarifies it, previously I was more thinking towards performance scaling concerns. You're specifically concerned with storage.

SQLite has great documentation on its limits: https://sqlite.org/limits.html. The maximum DB size is 256TB per database with up to 125 connected databases. It's certainly not impossible to build a RAID of that size but I take your point about MySQL more as horizontal sharding across mulitple machines. I agree that breaking up a large database using network filesystems is probably less performant than a more domain-aware solution like MySQL. FWIW, support for multiple databases is certainly something I'd like to add to TrailBase to give it a bit more flexibility in that regard. Just one thing to keep in mind for multiple DBs is that joins work but foreign-keys do not across DB boundaries. I would expect that most people are better off to stay within the limits of single DB.

As for reverse proxying and TLS, I've since added support for TLS termination to remove some of the needs for a reverse proxy, if one values a really shallow stack. I'd still like to eventually add certificate auto-renewals to make this more practical. As for reverse-proxy tasks, not much has changed. I've no plans to add much built-in functionality but I could totally see folks to grab something like pingora and extend TB.

The main reason I'm here is that Firebase has some issues,such as restricted access in certain regions.Therefore,I need a self-deployable solution to reduce the redundancy of similar infrastructure across different applications.

This makes me very happy to hear. Ownership and cloud-repatriation were certainly some of my main motivations 👍