LiteFS is not a magic bullet

Written by haloboy777 on 2023-06-27

Playing with distributed databases and deployments

As brilliant as this tool is, it's not totally void of shortcomings. LiteFS is not a magic bullet for small projects as I previously thought it would be.

I need to accomplish some tasks before I move on to proper deployment piepline for the rust backend

Create a docker file that allows me to just start the container anywhere with some urls as environment variables
To do that I need to copy litefs correctly into the docker container and run it
To schedule database syncing before starting the application to enusre data consistency

As I'm reading more and more about this. I'm not sure if it's the right decision to run litefs.

The thing is that I don't know if its right to run while maintaining lighting speed during its production deployment.

Turns out you can't run write queries on replica databases which renders the whole use of litefs null and void.

Why even bother.. lol.

Read the following paragraph

Cluster management using leases

SQLite operates as a single-writer database which means only one transaction can write at a time. Because of this, we utilize a lease system whereby a single node acts as the "primary" node at any given time. All writes should be directed to that node and it will propagate changes to the rest of the cluster.

However, nodes can die or become disconnected from the cluster so the primary must be able to change dynamically. LiteFS is built to be run on ephemeral environments where cluster membership can change quickly and frequently so it utilizes leases via Consul to determine the current primary node.

When a LiteFS node starts up, it checks a key within the Consul server for the current primary node. If no primary node exists, then the LiteFS node attempts to obtain a lease from Consul to become a primary node itself. If successful, it updates the Consul key to share its API URL so other LiteFS nodes can connect to it.

Now With this cleared.. I need to look at alternatives for replacing litefs as my primary data store.

I can use a managed service for all my database needs and be done with it or I can look at other opensource services like following

VLCN or ayb

Exciting thing about VLCN is that it provides multi-master replication and partition tolerance to SQLite via conflict free replicated data types (CRDTs)

And ayb also provides simillar feature set but it is in its pre-alpha stage so not a good candidate for deployment right now.

Turns out planetscale allow writes on replicas but it's counted as a proper paid database which I'm not paying for an application that I haven't built yet.

Then there's turso and Cloudflare D1 which in turn are doing the same thing that I can do with LiteFS without the added cost of using an external service.

D1 fineprint

Larger databases: During the alpha period, the maximum per-database size is limited to 100 MB. We plan to support not only larger databases, but more databases, in the near future.

Read replication: D1 will create and distribute replicas of your data across the Cloudflare global network, reducing the latency for your read queries. Cloudflare distributes your data close to where your users are and keeps your read replicas up to date with changes.

Most probably will go ahead with litefs and come up with an working application by the end of this week.

Till then stay classy.