Hey everyone, seems like you are enjoying this series. Now let's get started on some frameworks, databases and different solutions that will help us build these kind of apps.
We can of course build the whole system we discussed in the last blog on our own like Linear does (https://youtu.be/WxK11RsLqp4?t=2181). But in this series we will learn how to do the same with Replicache.
Before discussing in detail about Replicache let's discuss a few other alternatives in the market that let you do achieve this kinda of a setup like
RxDB https://rxdb.info/
Firebase Real time database
WatermelonDB https://watermelondb.dev/docs/CRUD
Couch DB https://couchdb.apache.org/
.... and many more
You can do your research as in which one of these frameworks/databases suits your use case the best. I personally liked 2 of them RxDB and Replicache. Below is a brief description of both tools
RxDB: is a observable, replicating and local first javascript database which your client application can use to store data first. Then you have to implement backend sync with RxDB plugins.
Replicache: is a client-side sync framework for building realtime, collaborative, local-first web apps. It claims to work with most backend stacks. In contrast to other local first tools, replicache does not work like a local database. Instead it runs on so called mutators
that unify behaviour on the client and server side. So instead of implementing and calling REST routes on both sides of your stack, you will implement mutators that define a specific delta behaviour based on the input data. To observe data in replicache, there are subscriptions
that notify your frontend application about changes to the state. Replicache can be used in most frontend technologies like browsers, react/remix and react native.
Now let's dive deep into how Replicache works and how we can build amazing real time sync apps using this framework.
BASIC FLOW OF REPLICACHE
Client Application:
Mutators:
mutators are functions that transactionally read and write keys and values to replicache which intern updates the underlying indexDB optimistically
on every mutation call on client it sends a network request to the push endpoint to sync it with the server datastore
We will do a deep dive on how we can write mutators on our client and server and how to call them but below are some screenshots of how your final mutator definitions will look like and they will be invoked
tx --> is a Read/Write Transaction client exposed by replicache which writes/reads data to and from the indexDB
rep --> replicache client of your application where all the mutators are registered and can be invoked from anywhere in your application
Subscriptions:
Subscriptions are how app is notified about changes made to the indexDB by replicache.
You can subscribe to the list of todos of a user and then directly render it to the client
In the above pattern whenever the indexDB state of todos change subscription re-runs and get's your client updated state and triggers a re-render
we will go through a detailed code flow later these screenshots are just to demonstrate how the final code will end up looking
Backend Server: backend server has a database containing the actual data and is the source of truth for all the clients. Replicache supports all backend stacks but expects you to write your api routes (2 of them which it uses) in a very specific format.
Push: Replicache makes a request to this endpoint on every client mutator execution. This endpoint has a corresponding mutator for each one of your client mutators(example: if you have a todoCreate client mutator, you need to have a todoCreate server mutator). Server mutators are responsible to execute the mutation server side and actually create the todo inside the database server rather than just storing them optimistically inside the indexDB like the client mutator. We will write very robust typescript to help us write mutators that match on client and server
server todoCreate mutator
Pull: Replicache fetches latest changes from the database server from the pull endpoint. This endpoint return a patch update that local replicache does not have inside the indexDB, in the form of a diff. Replicache client makes a request to this endpoint automatically every 60seconds to keep the UI upto date. But we want interactions to be real time which means pull should be triggered after every push. (imagine you creating a task on a team todo app and you team mate sees it after 60seconds that's not what we want) 🤔
Poke: Poke comes to rescue in the above situation, server sends a poke after every push to all the clients that need to get the state change from that given push. Poke is just a contentless server sent event delivered over pub-sub which gives the client a notification to pull.
just a empty publish by a web-socket service called ably to a channel that current userId has access to.
To Summarise:
User takes an action on your app invoking a client side mutator
Mutator modifies local Replicache(indexDB) and subscription fires to update UI
In background mutation is sent in batches to the push endpoint to execute them on the server on a canonical datastore
On complete server pokes the concerned clients to pull "because something has changed"
Clients pull and get the diff of the change --> local Replicache(indexDB) is update automatically by replicache hence triggering a re-render
Note:
It is extremely important to ensure that your datastore and/or the way you use it guarantees the consistency and isolation properties required for Replicache to work as designed. These properties are:
the effects of a transaction are revealed atomically
within a transaction, reads are consistent, ie, reading the same item twice always results in the same value, unless changed within the transaction
a transaction sees the effects of all previously committed transactions
For example, MySQL's SERIALIZABLE isolation level provides these guarantees.
Key Terminologies
Client:
An instance of replicache class in memory is called a client, identified by a randomly generated ID called
clientID
.Typically one client per browser tab is short lived being instantiated for the lifetime of the application in the tab
const rep = new Replicache({ name: `rep:${userId}`, // used by clientGroup to create an indexDB ... })
ClientGroup:
Set of clients that share data locally, changes made by one client is visible to all clients in that client group even when the user is offline.
Identified by random ID called
clientGroupID
Under normal circumstances all clients within the same browser profile belong to the one
clientGroupID
The client group sits on top of indexDB identified by the
name
parameter to theReplicache
constructor. All clients in the group that have the samename
, share access to the same cache.Each client keeps an ordered map of key/value pairs which is shown above.
REPLICACHE SYNC DEEP-DIVE (optional reading)
Local execution + Push
When a mutator is invoked, Replicache applies its changes to the local Client View.
It also queues a corresponding pending mutation record to be pushed to the server.
And this record is persisted in case the tab closes before it can be pushed.
When created, a mutation is assigned a mutation id, a sequential integer uniquely identifying the mutation in this client. The mutation id also describes a causal order to mutations from this client, and that order is respected by the server.
Every client has a lastMutationID stored inside our database, when we execute a mutation from a given client, we fetch that client first inside the push route
We compare the current
mutationID
that we are processing with thenextMutationID
that we are expecting which is basically client.lastMutationID + 1if current
mutationID
is less thannextMutationID
it is already applied so skipif more this mutation is coming from the future 🤯 (might happen in rare cases in dev env not in prod)
Pending mutations are sent in batches to the push endpoint on your server
Mutations carry exactly the information the server needs to execute the mutator that was invoked on the client.
That is,
mutationID
theclientID
that the mutation was created on, the name of the mutator invoked, and its arguments.The push endpoint executes the pushed mutations in order by executing the named mutator with the given arguments, applying the mutation effects to the server datastore.
It also updates the corresponding
lastMutationID
for the client that is pushing.
Speculative Execution and Confirmation
It is important to understand that the push endpoint is not necessarily expected to compute the same result as the mutator on the client.
This is intentional. The server may have newer or different data than the client (due to new pushes from other clients working on the same dataset, like co-workers in an organisation).
That’s fine — the pending mutations applied on the client are speculative or optimistic until they are applied on the server.
In Replicache, the server is authoritative. The client-side mutators create speculative results, then the mutations are pushed and executed by the server, creating confirmed, canonical results.
The confirmed results are later pulled by the client, with the server-calculated state taking precedence over the speculative result.
This precedence happens because, once confirmed by the server, a speculative mutation is no longer re-run by the client on the new server state (a mutation that has already happened on server is not pushed again).
Pull
Periodically, Replicache requests an update to the Client View by calling the pull endpoint.
The pull request contains a
cookie
and aclientGroupID
, and the response contains a newcookie
, apatch
, and a set oflastMutationIDChanges
.The
lastMutationIDChanges
in the response tell Replicache which mutations have been confirmed by the server for each client in the client group.Since clients in the same client group share the same cache (indexDB), as discussed before, mutations with IDs less than those sent from the server in the
lastMutationIDChanges
are not sent to the server again.
Rebase
Once the client receives a pull response, it needs to apply the patch to the local state to bring the client's state up to date with that of the server.
But it can’t apply the patch to the current local state, because that state can include changes caused by pending mutations.
It's not clear what a general strategy would be for applying the patch on top of local changes.
So it doesn't. Instead, hidden from the application's view, it rewinds the state of the Client View to the last version it got from the server, applies the patch to get to the state the server currently has, and then replays any pending mutations on top (just like a git rebase).
It then atomically reveals this new state to the app, which triggers subscriptions and the UI to re-render.
In order to support the capability to rewind the Client View and apply changes out of view of the app, Replicache is modeled under the hood like git. It maintains historical versions of the Client View and, like git branches, has the ability to work with a historical version of the Client View behind the scenes. So when the client pulls new state from the server, it forks from the previous Client View received from the server, applies the patch, rebases (re-runs) pending mutations, and then reveals the new branch to the app.
Conflict Resolution
The potential for merge conflicts is unavoidable in a system like Replicache. Clients and the server operate on the key-value space independently.
The potential for merge conflicts arises in Replicache in two places.
When a speculative mutation from a client is applied on the server. The state on the server that the mutation operates on could be different from the state it was originally applied on in the client.
When a speculative mutation is rebased in the client. The mutation could be re-applied on state that is different from the previous time it ran.
Replicache handles conflict resolution by allowing developers to define their own rules through code. Mutators are JavaScript functions, so they can implement any conflict resolution strategy that fits the application. Here are two examples:
A
reserveRoom
mutator might first find a room in theAVAILABLE
state and mark it asRESERVED
for the user. Later, when this mutator runs on the server, it might find the room is alreadyRESERVED
for another user. In this case, the server mutator takes a different path: it leaves the room reservation as is and sets a flag in the user's state indicating their booking attempt failed. When the user's client pulls the latest state, this flag is included in the Client View. The app likely watches for this flag and updates the UI to show the room as unavailable, notifying the user that the reservation failed.An
addItem
mutator for a Todo app might not need any conflict resolution. It can simply add the new item to the end of the list, regardless of the current state of the list.
So this is all you had to know about replicache, now let's write our actual app in the upcoming blogs.