Ronit's Engineering Directory
Initialise replicache with a robust production ready type-system

Initialise replicache with a robust production ready type-system

Blog-4 of the 8 Blog series

Ronit Panda's photo
Ronit Panda
·

13 min read

Hello everyone!, you have made a long way through so pat yourself on the back and let's begin. In this blog we will work on setting up replicache inside our NextJS app and then setting up a robust type system inside our monorepo and understand why we need such a type system in the first place. Including that we will also bootstrap a basic set of push and pull endpoints.
I will coding on a monorepo bootstrapped with turbo repo and will attach the link to the GitHub repository so you can browse the full codebase and follow along

Link to Github Repository: https://github.com/rtpa25/replicache-cvr [everything about starting up the repository and individual package description is provided in the readme]

Disclaimer: I will not be giving detailed steps of every single part of the codebase, even if it's just a todo app, a lot of code has gone to it. And detailing every aspect of is beyond the scope of this blog series and will become too long, so I will be sticking strictly to the parts that are required for you to have replicache further details you can easily find inside my repository

Just to give a brief overview of the structure of the monorepo

  1. Apps

    • web:

      • Next app that is the frontend of the application

      • styled with tailwind and nextUI

      • react query for data fetching and zustand for state management

    • api:

      • Simple express app

  2. Packages

    • eslint-config:

      • contains es-lint config for different types of projects inside the monorepo

    • lib:

      • includes initialisation of various services like redis, sockets (Ably)

      • and other utility functions like logger, jwt etc

    • models

      • prisma schema and migrations, also exports the whole prisma client, so any types that are needed are imported from the models package

      • zod schemas that are shared across apps

      • all the type-system that we have around mutators

      • I also export replicache from this package, so as not to manage multiple versions of this lib over my client and server

      • various other types and utility classes

    • typescript-config

      • consists of ts-config files for different environments

Let's Code!!

First thing is to setup replicache inside your client side app, but before that make sure to get your self a replicache license key by running the below command

npx replicache@latest get-license

so create a use-replicache hook inside your react app. and let's instantiate replicache

import { nanoid } from "nanoid";
import * as React from "react";
import { create } from "zustand";
import { immer } from "zustand/middleware/immer";

import { Replicache } from "@repo/models";

import { api } from "~/lib/api";

import { env } from "~/env";
import { useUser } from "~/hook/user-user";

type State = {
  // M is a type for all mutators in the app (client and server)
  rep: Replicache | null;
};

type Actions = {
  setRep: (rep: Replicache) => void;
};

const useReplicacheStore = create<State & Actions>()(
  immer((set) => ({
    rep: null,
    setRep: (rep) => set({ rep }),
  })),
);

export const useReplicache = () => {
  return { rep: useReplicacheStore((state) => state.rep) };
};

export const useLoadReplicache = () => {
  const { data } = useUser();
  const user = data?.user;
  const { rep, setRep } = useReplicacheStore((state) => state);

  React.useEffect(() => {
    if (!user?.id) return;
    const iid = nanoid();

    const r = new Replicache({
      name: user.id,
      licenseKey: env.NEXT_PUBLIC_REPLICACHE_LICENSE_KEY, 
    });

    setRep(r);

    return () => {
      void r.close();
    };
  }, [setRep, user?.id]);
};

At this point we have a bare minimum replicache client setup inside our next app, as discussed one of the main fundamental working principles of how replicache works are mutators.

We define both client-mutators and server-mutators, both should work on the same set of user-provided arguments. Client-mutators need to be registered with replicache and when client mutators are called replicache makes a network request to the push endpoint which consists the information about the mutation that occurred.

Now it's our server's push endpoints responsibility to run the server-mutator from the mutation name and sync with our backend database and other clients.

So from here you can see we need same name for mutators on both client and server and pre-defined arguments because endpoint should be able to process any mutation whatsoever (there can be 100s of mutators inside a large scale production application)

So let's move to our models package which contains a folder called mutator

inside there is a mutator.model.ts file which looks something like this

import { type WriteTransaction } from "replicache";

import { type TodoMutators } from "./todo.model";
import { type TransactionalPrismaClient } from "../prisma-client";

export type MutatorContext = {
  userId: string;
};
export enum MutatorType {
  CLIENT = "client",
  SERVER = "server",
}
export type MutatorTypes = MutatorType.CLIENT | MutatorType.SERVER;

/*
* We need a transactional prisma client here for our backend mutators
* because as discussed earlier in the last blog
* the consistency and isolation properties of a serilizable transaction
* is required for Replicache to work as designed
*/
export type Mutator<Type = MutatorTypes, Args = object> = Type extends MutatorType.CLIENT
  ? (tx: WriteTransaction, args: Args) => Promise<void>
  : (
      body: {
        tx: TransactionalPrismaClient;
        args: Args;
      } & MutatorContext,
    ) => Promise<void>;

// As the number of mutators grows, we can add more types here. like `& MoreMutators<Type>`
export type M<Type = MutatorTypes> = TodoMutators<Type>;

now let's write the type for TodoMutators

in the same folder go to todo.model.ts

import { type Mutator, type MutatorTypes } from "./mutator.model";
import {
  type TodoCreateType,
  type TodoDeleteType,
  type TodoUpdateType,
} from "../schemas/todo.schema";

export type TodoMutators<Type = MutatorTypes> = {
  todoCreate: Mutator<Type, TodoCreateType>;
  todoUpdate: Mutator<Type, TodoUpdateType>;
  todoDelete: Mutator<Type, TodoDeleteType>;
};

Moving to the todo.schema.ts file inside the schema directory where we write the input schemas for our various mutators

I am using zod for schema validation you can use any other library of your choice

import { type Prisma } from "@prisma/client";
import { z } from "zod";

export type TodoType = Prisma.TodoGetPayload<object>;

export const todoCreateSchema = z.object({
  id: z.string().nanoid(),
  text: z.string().min(1).max(100),
});
export type TodoCreateType = z.infer<typeof todoCreateSchema>;

export const todoUpdateSchema = z.object({
  id: z.string().nanoid(),
  text: z.string().min(1).max(100).optional(),
  completed: z.boolean().optional(),
});
export type TodoUpdateType = z.infer<typeof todoUpdateSchema>;

export const todoDeleteSchema = z.object({
  id: z.string().nanoid(),
});
export type TodoDeleteType = z.infer<typeof todoDeleteSchema>;

once these are setup let's go to our next app and update the typings for our replicache client inside the use-replicache hook

import { nanoid } from "nanoid";
import * as React from "react";
import { create } from "zustand";
import { immer } from "zustand/middleware/immer";

import { type M, type MutatorType, Replicache } from "@repo/models";

import { api } from "~/lib/api";

import { env } from "~/env";
import { useUser } from "~/hook/user-user";

type State = {
  // M is a type for all mutators in the app (client and server)
  rep: Replicache<M<MutatorType.CLIENT>> | null; // these are the additions
};

type Actions = {
  setRep: (rep: Replicache<M<MutatorType.CLIENT>>) => void;
};

const useReplicacheStore = create<State & Actions>()(
  immer((set) => ({
    rep: null,
    setRep: (rep) => set({ rep }),
  })),
);

export const useReplicache = () => {
  return { rep: useReplicacheStore((state) => state.rep) };
};

export const useLoadReplicache = () => {
  const { data } = useUser();
  const user = data?.user;
  const { rep, setRep } = useReplicacheStore((state) => state);

  React.useEffect(() => {
    if (!user?.id) return;
    const iid = nanoid();

    const r = new Replicache({
      name: user.id,
      licenseKey: env.NEXT_PUBLIC_REPLICACHE_LICENSE_KEY,
    });

    setRep(r);

    return () => {
      void r.close();
    };
  }, [setRep, user?.id]);
};

Once you have added the typings to your global state you must be getting typescript errors on the setRep line because you have yet to define and hook up the actual client replicache mutators, so let's get going

inside your web app create a new folder called mutators inside your src/ directory

and add an index.ts file

import { type M, type MutatorType } from "@repo/models";

import { clientTodoMutators } from "~/mutators/todo.mutator";

export const clientMutators: (userId: string) => M<MutatorType.CLIENT> = (userId) => ({
  ...clientTodoMutators(userId),
});

you might be thinking why do we need to build this higher order function userID as an argument, this is because we frequently use userId inside these mutators and passing it as a prop here just makes more sense, than individually passing to every single mutator.

now create the actual todo.mutator file and your client todo mutators

import {
  type MutatorType,
  type TodoMutators,
} from "@repo/models";

export const clientTodoMutators: (userId: string) => TodoMutators<MutatorType.CLIENT> = (
  userId,
) => ({
  async todoCreate(tx, args) {
    throw new Error("method not implemented");
  },
  async todoDelete(tx, args) {
    throw new Error("method not implemented");
  },
  async todoUpdate(tx, args) {
    throw new Error("method not implemented");
  },
});

now let's finally register the mutators inside replicache so move to use-replicache hook file

import { nanoid } from "nanoid";
import * as React from "react";
import { create } from "zustand";
import { immer } from "zustand/middleware/immer";

import { type M, type MutatorType, Replicache } from "@repo/models";

import { api } from "~/lib/api";

import { env } from "~/env";
import { useUser } from "~/hook/user-user";
import { clientMutators } from "~/mutators";

type State = {
  // M is a type for all mutators in the app (client and server)
  rep: Replicache<M<MutatorType.CLIENT>> | null;
};

type Actions = {
  setRep: (rep: Replicache<M<MutatorType.CLIENT>>) => void;
};

const useReplicacheStore = create<State & Actions>()(
  immer((set) => ({
    rep: null,
    setRep: (rep) => set({ rep }),
  })),
);

export const useReplicache = () => {
  return { rep: useReplicacheStore((state) => state.rep) };
};

export const useLoadReplicache = () => {
  const { data } = useUser();
  const user = data?.user;
  const { rep, setRep } = useReplicacheStore((state) => state);

  React.useEffect(() => {
    if (!user?.id) return;
    const iid = nanoid();

    const r = new Replicache({
      name: user.id,
      licenseKey: env.NEXT_PUBLIC_REPLICACHE_LICENSE_KEY,
      mutators: clientMutators(user.id),
    });

    setRep(r);

    return () => {
      void r.close();
    };
  }, [setRep, user?.id]);
};

Now before actually moving forward implementing the client and the server mutators let's understand how replicache stores our data inside the browser cache.

This depends on how you want to store the data inside IndexDB, replicache also exposes a helpful library called @rocicorp/rails, but in this series we will take full control and do it ourselves

there is a file inside the models package called idb-key.ts

import { type ReadonlyJSONValue } from "replicache";

/** Removes undefined value, and joins string to '/' */
function constructIDBKey(arr: (string | null | undefined | number)[]) {
  return arr.filter((i) => i !== undefined).join("/");
}

export const IDB_KEY = {
  /**
   * Last param should be '', to make it `/todo/` rather than `/todo`
   *
   * @example
   * await tx.scan(IDB_KEY.TODO({})))
   * 'todo/' --> list of todos
   *
   * await tx.set(IDB_KEY.TODO({id: '1'})))
   * 'todo/1' --> todo with id '1'
   */
  TODO: ({ id = "" }: { id?: string }) => constructIDBKey(["todo", id]),
};
export type IDBKeys = keyof typeof IDB_KEY;

/**
 * Normalize data to readonly json value
 * to be stored onto index db
 *
 * @returns
 */
export function normalizeToReadonlyJSON(args: unknown) {
  return args as ReadonlyJSONValue;
}

here we define the keys of the value that we want to store inside indexDB that replicache manages for us.

When we write the pull endpoint you will be able to get more clarity of how this is known to replicache but for now let's keep on coding

hop back to nextjs app and todo.mutator file

import {
  IDB_KEY,
  type MutatorType,
  normalizeToReadonlyJSON,
  type TodoMutators,
} from "@repo/models";

import { TodoManager } from "~/managers/todo.manager";

export const clientTodoMutators: (userId: string) => TodoMutators<MutatorType.CLIENT> = (
  userId,
) => ({
  async todoCreate(tx, args) {
    const todo = TodoManager.createTodo({
      ...args,
      userId,
    });

    await tx.set(IDB_KEY.TODO({ id: args.id }), normalizeToReadonlyJSON(todo));
  },
  async todoDelete(tx, args) {
    const todo = await TodoManager.getTodoById({
      id: args.id,
      tx,
    });

    if (todo === undefined) {
      throw new Error(`Todo with id ${args.id} not found`);
    }

    await tx.del(IDB_KEY.TODO({ id: args.id }));
  },
  async todoUpdate(tx, args) {
    const todo = await TodoManager.getTodoById({
      id: args.id,
      tx,
    });

    if (todo === undefined) {
      throw new Error(`Todo with id ${args.id} not found`);
    }

    // should never happen, as in every user's indexDB resides data that only that user has access to
    if (todo.userId !== userId) {
      throw new Error(`Todo with id ${args.id} not found`);
    }

    const updatedTodo = TodoManager.updateTodo({
      args,
      oldTodo: todo,
    });
    await tx.set(IDB_KEY.TODO({ id: args.id }), normalizeToReadonlyJSON(updatedTodo));
  },
});

managers/todo.manager.ts --> helper class (like a service layer but for the web)

import {
  IDB_KEY,
  type ReadTransaction,
  type TodoCreateType,
  type TodoType,
  type TodoUpdateType,
} from "@repo/models";

import { normalizeReplicacheData } from "~/lib/replicache";

export class TodoManager {
  static createTodo(args: TodoCreateType & { userId: string }) {
    const todo: TodoType = {
      id: args.id,
      title: args.text,
      rowVersion: 0,
      completed: false,
      userId: args.userId,
    };
    return todo;
  }

  static async getTodoById({ id, tx }: { id: string; tx: ReadTransaction }) {
    const todo = (await tx.get(IDB_KEY.TODO({ id: id }))) as TodoType | undefined;
    return todo;
  }

  static updateTodo({ oldTodo, args }: { oldTodo: TodoType; args: TodoUpdateType }) {
    return {
      ...oldTodo,
      title: args.text,
      completed: args.completed,
    } as TodoType;
  }

  static async getallTodos({ tx }: { tx: ReadTransaction }) {
    const _todos = await tx
      .scan({
        prefix: IDB_KEY.TODO({}),
      })
      .entries()
      .toArray();

    const todos = normalizeReplicacheData<TodoType>(_todos);
    return todos;
  }
}

With this we complete writing the client mutators which will set data to the local indexDB when called from the code which will look like this

rep.mutate.todoUpdate({
    id: todo.id,
    text,
});

rep.mutate.todoUpdate({
    id: todo.id,
    completed: true
});

rep.mutate.todoDelete({ id: todo.id });

because we have hooked up the proper type system we will get auto complete for all these mutation arguments

I know this has spanned quite long but a little bit more and we further continue in the next blog

once we are done with the mutators the user can call these functions, but once called replicache will try to call to our push endpoint which we have not implemented yet and not even hooked up to replicache so let's get that done as the last part of this blog

again hop back to the use-replicache.tsx file

import { AxiosError } from "axios";
import { nanoid } from "nanoid";
import * as React from "react";
import { create } from "zustand";
import { immer } from "zustand/middleware/immer";

import { type M, type MutatorType, type PullResponseOKV1, Replicache } from "@repo/models";

import { api } from "~/lib/api";

import { env } from "~/env";
import { useUser } from "~/hook/user-user";
import { clientMutators } from "~/mutators";

type State = {
  // M is a type for all mutators in the app (client and server)
  rep: Replicache<M<MutatorType.CLIENT>> | null;
};

type Actions = {
  setRep: (rep: Replicache<M<MutatorType.CLIENT>>) => void;
};

const useReplicacheStore = create<State & Actions>()(
  immer((set) => ({
    rep: null,
    setRep: (rep) => set({ rep }),
  })),
);

export const useReplicache = () => {
  return { rep: useReplicacheStore((state) => state.rep) };
};

export const useLoadReplicache = () => {
  const { data } = useUser();
  const user = data?.user;
  const { rep, setRep } = useReplicacheStore((state) => state);

  React.useEffect(() => {
    if (!user?.id) return;
    const iid = nanoid();

    const r = new Replicache({
      name: user.id,
      licenseKey: env.NEXT_PUBLIC_REPLICACHE_LICENSE_KEY,
      mutators: clientMutators(user.id),
      schemaVersion: env.NEXT_PUBLIC_SCHEMA_VERSION ?? "1",
      // pushURL: "",
      // pullURL: "",
    });

    r.pusher = async (opts) => {
      try {
        const response = await api.replicachePush(opts, iid);
        return {
          httpRequestInfo: {
            httpStatusCode: response.status,
            errorMessage: "",
          },
        };
      } catch (error) {
        if (error instanceof AxiosError)
          return {
            httpRequestInfo: {
              httpStatusCode: error.status ?? 500,
              errorMessage: error.message,
            },
          };
        return {
          httpRequestInfo: {
            httpStatusCode: 500,
            errorMessage: "Unknown error",
          },
        };
      }
    };

    r.puller = async (opts) => {
      try {
        const response = await api.replicachePull(opts, iid);
        return {
          response: response.data as PullResponseOKV1,
          httpRequestInfo: {
            errorMessage: "",
            httpStatusCode: response.status,
          },
        };
      } catch (error) {
        if (error instanceof AxiosError)
          return {
            httpRequestInfo: {
              httpStatusCode: error.status ?? 500,
              errorMessage: error.message,
            },
          };
        return {
          httpRequestInfo: {
            httpStatusCode: 500,
            errorMessage: "Unknown error",
          },
        };
      }
    };

    setRep(r);

    return () => {
      void r.close();
    };
  }, [setRep, user?.id]);
};

as you can understand from this code there are two ways of letting replicache know about our push and pull endpoints

  1. push & pull url :- not really good when you need to pass specific headers or cookies or need to do some error handling

  2. pusher and puller :- is the setup we are using because we want to handle errors and send auth tokens to our backend

NOTE: I have implemented a dummy auth system inside this mono repo as well but will not cover in this blog series because it is not directly linked and quite easy to understand if you look how I implement in a code level.

Not a secure auth system should not be used in real production apps maybe a blog on that in the near future.

Now finally we have setup majority of the client side configuration. In the upcoming blog we learn what are the recommended ways of structuring our backend push and pull route according to replicache, further we will pick the most scalable and complex strategy and proceed with writing our endpoints.

TypeScriptNext.jsNode.jsExpressmonorepoturboreporealtimerealtime apps