Quickstart

The smallest pipeline in this repository is examples/hello. It has one source and one model. The source emits three customers. The model reads them back and lowercases the email address. You will run the source on your machine, then start the engine and run the whole graph.

You need Node 22, pnpm, and Docker. Docker is for the engine. A local run of one asset does not start it.

Build the CLI

From the repository root:

pnpm install
pnpm -r build

Inside this repository the command is pnpm exec puppi.

Write the pipeline

examples/hello/src/pipeline.ts:

import { model, source } from "@puppi/sdk";
import { z } from "zod";

const Customer = z.object({
  id: z.string(),
  email: z.string(),
  plan: z.string(),
});

export const customers = source({
  key: "demo/customers",
  schema: Customer,
  run: async (ctx) => {
    await ctx.emit([
      { id: "cus_1", email: "ada@example.com", plan: "pro" },
      { id: "cus_2", email: "grace@example.com", plan: "free" },
      { id: "cus_3", email: "alan@example.com", plan: "pro" },
    ]);
  },
});

export const coreCustomer = model({
  key: "core/customer",
  schema: Customer,
  run: async (ctx) => {
    const upstream = ctx.input("demo/customers");

    const { rows } = await ctx.query<z.infer<typeof Customer>>(
      `select id, lower(email) as email, plan from ${upstream} order by id`,
    );

    await ctx.emit(rows);
  },
});

ctx.input("demo/customers") is the edge from the model to the source. The value it returns is the table name, so the SQL can name that table. schema is the contract the engine checks after each write.

Point the pipeline at storage

examples/hello/puppi.config.ts names where rows are stored:

import { defineConfig } from "@puppi/sdk";

export default defineConfig({
  entry: "src/pipeline.ts",
  out: ".",
  defaultStorage: "warehouse",
  connections: {
    warehouse: { type: "postgres" },
  },
});

Run one asset

cd examples/hello
pnpm exec puppi run --local demo/customers

The command loads the bundle, calls that asset’s run, and prints the rows, the columns it observed, and the schema-contract verdict. The heading on the output says the engine is not involved and nothing is written. There is no materialization, no latest, and no lineage yet.

--limit changes how many rows are printed. The default is 10. A model has no table until you pass one:

pnpm exec puppi run --local core/customer --input demo/customers=<table>

Run it on the engine

The compose stack’s API token is dev. examples/hello/bindings.local.json names the environment hello. Export both before the first command:

export PUPPI_API_TOKEN=dev
export PUPPI_ENV=hello

pnpm exec puppi dev
pnpm exec puppi deploy
pnpm exec puppi run
pnpm exec puppi assets

puppi dev starts Postgres and the engine, waits until /health answers, and creates the hello environment from bindings.local.json. Run it again after a restart.

puppi deploy builds the project, uploads the bundle, and points the environment at that deployment.

puppi run starts a run of the whole graph and waits until it finishes. puppi assets prints each asset’s latest materialization and whether it is stale.

The engine listens on http://127.0.0.1:4000. The UI is on that same port.

Start a project of your own

puppi init writes a project of this shape into the current directory, or into a path you pass. The scaffold’s assets are example/people and core/person.

puppi init my-pipeline
puppi init --template hello

The command writes puppi.config.ts, src/pipeline.ts, bindings.local.json, package.json, tsconfig.json, and .gitignore. It leaves existing files in place unless you pass --force. --template copies hello, atlas, or pokedex out of this checkout.

The scaffold’s package.json depends on @puppi/sdk, @puppi/cli, and @puppi/executor-js. Those packages are not on npm. The commands above work from this checkout, where pnpm exec puppi is already built.

The CLI reference lists every flag.