How connectors work

A connector is a typed client for an HTTP API. The pipeline calls it through ctx.connection(). Config and secrets arrive from the binding when the attempt starts. The client does not declare assets.

The connectors in this repository are listed in the catalog. Each one is a package named @puppi/adapter-<type> under connectors/.

Register a connector

Pass the module to defineConfig. The module’s default export is the connector definition, and it exports Client. defineConfig registers that class in the bundle the engine runs.

import * as pokeapi from "@puppi/adapter-pokeapi";
import { defineConfig } from "@puppi/sdk";

export default defineConfig({
  connections: {
    warehouse: { type: "postgres" },
    pokeapi,
  },
});

postgres, filesystem, s3, and http are built in. Declare those with { type: "postgres" }. The executor already knows how to construct them. A catalog connector has to travel inside the bundle, which is what passing the module does.

Two connection names may share one module. A second, different client for the same type is refused.

Call it from an asset

The name in ctx.connection() is the connection name from the config, and it has to be a string literal. The CLI records that literal, and the engine resolves a binding for it.

import { source } from "@puppi/sdk";
import { z } from "zod";

const Customer = z.object({
  id: z.string(),
  email: z.string().nullable(),
});

export const customers = source({
  key: "stripe/customers",
  connection: "stripe",
  schema: Customer,
  run: async (ctx) => {
    const stripe = ctx.connection("stripe");
    const rows: z.infer<typeof Customer>[] = [];

    for await (const page of stripe.customers()) {
      for (const customer of page.data) {
        rows.push({ id: customer.id, email: customer.email ?? null });
      }
    }

    await ctx.emit(rows);
  },
});

A list method yields { cursor, data }. cursor is the next page token, or null on the last page. That token resumes a walk of the API. An append source’s cursor field is a different thing: the watermark column on the row.

ctx.connection() returns unknown until puppi build has written puppi-env.d.ts and the project can resolve @puppi/adapter-<type>. The pipeline does not import the client class. The executor constructs it.

Bind credentials

puppi.config.ts names the connection and its type. The binding says where it is reached, and which secrets it uses. puppi dev reads bindings.local.json. puppi env bind writes the same facts to an environment that is already running.

puppi env secret stripe_key --from-env STRIPE_KEY
puppi env bind stripe \
  --config '{"base_url":"https://api.stripe.com"}' \
  --secret-ref api_key=stripe_key

The secret’s value goes in and does not come back. puppi env show lists secret names and versions. The pipeline never receives the value. The client does, as a constructor argument.

A binding may omit api_version when the connector has a default. Any other value, one the package does not list, fails when the client is constructed.

To add a connector this repository does not ship, use puppi connector new.