Lakebase Off-Platform

Prerequisites

Verify these Databricks workspace features are enabled before starting. If any check fails, ask your workspace admin to enable the feature.

• Databricks CLI authenticated. Run databricks auth profiles and confirm at least one profile shows Valid: YES. If none do, authenticate with databricks auth login --host <workspace-url> --profile <PROFILE>.
• Lakebase Postgres available in the workspace. Run databricks postgres list-projects --profile <PROFILE> and confirm the command succeeds (an empty list is fine — you are about to create the first project). A not enabled or permission error means Lakebase is not available to this identity.

This template collects the environment variables needed to reach Lakebase from an app running outside Databricks App Platform. Verify these Databricks workspace features are enabled before starting.

• Databricks CLI authenticated. Run databricks auth profiles and confirm at least one profile shows Valid: YES. If none do, authenticate with databricks auth login --host <workspace-url> --profile <PROFILE>.
• Lakebase Postgres available. Run databricks postgres list-projects --profile <PROFILE> and confirm the command succeeds. A not enabled error means Lakebase is not available to this identity.
• A provisioned Lakebase project. Complete the Create a Lakebase Instance template first. You will read connection values from its branch, endpoint, and database.
• Machine-to-machine OAuth for production (optional). If you plan to run in production with a service principal, have DATABRICKS_CLIENT_ID / DATABRICKS_CLIENT_SECRET ready for that service principal. For local development, a workspace token from databricks auth token --profile <PROFILE> is sufficient.

This template fetches and caches Lakebase Postgres credentials from a Node.js process. Verify these Databricks workspace features are enabled before starting.

• Databricks CLI authenticated. Run databricks auth profiles and confirm at least one profile shows Valid: YES. If none do, authenticate with databricks auth login --host <workspace-url> --profile <PROFILE>.
• Lakebase Postgres available. Run databricks postgres list-projects --profile <PROFILE> and confirm the command succeeds. A not enabled error means Lakebase is not available to this identity.
• A provisioned Lakebase project. Complete the Create a Lakebase Instance template first so you have a LAKEBASE_ENDPOINT resource path to pass to the credentials API.
• An env management setup. Complete the Lakebase Env Management for Off-Platform Apps template first — this template imports the validated env module and expects DATABRICKS_HOST, LAKEBASE_ENDPOINT, and either DATABRICKS_TOKEN or DATABRICKS_CLIENT_ID + DATABRICKS_CLIENT_SECRET to be set.

This template connects an off-platform Node.js app (e.g. AWS, Vercel, Netlify) to Lakebase Postgres. Verify these Databricks workspace features are enabled before starting.

• Databricks CLI authenticated. Run databricks auth profiles and confirm at least one profile shows Valid: YES. If none do, authenticate with databricks auth login --host <workspace-url> --profile <PROFILE>.
• Lakebase Postgres available. Run databricks postgres list-projects --profile <PROFILE> and confirm the command succeeds. A not enabled error means Lakebase is not available to this identity.
• A provisioned Lakebase project. Complete the Create a Lakebase Instance template first so you have an endpoint host, database, and endpoint resource path available as PGHOST, PGDATABASE, and LAKEBASE_ENDPOINT.
• An env management setup for off-platform auth. Complete the Lakebase Env Management for Off-Platform Apps and Lakebase Token Management templates first — this template imports env and getLakebasePostgresToken from those modules.

Create a Lakebase Instance

Provision a managed Lakebase Postgres project on Databricks and collect the connection values needed by downstream templates.

1. Create a Lakebase project

Create a new Lakebase Postgres project. This provisions a managed Postgres cluster with a default branch and endpoint:

databricks postgres create-project <project-name> --profile <PROFILE>

2. Verify the project resources

Confirm the branch, endpoint, and database were created:

databricks postgres list-branches \
  projects/<project-name> \
  --profile <PROFILE> -o json

databricks postgres list-endpoints \
  projects/<project-name>/branches/production \
  --profile <PROFILE> -o json

databricks postgres list-databases \
  projects/<project-name>/branches/production \
  --profile <PROFILE> -o json

3. Note the connection values

Record these values from the command output above. They are required by the Lakebase Data Persistence template and other Lakebase-dependent templates:

References

• What is Lakebase?
• CLI reference for Lakebase

Lakebase Environment Management for Off-Platform Apps

Define and validate the environment variables needed to connect to Lakebase from apps deployed outside Databricks App Platform (for example on AWS, Vercel, or Netlify).

1. Collect connection values via the Databricks CLI

Every value below can be obtained from the CLI. Run each command and record the result.

Workspace host (DATABRICKS_HOST):

databricks auth profiles

Use the Host column for your profile (e.g. https://dbc-xxxxx.cloud.databricks.com).

Lakebase endpoint and Postgres host (LAKEBASE_ENDPOINT, PGHOST):

databricks postgres list-endpoints \
  projects/<project-name>/branches/production \
  --profile <PROFILE> -o json

• LAKEBASE_ENDPOINT = the name field (e.g. projects/<project>/branches/production/endpoints/primary)
• PGHOST = the status.hosts.host field

Postgres database name (PGDATABASE):

databricks postgres list-databases \
  projects/<project-name>/branches/production \
  --profile <PROFILE> -o json

Use the status.postgres_database field (typically databricks_postgres).

Postgres user (PGUSER):

For local development with token auth, this is your Databricks email:

databricks current-user me --profile <PROFILE> -o json

Use the userName field.

For production with M2M auth, this is the service principal's application ID used for DATABRICKS_CLIENT_ID.

Auth credentials:

For local development, get a short-lived workspace token:

databricks auth token --profile <PROFILE> -o json

Use the access_token field for DATABRICKS_TOKEN. This token expires after about one hour; the Token Management template covers automated refresh.

For production, use OAuth M2M credentials (DATABRICKS_CLIENT_ID + DATABRICKS_CLIENT_SECRET) from a service principal configured in your workspace.

2. Validate env at startup with Zod

Create src/lib/env.ts. Parsing process.env through a Zod schema on import ensures the app fails fast with a clear error when a variable is missing:

import { z } from "zod";

const baseSchema = z.object({
  DATABRICKS_HOST: z.string().min(1),
  LAKEBASE_ENDPOINT: z.string().min(1),
  PGHOST: z.string().min(1),
  PGPORT: z.coerce.number().default(5432),
  PGDATABASE: z.string().min(1),
  PGUSER: z.string().min(1),
  PGSSLMODE: z.enum(["require", "prefer", "disable"]).default("require"),
  DATABRICKS_TOKEN: z.string().optional(),
  DATABRICKS_CLIENT_ID: z.string().optional(),
  DATABRICKS_CLIENT_SECRET: z.string().optional(),
});

type AppEnv = z.infer<typeof baseSchema>;

function validateAuth(env: AppEnv): AppEnv {
  const hasToken = Boolean(env.DATABRICKS_TOKEN);
  const hasM2M =
    Boolean(env.DATABRICKS_CLIENT_ID) && Boolean(env.DATABRICKS_CLIENT_SECRET);
  if (!hasToken && !hasM2M) {
    throw new Error(
      "Set DATABRICKS_TOKEN or both DATABRICKS_CLIENT_ID and DATABRICKS_CLIENT_SECRET",
    );
  }
  return env;
}

export const env = validateAuth(baseSchema.parse(process.env));

3. Commit an .env.example

Commit this file so every developer (and CI) knows which variables are required. Set the same keys in your hosting platform's secret/env configuration:

DATABRICKS_HOST=https://<workspace-host>
LAKEBASE_ENDPOINT=projects/<project>/branches/production/endpoints/primary
PGHOST=<status.hosts.host from list-endpoints>
PGPORT=5432
PGDATABASE=<status.postgres_database from list-databases>
PGUSER=<your Databricks email or service principal application ID>
PGSSLMODE=require

# Option A: local dev, token auth (expires ~1h, use refresh script)
DATABRICKS_TOKEN=

# Option B: production, M2M auth (service principal)
DATABRICKS_CLIENT_ID=
DATABRICKS_CLIENT_SECRET=

4. Import env early in your server entry point

Import env at the top of your server bootstrap file. The Zod parse runs on import, so any missing or invalid variable throws before the app starts accepting requests.

References

• Databricks OAuth machine-to-machine auth
• Lakebase credentials API

Lakebase Token Management

Fetch, cache, and automatically refresh the short-lived Postgres credentials that Lakebase requires. Supports both direct token auth (local dev) and M2M OAuth (production).

1. Add a token manager for workspace auth and Lakebase credentials

Create src/lib/lakebase/tokens.ts:

import { env } from "@/lib/env";

const REFRESH_BUFFER_MS = 2 * 60 * 1000;

type CachedToken = {
  value: string;
  expiresAt: number;
};

type AuthStrategy =
  | { kind: "token"; token: string }
  | { kind: "m2m"; host: string; clientId: string; clientSecret: string };

let cachedWorkspaceToken: CachedToken | null = null;
let workspaceRefreshPromise: Promise<CachedToken> | null = null;
let cachedLakebaseToken: CachedToken | null = null;
let lakebaseRefreshPromise: Promise<CachedToken> | null = null;

function isFresh(token: CachedToken | null): token is CachedToken {
  return token !== null && Date.now() < token.expiresAt - REFRESH_BUFFER_MS;
}

function authStrategyFromEnv(): AuthStrategy {
  if (env.DATABRICKS_TOKEN) {
    return { kind: "token", token: env.DATABRICKS_TOKEN };
  }
  return {
    kind: "m2m",
    host: env.DATABRICKS_HOST.replace(/\/$/, ""),
    clientId: env.DATABRICKS_CLIENT_ID!,
    clientSecret: env.DATABRICKS_CLIENT_SECRET!,
  };
}

async function fetchWorkspaceTokenM2M(
  host: string,
  clientId: string,
  clientSecret: string,
): Promise<CachedToken> {
  const response = await fetch(`${host}/oidc/v1/token`, {
    method: "POST",
    headers: { "Content-Type": "application/x-www-form-urlencoded" },
    body: new URLSearchParams({
      grant_type: "client_credentials",
      client_id: clientId,
      client_secret: clientSecret,
      scope: "all-apis",
    }),
  });
  if (!response.ok) {
    throw new Error(`M2M token request failed: ${response.status}`);
  }
  const data = (await response.json()) as {
    access_token?: string;
    expires_in?: number;
  };
  if (!data.access_token || !data.expires_in) {
    throw new Error("Invalid M2M token response");
  }
  return {
    value: data.access_token,
    expiresAt: Date.now() + data.expires_in * 1000,
  };
}

async function getWorkspaceToken(auth: AuthStrategy): Promise<string> {
  if (auth.kind === "token") {
    return auth.token;
  }
  if (isFresh(cachedWorkspaceToken)) {
    return cachedWorkspaceToken.value;
  }
  if (!workspaceRefreshPromise) {
    workspaceRefreshPromise = fetchWorkspaceTokenM2M(
      auth.host,
      auth.clientId,
      auth.clientSecret,
    )
      .then((token) => {
        cachedWorkspaceToken = token;
        return token;
      })
      .finally(() => {
        workspaceRefreshPromise = null;
      });
  }
  return (await workspaceRefreshPromise).value;
}

async function fetchLakebaseCredential(
  databricksHost: string,
  workspaceToken: string,
): Promise<CachedToken> {
  const response = await fetch(
    `${databricksHost}/api/2.0/postgres/credentials`,
    {
      method: "POST",
      headers: {
        Authorization: `Bearer ${workspaceToken}`,
        "Content-Type": "application/json",
        Accept: "application/json",
      },
      body: JSON.stringify({ endpoint: env.LAKEBASE_ENDPOINT }),
    },
  );
  if (!response.ok) {
    throw new Error(`Lakebase credential request failed: ${response.status}`);
  }
  const data = (await response.json()) as {
    token?: string;
    expire_time?: string;
  };
  if (!data.token || !data.expire_time) {
    throw new Error("Invalid Lakebase credential response");
  }
  return {
    value: data.token,
    expiresAt: new Date(data.expire_time).getTime(),
  };
}

export async function getLakebasePostgresToken(): Promise<string> {
  if (isFresh(cachedLakebaseToken)) {
    return cachedLakebaseToken.value;
  }
  if (!lakebaseRefreshPromise) {
    lakebaseRefreshPromise = (async () => {
      const auth = authStrategyFromEnv();
      const workspaceToken = await getWorkspaceToken(auth);
      return fetchLakebaseCredential(
        env.DATABRICKS_HOST.replace(/\/$/, ""),
        workspaceToken,
      );
    })()
      .then((token) => {
        cachedLakebaseToken = token;
        return token;
      })
      .finally(() => {
        lakebaseRefreshPromise = null;
      });
  }
  return (await lakebaseRefreshPromise).value;
}

2. Add a script to refresh DATABRICKS_TOKEN for local dev

CLI-issued tokens expire after about one hour. Create scripts/refresh-lakebase-token.ts to write a fresh token into your local env file:

import { execSync } from "node:child_process";
import { readFileSync, writeFileSync, existsSync } from "node:fs";

const envFile = process.argv[2] ?? ".env.local";
const profile = process.env.DATABRICKS_CONFIG_PROFILE ?? "DEFAULT";

const raw = execSync(`databricks auth token --profile "${profile}" -o json`, {
  encoding: "utf-8",
});
const parsed = JSON.parse(raw) as { access_token?: string };
if (!parsed.access_token) {
  throw new Error("Failed to get access token from Databricks CLI");
}

if (!existsSync(envFile)) {
  throw new Error(`Env file not found: ${envFile}`);
}

const content = readFileSync(envFile, "utf-8");
const tokenLine = `DATABRICKS_TOKEN="${parsed.access_token}"`;
const updated = content.includes("DATABRICKS_TOKEN=")
  ? content.replace(/^DATABRICKS_TOKEN=.*/m, tokenLine)
  : `${content.trimEnd()}\n${tokenLine}\n`;

writeFileSync(envFile, updated);
console.log(`Updated DATABRICKS_TOKEN in ${envFile}`);

3. Verify token and credential flow

databricks auth token --profile <PROFILE> -o json

curl -sS -X POST "https://<workspace-host>/api/2.0/postgres/credentials" \
  -H "Authorization: Bearer <workspace-access-token>" \
  -H "Content-Type: application/json" \
  -d '{"endpoint":"projects/<project>/branches/<branch>/endpoints/<endpoint>"}'

The response should include token and expire_time.

References

• Databricks CLI auth token command
• Lakebase credentials API

Drizzle ORM with Lakebase in an Off-Platform App

Connect Drizzle ORM to Lakebase in any Node.js server outside Databricks App Platform. Uses a pg Pool with a password callback for automatic credential refresh.

1. Install Drizzle and the node-postgres driver

npm install drizzle-orm pg
npm install -D drizzle-kit @types/pg tsx

drizzle-orm and drizzle-kit must be on the same major version. If drizzle-kit errors with "This version of drizzle-kit is outdated," check that both packages share the same major (e.g. both 0.x or both 1.x).

2. Create a Lakebase-backed pg pool

Create src/lib/db/pool.ts:

import { Pool, type PoolConfig } from "pg";
import { env } from "@/lib/env";
import { getLakebasePostgresToken } from "@/lib/lakebase/tokens";

function sslConfig(mode: "require" | "prefer" | "disable"): PoolConfig["ssl"] {
  switch (mode) {
    case "require":
      return { rejectUnauthorized: true };
    case "prefer":
      return { rejectUnauthorized: false };
    case "disable":
      return false;
  }
}

export function createLakebasePool(): Pool {
  return new Pool({
    host: env.PGHOST,
    port: env.PGPORT,
    database: env.PGDATABASE,
    user: env.PGUSER,
    password: () => getLakebasePostgresToken(),
    ssl: sslConfig(env.PGSSLMODE),
    max: 10,
    idleTimeoutMillis: 30_000,
    connectionTimeoutMillis: 10_000,
  });
}

3. Define a Drizzle schema

Create src/lib/items/schema.ts with a starter table. Adapt the table name, columns, and types to your domain (e.g. products, orders, users):

import { pgTable, serial, text, timestamp } from "drizzle-orm/pg-core";

export const items = pgTable("items", {
  id: serial("id").primaryKey(),
  name: text("name").notNull(),
  createdAt: timestamp("created_at", { withTimezone: true })
    .notNull()
    .defaultNow(),
});

Add more schema files under src/lib/<domain>/schema.ts as your app grows. The drizzle.config.ts glob (./src/lib/*/schema.ts) picks them all up automatically.

4. Initialize Drizzle with the pool

Create src/lib/db/client.ts. Import every domain schema and spread it into the schema option:

import { drizzle } from "drizzle-orm/node-postgres";
import { createLakebasePool } from "@/lib/db/pool";
import * as itemsSchema from "@/lib/items/schema";

const pool = createLakebasePool();
export const db = drizzle({ client: pool, schema: { ...itemsSchema } });

5. Handle drizzle-kit migrations with a temporary DATABASE_URL

drizzle-kit needs a connection string and cannot use pg password callbacks. Build a one-time URL with a fresh Lakebase credential in scripts/db-migrate.ts:

import { execSync } from "node:child_process";
import { env } from "@/lib/env";
import { getLakebasePostgresToken } from "@/lib/lakebase/tokens";

async function runMigrations() {
  const token = await getLakebasePostgresToken();
  const encodedUser = encodeURIComponent(env.PGUSER);
  const encodedPassword = encodeURIComponent(token);

  const databaseUrl =
    `postgresql://${encodedUser}:${encodedPassword}` +
    `@${env.PGHOST}:${env.PGPORT}/${env.PGDATABASE}` +
    `?sslmode=${env.PGSSLMODE}`;

  execSync("npx drizzle-kit migrate", {
    stdio: "inherit",
    env: { ...process.env, DATABASE_URL: databaseUrl },
  });
}

runMigrations().catch((error) => {
  console.error(error);
  process.exit(1);
});

6. Keep drizzle.config.ts minimal

Lakebase Postgres passwords are short-lived tokens, so there is no static DATABASE_URL to store in .env. The migration script from step 5 builds a temporary URL with a fresh credential and passes it as DATABASE_URL when it shells out to drizzle-kit migrate. Commands like generate only read schema files and never connect, so dbCredentials is optional:

import { defineConfig } from "drizzle-kit";

export default defineConfig({
  schema: "./src/lib/*/schema.ts",
  out: "./src/lib/db/migrations",
  dialect: "postgresql",
  ...(process.env.DATABASE_URL && {
    dbCredentials: { url: process.env.DATABASE_URL },
  }),
});

7. Verify schema generation and migration

Generate reads schema files locally (no database connection):

npx drizzle-kit generate

Migrate fetches a fresh Lakebase credential and applies the generated SQL:

npx dotenv -e .env.local -- npx tsx scripts/db-migrate.ts

tsx does not load .env.local automatically (that is a Next.js-specific behavior), so use dotenv-cli or your framework's env-loading mechanism to inject the variables.

If both commands succeed, your Drizzle schema and Lakebase connection are working.

References

• Drizzle ORM with PostgreSQL
• Lakebase credentials API