# Concepts

This page defines the core abstractions in the Sleeper Hit Studio Story API. Understanding these concepts is prerequisite to using the API effectively and interpreting its responses.

---

## Projects

A **StoryProject** is a named workspace that groups sources, plans, and jobs together. Every source, plan, and job belongs to exactly one project. Projects are long-lived — you can attach more sources and generate new plans against the same project over time.

**Key properties:**
- `id` — stable string ID.
- `name` — human-readable label (1–200 chars).
- `description` — optional free-form context (up to 4000 chars).
- `metadata` — arbitrary JSON blob for your own system's cross-reference data.
- `archivedAt` — soft-delete timestamp; archived projects still exist and can be read.
- `sourceCount` — count of non-deleted sources attached to this project.

A project carries no generative logic itself. It is purely a grouping and ownership boundary — every resource created under a project is scoped to the account that owns the project.

---

## Sources

A **StorySource** is a single piece of raw material attached to a project. The planner reads sources and uses them as evidence for the narrative. Sources can be inline text, markdown, a URL to a web page, or a URL pointing at a PDF.

### Source types

| Type | Extraction | Content type required |
|---|---|---|
| `text` | Synchronous — `READY` immediately | n/a (inline) |
| `markdown` | Synchronous — `READY` immediately | n/a (inline) |
| `url` | Async (pg-boss worker) — poll `status` | `text/html`, `text/plain`, `text/markdown`, `application/json`, `application/xhtml+xml` |
| `pdf` | Async (pg-boss worker) — poll `status` | `application/pdf` |

### Source lifecycle (`status`)

```
PENDING  -->  EXTRACTING  -->  READY
                          -->  FAILED
```

For inline types (`text`, `markdown`) the transition is instant and the response already carries `READY`. For async types (`url`, `pdf`) the response returns `PENDING`; poll `GET /story-projects/{projectId}/sources/{sourceId}` until terminal.

### Source fields

When `status: READY` a source carries:
- `byteSize` — byte count of the extracted text.
- `tokenEstimate` — estimated token count (used for quota calculations).
- `contentHash` — SHA-256 of the extracted content (deduplication signal).
- `extractedTextPreview` — first 4000 characters of extracted text.
- `extractedAt` — timestamp when extraction completed.
- `metadata` — type-specific metadata (e.g. `pageCount` for PDF sources).

On `FAILED`:
- `failureCode` — machine-readable reason (e.g. `source_fetch_failed` or `source_incomplete`).
- `failureMessage` — human-readable description of what went wrong.

### Retention policies

Every source has a `retentionPolicy` that controls what the API stores after extraction:

| Policy | What is kept |
|---|---|
| `EXTRACTED_ONLY` (default) | Extracted text + metadata; raw bytes discarded |
| `STORE_ORIGINAL` | Extracted text + raw bytes |
| `METADATA_ONLY` | Metadata only; extracted text discarded after digest |

`METADATA_ONLY` sources can still participate in planning (via the digest), but the planner cannot fall back to raw extracted text if the digest is `SKIPPED`.

### SSRF guards (URL and PDF)

The API blocks sources that could expose internal infrastructure:
- Private IP ranges (RFC 1918), link-local (`169.254.x.x`), loopback, multicast.
- Cloud-metadata endpoints (AWS `169.254.169.254`, GCP `metadata.google.internal`, etc.).
- Maximum 3 redirects followed.
- Maximum 10 MB response body.
- Content-type must match the allowlist for the source type.

Sources that fail these checks land at `FAILED` with `failureCode: source_fetch_failed`.

---

## Source digest

After a source reaches `READY` with extracted text, the API queues a structured digest pass. The digest is produced with structured-output generation — no free-form JSON from the model; the output is validated against a strict schema.

### Digest lifecycle (`digestStatus`)

```
NOT_STARTED  -->  PENDING  -->  RUNNING  -->  READY
                                         -->  FAILED
                                         -->  SKIPPED
```

`SKIPPED` is the normal terminal state when:
- The digest sidecar or provider is not configured in this environment.
- The source has `retentionPolicy: METADATA_ONLY` and no extracted text is available.
- The extracted text is too short to be useful.
- The source is a full-context producer item (`producer` set, `metadata.sourceContextMode: "full"`): its plan reads every character, so it is skipped by design (`digestFailureMessage: "Used in full by its episode plan."`).

A `SKIPPED` digest does not prevent the source from being used in planning. It is terminal: stop polling.

### Digest fields (when `digestStatus: READY`)

```json
{
  "summary": "3-6 sentence plain-prose summary of the source",
  "thesis": "Single sentence capturing the dominant point",
  "claims": [
    {
      "statement": "Acme reduced onboarding time by 60%",
      "kind": "estimate",
      "attribution": "Acme internal study, 2025"
    }
  ],
  "evidence": [
    { "claimIndex": 0, "support": "Quote from the study supporting claim 0" }
  ],
  "gaps": ["Does not address enterprise pricing"],
  "rightsNotes": ["Contains proprietary Acme research — check usage rights"]
}
```

Claim kinds: `fact`, `estimate`, `opinion`, `forecast`, `quotation`, `definition`.

The digest is source-grounded: the model is instructed not to invent claims the source does not make. `rightsNotes` surfaces any usage concerns the model detects (copyright language, restricted data, etc.).

---

## StoryPlan

A **StoryPlan** is the central planning abstraction. It is a source-grounded narrative structure with thesis, evidence, beats, visual system, and audio system — plus a deterministic Studio Credit quote for the requested artifacts.

Plans are generated asynchronously. The API creates the plan record immediately and the generation worker runs in the background.

### Plan lifecycle (`status`)

```
PENDING  -->  RUNNING  -->  REQUIRES_APPROVAL  (always for table_read)
                       -->  APPROVED  (eligible non-table autoApprove, or after /approve)
                       -->  FAILED
                       -->  SUPERSEDED  (when a new plan is created for the same project)
```

`REJECTED` is the terminal state when the account explicitly calls `POST /story-plans/{planId}/reject`.

### The `autoApprove` flag

- `autoApprove: true` — eligible non-table plans transition directly to `APPROVED` on success. The API rejects this flag for every plan containing `table_read`; those plans always retain the exact human-review boundary.
- `autoApprove: false` (default) — the worker lands at `REQUIRES_APPROVAL`. Inspect `plan.plan` (the narrative envelope) and `plan.quote` (the credit estimate), show the exact plan to the user, then call `POST /story-plans/{planId}/approve` with `{ "userConfirmed": true }` only after explicit approval (or call `/reject`).

### The plan envelope

When `status` is `APPROVED` (or `REQUIRES_APPROVAL`), the `plan.plan` field carries the full narrative envelope:

```json
{
  "version": 1,
  "projectId": "cmp...",
  "title": "Q3 Platform Launch: The Collaboration Revolution",
  "target": {
    "audience": "enterprise buyers",
    "objective": "introduce the Q3 platform launch",
    "outcome": "book a demo"
  },
  "sourceDigest": {
    "summary": "Cross-source synthesis...",
    "sourceIds": ["cmq..."],
    "evidence": [
      { "sourceId": "cmq...", "claim": "60% onboarding reduction", "support": "...", "confidence": "high" }
    ],
    "gaps": [],
    "rightsNotes": []
  },
  "thesis": {
    "oneLine": "Real-time collaboration at enterprise scale, proven in weeks not months.",
    "supportingClaims": ["60% onboarding reduction"],
    "avoidClaims": ["competitor comparisons"]
  },
  "journey": {
    "beats": [
      {
        "id": "hook-1",
        "role": "hook",
        "title": "The collaboration problem",
        "narration": "Every enterprise team faces the same moment...",
        "visualIntent": "Split screen: fragmented tools vs unified platform",
        "evidenceIds": ["ev-0"],
        "estimatedSeconds": 12
      }
    ]
  },
  "visualSystem": {
    "style": "Corporate documentary with kinetic data viz",
    "palette": ["#1a1a2e", "#16213e", "#0f3460"],
    "typography": "Sans-serif, high legibility at distance"
  },
  "audioSystem": {
    "voice": "Authoritative narrator, measured pace",
    "music": "Ambient corporate — builds through conflict beat",
    "sfxPolicy": "Minimal UI chimes on key data reveals"
  },
  "artifacts": [
    {
      "type": "table_read",
      "mode": "documentary",
      "renderPlan": {}
    }
  ]
}
```

### Beat roles

Every beat in `journey.beats` has a `role` from the seven-point story arc:

| Role | Purpose |
|---|---|
| `hook` | Open with the tension or question that pulls the audience in |
| `context` | Establish the world, the stakes, or the before state |
| `conflict` | Name the core problem or friction |
| `proof` | Present evidence, data, or demonstration |
| `turn` | The pivot — the resolution or revelation |
| `payoff` | The after state; what is now possible |
| `cta` | Call to action |

### The credit quote

Every plan carries a `quote` computed deterministically from the source count and artifact requests. Planning and source digest lines are included/unmetered; artifact and add-on lines are the billable work:

```json
{
  "currency": "studio_credits",
  "total": 11,
  "lineItems": [
    { "kind": "planning", "label": "StoryPlan generation", "credits": 0 },
    { "kind": "source_digest", "label": "Source digest (1 source)", "credits": 0 },
    { "kind": "artifact", "label": "Table read", "credits": 11 }
  ],
  "notes": [],
  "computedAt": "2026-05-29T10:05:00.000Z"
}
```

The quote is provisional — final reserve and settle happen at the job creation step. The actual cost cannot differ if the plan inputs have not changed, but treat the quote as an estimate if you are comparing across environments or API versions.

**Post-creation operation costs** (reserved at call time, not quoted in the plan):

| Operation | Credit key | Credits |
|---|---|---|
| Refine a table read (`POST /artifacts/{id}/refine`) | `TABLE_READ_REFINE` | 8 |

Finalizing a durable MP3 or MP4 (`POST /artifacts/{id}/finalize`, mode `audio` or `video`) is free.

### Refining a plan (via the refine endpoint — see Artifacts)

To revise a generated table read without rebuilding the entire plan-to-job flow, use `POST /api/v1/artifacts/{artifactId}/refine`. This is a post-creation operation on the **artifact** (not the plan) — it re-versions the same artifact in place. See [the Artifacts section below](#the-table-read-artifact) and [api-reference.md](api-reference.md#artifacts) for full details.

**Alternative pattern:** Create a new plan against the same project with updated `target`, `styleConstraints`, or `sourceIds`. The previous plan automatically transitions to `SUPERSEDED`. This is appropriate when you want to change the narrative direction substantially enough to warrant a fresh plan and a new artifact.

---

## Story Jobs

A **StoryJob** runs queued artifact generation against an `APPROVED` plan. Creating a job reserves the quoted Studio Credits.

### Job lifecycle (`status`)

```
PENDING  -->  RESERVED  -->  RUNNING  -->  READY
              ^              |
              | provider     | capacity/policy block
              +--- backoff ---+
                                      -->  PARTIAL  (some artifacts ready, others running)
                                      -->  FAILED
              -->  CANCELED  (on explicit cancel, credits released)
```

- `RESERVED` — the credit reservation succeeded. Ordinarily the worker has not started yet; if `progress.stage` is `provider_backoff`, Sleeper Hit preserved the same job/checkpoint and already scheduled its delayed retry. Keep polling—no client recovery loop is needed.
- `RUNNING` — the worker is actively generating artifacts.
- `PARTIAL` — at least one artifact is `READY` while others are still running. You can read ready artifacts immediately.
- `READY` — all artifacts are in a terminal state (all `READY` or some `FAILED`).

### Canceling a job

Cancel a job that has not started: `PENDING`, `RESERVED`, or `FAILED` before it ever started (a refused reservation or a lost enqueue — canceling that one means it can no longer be resumed). A `RUNNING` job cannot be canceled, and neither can a `FAILED` job that had started generating:

```
POST /api/v1/story-jobs/{jobId}/cancel
```

Reserved-but-unsettled credits are released. The cancel operation is idempotent — canceling a `CANCELED` job is a no-op.

### Idempotency on job creation

Always supply an `Idempotency-Key` when creating a job. If the request fails (network timeout, 5xx) you can retry with the same key and the API will replay the original response instead of creating a second job and charging credits twice.

---

## Artifacts

A **StoryArtifact** is a single generated output produced by a job. One job can produce multiple artifacts, such as a table read plus a pitch deck.

### Artifact lifecycle (`status`)

```
PENDING  -->  RUNNING  -->  READY
                       -->  FAILED
```

When `READY`, the `manifest` field is populated with the artifact's consumable output.

### The table read artifact

The first live artifact type. A table read is a live, on-demand audio/video performance driven by the Pipecat/Daily stack — not a static file.

**Modes:**
- `documentary` — narrated with supporting evidence, factual tone.
- `podcast` — conversational, two or more voices, interview-style.
- `drama` — character-voiced, scripted scene.

`narrationPolicy` is separate from mode. Choose narrator/no-narrator as a
creation-time posture for the piece: podcasts, documentaries, dramas, panels,
audio essays, and true-crime or archival casefiles can each be narrated or
narrator-free when that fits the form.

**The manifest** (when `status: READY`):

```json
{
  "type": "table_read",
  "status": "ready",
  "shareToken": "<expiring JWT>",
  "theaterUrl": "https://sleeperhit.studio/share/table-read/<token>",
  "theaterFullscreenUrl": "https://sleeperhit.studio/share/table-read/<token>?view=fullscreen",
  "audio": {
    "mode": "live_on_demand",
    "liveUrl": "https://sleeperhit.studio/share/table-read-audio/<token>",
    "startLiveUrl": "https://sleeperhit.studio/api/share/table-read/start-live",
    "recordingUrl": null,
    "recordingDurationMs": null,
    "finalize": null
  },
  "video": {
    "status": "not_rendered",
    "renderUrl": "/api/v1/artifacts/<id>/render-video",
    "url": null
  },
  "expiresAt": "2026-06-29T10:00:00.000Z"
}
```

The top-level artifact also carries `currentVersion` (integer, starts at 1) and `refineCount` (integer, starts at 0) which update with each refine operation.

**The three player URLs:**

| URL | Description |
|---|---|
| `theaterUrl` | Framed visual theater page with attribution chrome. Suitable for linking. |
| `theaterFullscreenUrl` | The same theater stage rendered chrome-free and immersive. Best for `<iframe>` embeds or kiosk displays. |
| `audio.liveUrl` | Minimal-chrome, audio-only player. No visual stage; the end-user clicks play and the live audio streams. |

All three URLs are authorized by the `shareToken` (not your API key) and are "unlisted" — public but unguessable. End-users can open them without holding your API key.

**Starting the live performance programmatically:**

Both the theater and audio-only players start the live read by POSTing the share token to `audio.startLiveUrl`. You can also call this directly from your own UI:

```js
const res = await fetch(manifest.audio.startLiveUrl, {
  method: 'POST',
  headers: { 'Content-Type': 'application/json' },
  body: JSON.stringify({ token: manifest.shareToken }),
})
const { dailyRoomUrl } = await res.json()
// dailyRoomUrl is the Daily.co room the end-user joins
```

**Refine — conversational post-creation revision:**

`POST /api/v1/artifacts/{artifactId}/refine` (scope `artifact:publish`, requires `Idempotency-Key`)

```json
{ "instruction": "rename the host to Dana; make the skeptic warmer", "scope": "screenplay" }
```

`scope` options: `"auto"` (default, model self-routes), `"screenplay"` (faster — rewrites screenplay + recasts voices in place), `"plan"` (re-grounds against sources then re-adapts).

Returns `{ storyJob }`. Poll `GET /api/v1/story-jobs/{jobId}` until terminal (`READY` / `FAILED`), exactly like a standard job. On completion, `GET /api/v1/artifacts/{artifactId}` reflects updated `currentVersion` and `refineCount`.

**The artifact id, share token, and all three player URLs are STABLE across refines.** Embed once — the updated read is served at the same URLs.

Fetch a prior revision: `GET /api/v1/artifacts/{artifactId}?revision=N` (v1 = initial generation).

Credit cost: TABLE_READ_REFINE — **8 credits**.

**Recast, coverage, music, and sound effects:**

These controls mutate or analyze the same table read after creation without changing the artifact id or share URLs.

| Operation | Scope | Effect |
|---|---|---|
| `POST /api/v1/artifacts/{artifactId}/voice` | `artifact:publish` | Reassign one character voice in place; no new revision, no credit charge |
| `POST /api/v1/artifacts/{artifactId}/coverage` | `story:write` | Generate async professional coverage; read it with `GET /coverage` |
| `POST /api/v1/artifacts/{artifactId}/music` | `artifact:publish` | Generate/regenerate adaptive soundtrack, or tune/mute/replace direction for the whole read, one scene, or one entry |
| `POST /api/v1/artifacts/{artifactId}/sfx` | `artifact:publish` | Add, update, retime, mute, regenerate, or remove line-level sound-effect cues |
| `GET /api/v1/artifacts/{artifactId}/music` and `/sfx` | `artifact:read` | Poll/read music status and SFX cues |

**Finalize — durable MP3:**

`POST /api/v1/artifacts/{artifactId}/finalize` with `{ "mode": "audio" }` (scope `artifact:publish`, requires `Idempotency-Key`)

Freezes a completed Pipecat/Daily table-read recording into a permanent full-mix MP3. If no completed recording exists yet, the endpoint returns `needs_recording` and does not reserve credits. Poll `GET /api/v1/artifacts/{artifactId}`:

`manifest.audio.finalize.status` stages: `needs_recording`, or `queued` → `rendering` → `uploading` → `complete` (or `failed` — inspect `manifest.audio.finalize.error`).

On `complete`: `manifest.audio.recordingUrl` (MP3 download URL) and `manifest.audio.recordingDurationMs` are populated.

Free. Replay-safe: re-calling when already complete or in-flight returns the current state without rendering again.

**Finalize — durable MP4:**

`POST /api/v1/artifacts/{artifactId}/finalize` with `{ "mode": "video" }` (scope `artifact:publish`, requires `Idempotency-Key`)

Queues the Remotion MP4 export (same render as `render-video`). Poll `manifest.video.status` and `manifest.video.url`.

Free.

**The opt-in MP4 render (legacy alias):**

`video.status` is `not_rendered` until you explicitly request it. `POST /api/v1/artifacts/{artifactId}/render-video` remains available and is equivalent to `finalize?mode=video` for the MP4 render. The status transitions `not_rendered` → `queued` → `rendering` → `complete` (or `failed`; `stale` means the read changed after the last render). Poll `manifest.video.status` and `manifest.video.url`.

**Live scrubber (planned):** A timeline scrubber on the theater and audio-only players. Currently the live read plays from the beginning.

### The pitch deck artifact

Pitch decks are generated from an approved StoryPlan and a completed table read for the same plan/project. This lets the pitch journey reuse canonical script text, avatars, and voices rather than creating a disconnected package.

When `status: READY`, the manifest carries:

| Field | Description |
|---|---|
| `pitchDeckJobId` | Underlying journey-plan pitch deck job. |
| `share.url` | Public pitch-deck page. |
| `video.status`, `video.url`, `video.durationMs` | Rendered pitch-deck video. |
| `pdf.status`, `pdf.url` | Companion PDF deck. |
| `sourceTableReadArtifactId`, `sourceTableReadJobId` | Table-read source used for script, avatars, and voices. |

### Retired artifact types

`storyboard` and `production_video` were retired with storyboarding on 2026-10-09. A request for either is
refused with `400 validation_failed`, whose `details.allowedTypes` names the live types (`pitch_deck`,
`table_read`, `trailer`). `production_video` artifacts made before then are still read and published. Always
check `GET /api/v1/capabilities` before generating to confirm which types are `available`.

---

## Credits

**Studio Credits** are the single shared meter for all generation work. They accumulate across two buckets:

- **Monthly bucket** — allocated at the start of each billing cycle, expires at cycle end.
- **Perpetual balance** — credits purchased outright, never expire.

The `balance` field on the credits response is the sum of both buckets.

### Credit lifecycle for a job

1. **Quoted** — the plan's `quote.total` gives the projected cost. No credits are spent.
2. **Reserved** — when a job is created (`POST /story-jobs`), the quoted amount is reserved. If `balance < quote.total` the API returns `402 insufficient_credits`.
3. **Settled** — as each artifact becomes shareable (transitions to `READY`), its credit line items are settled from the reservation.
4. **Released** — if a job is canceled or an artifact fails, any reserved-but-unsettled credits are released back to the balance.

A `402 insufficient_credits` response carries `details.required` and `details.available` so you can surface the shortfall and prompt a top-up before retrying.

---

## Share tokens

The `shareToken` on a table read manifest is an expiring JWT. It is distinct from your customer API key and has no API write capabilities — it only authorizes the read-only player and live-start endpoints.

You can safely give share token–based URLs to end-users. When the token expires (`expiresAt`), the player URLs stop working and a new job must be run to get a new token.

---

## Scopes

Every customer API key has a set of scopes that gate which endpoints it can call. The default scopes on a new key are `story:read` and `credits:read`. The full set:

| Scope | Gates |
|---|---|
| `story:read` | GET on projects, sources, plans, jobs, capabilities |
| `story:write` | POST/PATCH/DELETE on projects, sources, plans, jobs |
| `source:read` | GET on sources |
| `source:write` | POST source creation, DELETE sources |
| `artifact:read` | GET on artifacts |
| `artifact:publish` | POST refine, voice recast, finalize, render-video, adaptive soundtrack, and SFX cue mutations |
| `credits:read` | GET on the credits balance |
| `webhook:write` | Planned — register webhook endpoints |

The `insufficient_scope` error code signals a missing scope. The `error.message` names the specific scope that was required.

---

## Idempotency

Every POST that reserves credits or queues generation work accepts an `Idempotency-Key` header. Supply a UUID that is stable for the logical operation (not a new UUID per retry). The API replays the original response on retry with `Idempotency-Replayed: true` in the response headers.

If you submit a different request body with the same key, the API returns `idempotency_conflict`. Generate a new key only when you intend a genuinely new operation.

Persist the idempotency key alongside your own job record before sending the request. If the network fails before you receive the response, retry with the same key.

A saved response is kept for 7 days. After that the key is free again: a request that reuses it is treated as new and does the work again, so retry within 7 days, and read the result of an older request with the matching GET instead.

---

## Versioning

The current API base is `/api/v1`. Breaking changes ship behind a new base prefix (e.g. `/api/v2`). Additive changes — new endpoints, new optional fields, new artifact types, new scopes — land inside `/api/v1` and are reflected immediately in `GET /api/v1/capabilities` and `GET /api/v1/openapi.json`.

Match on stable `error.code` values, not on HTTP status codes alone or human message text. Both status codes and messages may become more specific over time; codes are permanent.