AI Video Content Studio โ€” Master Build Report

A beginner-friendly guide to building the video generation platform. Every concept explained, every connection mapped.

๐Ÿ“‹ 11 Phases ๐Ÿ—„๏ธ 12 Convex Tables ๐Ÿ”ง 14 RunningHub Workflows ๐Ÿ’ฐ Auto Spend Tracking ๐ŸŽฏ Internal Tool โ€” No Payments
Total Phases
11
Sequential โ€” no skipping
DB Tables
12
All built in Phase 1
Workflows
14
RunningHub ComfyUI
Providers
3
Video ยท Image ยท TTS
Video Types
5
Talk ยท Convo ยท UGC ยท Product ยท Timelapse
Quality Gate
Ph 7
Shared scene โ€” ship or kill
โš™๏ธHow the App Works (Architecture)
๐Ÿ’ก What is "architecture"?
Architecture just means "how the pieces of the app talk to each other." Think of it like a plumbing diagram โ€” data flows from one pipe to the next. Here's the flow:
User clicks button in browser โ†“ Next.js UI (React) โ€” the website you see โ†“ Convex โ€” the brain (database + job queue + real-time updates) โ†“ RunningHub API โ€” sends workflow to a GPU cloud computer โ†“ ComfyUI workflows โ€” the actual AI models that generate images/video โ†“ Cloudflare R2 โ€” stores the finished media files (like S3 but cheaper) โ†“ Convex updates status โ†’ Next.js shows result in real-time
๐Ÿ”— How this connects to the project
Every single feature in this app follows this same flow. Whether it's generating a character portrait, a talking head video, or a timelapse โ€” the data always travels this same path. The difference is which ComfyUI workflow gets called and what inputs it receives.
๐Ÿ“–Glossary โ€” What Is Everything?
๐Ÿ’ก Read this first
If you're new to development, scan this glossary before diving into the phases. These are the core building blocks of the entire project.
Next.js
The website framework. It's React (the UI library) plus built-in routing, server-side rendering, and API routes. Your app's pages, buttons, forms โ€” all Next.js.
Convex
The backend/database. Replaces Supabase, Firebase, or a custom server. It stores all your data (characters, videos, jobs) AND runs background tasks (like polling job status). Real-time updates built in โ€” when data changes, the UI updates automatically.
Clerk
Authentication (login/signup). Handles user accounts, sessions, and security. You don't build login forms โ€” Clerk provides them. It gives each user a unique ID that Convex uses to separate their data.
Cloudflare R2
File storage (like a hard drive in the cloud). Stores images, videos, audio files. Similar to Amazon S3 but cheaper. Files are accessed via URLs (presigned URLs for security).
Presigned URL
A temporary, secure URL that lets someone upload or download a file to R2 without needing your API keys. Like a one-time-use key to a storage locker. Expires after a few minutes.
RunningHub
A cloud service that runs ComfyUI workflows on powerful GPUs. You send it a workflow JSON + inputs, it runs the AI models, and returns the results (images, videos). You pay per GPU-second.
ComfyUI
An open-source AI workflow tool that chains together AI models. Think of it like a visual programming language for AI. RunningHub hosts it so you don't need your own GPU.
Workflow JSON
A configuration file that tells ComfyUI which AI models to run, in what order, with what settings. Like a recipe โ€” "take this image, run it through this model, save the output."
Provider (Video/Image/TTS)
A standardized wrapper around an external API. Instead of calling RunningHub directly in 10 different places, you call videoProvider.submitJob() โ€” one clean interface that every feature uses. This is the "adapter pattern."
Job Queue / Polling
AI generation takes minutes, not seconds. You can't just await the result. Instead: submit the job โ†’ save a record โ†’ check back every 10 seconds ("poll") โ†’ when done, process the result. Convex handles this with scheduled actions.
Spend Tracking
Every API call that costs money automatically records how much it cost. Built into the provider base class so you never forget to track it. Shows up on a dashboard.
workflowConfigs
A database table that acts as a "model registry." Each row is an AI model/workflow with its settings. Want to add a new model? Add a row โ€” no code changes needed. Models are config, not code.
Convex Schema
The blueprint for your database. Defines every table, every column, every data type. Like a spreadsheet template โ€” "this table has these columns with these types of values."
Convex Actions vs Mutations vs Queries
Queries = read data (like SELECT). Mutations = write data (like INSERT/UPDATE). Actions = call external APIs (RunningHub, OpenAI) โ€” can't directly write to DB, so they call mutations.
Flux Dev
An AI image generation model. Creates photorealistic portraits from text descriptions. Runs on RunningHub. This is what generates your character portraits.
InstantID
An identity injection model. Takes a face from one image and puts it into a new scene/pose while keeping the person looking the same. Critical for character consistency across videos.
Hallo2
An AI model that animates a still portrait image to "speak" given an audio file. The mouth moves, head nods โ€” creates a talking head video from just a photo + audio.
TTS (Text-to-Speech)
Converts written text into spoken audio. OpenAI's TTS API does this โ€” you give it text + a voice name, it returns an audio file.
Vercel
The hosting platform for Next.js. You push your code to GitHub, Vercel automatically builds and deploys it. Like Cloudflare Pages but optimized for Next.js.
Tailwind CSS
A CSS framework. Instead of writing custom CSS, you use pre-built classes: bg-red-500, text-xl, p-4. Makes styling fast and consistent.
shadcn/ui
A component library built on Tailwind + Radix UI. Gives you pre-built, accessible UI components (buttons, dialogs, tables, dropdowns) that you copy into your project and customize.
๐Ÿ—๏ธTech Stack (Locked Decisions)
๐Ÿ’ก Why are these "locked"?
Because the AI agent building this needs clear rules. If it's allowed to pick Supabase instead of Convex, or add Redux for state management, the whole architecture breaks down. These decisions were made for specific reasons and should not change.
LayerChoiceWhy (in plain English)
Frontend hostingVercelWhere the website lives. Best support for Next.js. NOT Cloudflare Pages.
FrameworkNext.js 15 App RouterThe website toolkit. "App Router" is the newer way to structure pages.
Backend / DBConvexDatabase + background jobs + real-time updates, all in one. Replaces Supabase + n8n.
AuthClerkLogin/signup without building it yourself. Integrates cleanly with Convex.
Media storageCloudflare R2Where images/videos/audio files live. Cheap, fast, S3-compatible.
AI executionRunningHubCloud GPUs that run ComfyUI workflows. You don't need your own GPU server.
StylingTailwind + shadcn/uiFast styling with pre-built components. No custom CSS headaches.
State managementConvex subscriptionsData changes in DB โ†’ UI updates automatically via WebSocket. No Redux needed.
File uploadsPresigned URLs โ†’ R2Files go directly to R2, bypassing your server. No file size limits.
Video playbackHTML5 <video>Built-in browser video player. Simple, universal, no library needed.
๐Ÿค–Model Strategy
๐Ÿ’ก What does "models are config, not code" mean?
Instead of hardcoding which AI model to use in the application code, every model is stored as a row in the workflowConfigs database table. Want to switch from Hallo2 to MuseTalk for talking head videos? Change a row in the table โ€” no code change needed. Want to add a brand new model? Insert a new row. This makes the system flexible without touching the app.

Image Generation (Making Portraits)

ModelWhat It DoesWhen It's Used
Flux DevCreates photorealistic portraits from text descriptionsPhase 3 โ€” character creation
InstantIDTakes a face and puts it into new poses/scenes while keeping the same personPhase 4+ โ€” consistency across videos
IP-Adapter FaceID Plus v2Same as InstantID but more flexible (fallback)If InstantID doesn't work well
GFPGAN / CodeFormerCleans up faces โ€” fixes artifacts, sharpens detailsAlways โ€” final step on every portrait

Video Generation (Making Things Talk)

ModelWhat It DoesWhen It's Used
Hallo2Animates a still photo to "speak" given audio โ€” mouth moves, head nodsPhase 4 โ€” single-person talking head
MuseTalkSame as Hallo2 but faster, slightly lower quality (fallback)When speed matters more than quality
LongCat Avatar 1.5Two people talking in one scene, each with their own audioPhase 7 โ€” shared-scene conversations

Text-to-Speech (Making Voices)

ServiceWhat It DoesCost
OpenAI TTSConverts text to spoken audio with 6 voice options$0.015 per 1,000 characters
ElevenLabsPremium TTS with voice cloning (can copy a real person's voice)Add when needed
๐Ÿ—„๏ธAll Convex Database Tables
๐Ÿ’ก What is a database table?
Think of it like a spreadsheet. Each table is a sheet with columns (fields) and rows (records). The characters table has columns like name, status, userId โ€” and each character you create is a new row. All 12 tables are defined in Phase 1, even if some aren't used until Phase 8.
TableFirst UsedWhat It Stores
charactersPhase 3Character info: name, description, status, which portrait was selected
characterVersionsPhase 3Each generated variant โ€” you generate 4-8 portraits, each is a "version"
characterAssetsPhase 3The actual image files: portraits, reference poses, face embeddings
scriptsPhase 4Video scripts โ€” text that gets turned into speech
videosPhase 4Video records: type, status, output URL, duration, which character
conversationClipsPhase 6Individual clips in a multi-person conversation (each line = one clip)
productsPhase 8Product info scraped from URLs: title, images, price, features
ugcJobsPhase 8UGC video jobs: character + product + pose + sub-task tracking
renderJobsPhase 2Every RunningHub job: status, progress, output URLs, errors
workflowConfigsPhase 2Model registry: which AI models exist, their settings, costs
spendEventsPhase 2Every API call that costs money: how much, when, for what
spendSummariesPhase 11Daily aggregated totals (populated by a nightly cron job)
๐Ÿ”„RunningHub Workflows
๐Ÿ’ก What is a RunningHub workflow?
A workflow is a recipe for ComfyUI. It says: "Take these inputs (an image, a prompt, some settings), run them through these AI models in this order, and give me the output." Each workflow is stored as a JSON file on RunningHub and identified by a workflow ID. Your app sends the ID + inputs via API, RunningHub runs it on a GPU, and returns the result.
WorkflowPhaseWhat It Does (Plain English)
character_create3Takes a text description โ†’ generates 4-8 portrait photos
character_refine5Takes a portrait โ†’ enhances detail (skin pores, hair strands, lighting) while keeping the same person
character_reference_pack5Generates the same character from different angles (front, profile, smile, speaking)
talking_head_yap4Photo + audio โ†’ animated talking head video (casual style)
talking_head_podcast_single5Photo + audio โ†’ podcast-style talking head (upper body, mic, studio)
talking_head_car5Photo + audio โ†’ person talking while "driving" (driver seat POV)
talking_head_walking5Photo + audio โ†’ full-body walking video with moving background
shared_scene_anchor7Takes 2 character photos โ†’ generates one image with both people in the same scene
shared_scene_video7Scene image + 2 audio files โ†’ video where both characters speak (lip sync per speaker)
product_segment8Product photo โ†’ removes background โ†’ clean transparent PNG
ugc_product_video8Character + product image + audio โ†’ video of character holding/presenting the product
product_showcase9Product images + voiceover audio โ†’ showcase video with product shots
timelapse103 keyframe images โ†’ smooth interpolation video (like a timelapse)
face_restore3+Cleans up any face image โ€” fixes AI artifacts, sharpens details
1 Project Scaffold + Auth + Database
Sets up the empty project. After this: you can log in and see an empty dashboard. Nothing works yet โ€” this is just the foundation.
๐Ÿ’ก What is "scaffolding"?
Scaffolding means creating the empty structure of the project โ€” the folders, the configuration files, the basic layout โ€” before building any actual features. Like framing a house before adding walls, plumbing, and electricity.
๐Ÿ”— How Phase 1 connects to everything else
Phase 1 creates the container that every other phase lives inside. The database schema (all 12 tables) is defined here even though most aren't used until later phases. The sidebar navigation, page routes, and Clerk auth are the skeleton that every feature plugs into.

Build Steps

  1. 1
    Initialize Next.js 15 projectRun npx create-next-app with TypeScript, Tailwind CSS, and App Router. This creates the folder structure, config files, and a dev server.
    ๐Ÿ“Œ What this solves: Gives you a working website you can open in a browser at localhost:3000. Every page, button, and form in the app will live inside this project.
  2. 2
    Install + configure ConvexRun npx convex dev to connect your project to a Convex backend. This creates the convex/ folder where your database schema and backend logic will live.
    ๐Ÿ“Œ What this solves: Convex is your database AND your background job runner. Without it, there's nowhere to store data (characters, videos, etc.) and no way to run long-running tasks like polling RunningHub jobs.
  3. 3
    Install + configure ClerkAdd Clerk for login/signup. Configure the Clerk โ†’ Convex integration so each user's Convex data is tied to their Clerk account.
    ๐Ÿ“Œ What this solves: Without auth, anyone could see anyone's characters and videos. Clerk gives each user a unique ID. Every database query filters by userId so users only see their own data.
  4. 4
    Create the complete Convex schemaconvex/schema.ts โ€” Define ALL 12 tables with their columns and data types. Even tables not used until Phase 8.
    ๐Ÿ“Œ What this solves: The schema is the blueprint for your entire data model. Defining it all upfront means you never have to restructure the database mid-build. Each table has typed columns (string, number, arrays, optional fields) and indexes for fast queries.
  5. 5
    Install shadcn/ui base componentsRun the shadcn CLI to add: Button, Card, Input, Select, Dialog, Tabs, Badge, Table.
    ๐Ÿ“Œ What this solves: Pre-built, accessible UI components. Instead of building a dropdown menu from scratch (hundreds of lines), you get a ready-made <Select> component that works with keyboard, screen readers, and looks good.
  6. 6
    Create dashboard layout with sidebar/dashboard/layout.tsx โ€” The sidebar navigation that appears on every page inside /dashboard.
    ๐Ÿ“Œ What this solves: The sidebar is the main navigation. Every feature page (Characters, Videos, Conversations, etc.) is accessible from here. It's a "layout" component โ€” it wraps around all dashboard pages automatically.
  7. 7
    Create placeholder pages for all routesEmpty pages for: Characters, Videos, Conversations, Products, Spend, Settings, Models. Just the page title and "coming soon."
    ๐Ÿ“Œ What this solves: Every URL exists from day one. When you click "Characters" in the sidebar, it goes to /dashboard/characters and shows a page (even if empty). This prevents broken links and lets you test navigation immediately.
  8. 8
    Create .env.local.exampleA template file listing every environment variable the app needs (Clerk keys, Convex URL, RunningHub key, R2 credentials, OpenAI key).
    ๐Ÿ“Œ What this solves: Environment variables are secret values (API keys, passwords) that the app needs but shouldn't be in the code. The .example file shows what's needed without exposing real values. You copy it to .env.local and fill in real values.
  9. 9
    Deploy to VercelPush to GitHub, connect to Vercel, deploy. Even if it's just empty shells.
    ๐Ÿ“Œ What this solves: Gets the app on the internet from day one. Every future change auto-deploys when you push to GitHub. You can share the URL and test from any device.

Questions You Might Have

โ“ Common Questions
  • "Why build ALL tables in Phase 1 if most aren't used until Phase 8?"
    Because changing the database schema later is painful โ€” it can break existing code. Defining everything upfront is cleaner. Empty tables cost nothing.
  • "What's the difference between /dashboard/layout.tsx and /dashboard/page.tsx?"
    layout.tsx wraps around ALL pages inside /dashboard (the sidebar + topbar). page.tsx is just the main content area for the /dashboard route specifically.
  • "Do I need a Vercel account?"
    Yes. Free tier works. Connect your GitHub repo and Vercel auto-deploys on every push.

Done When

  • npm run dev starts without errors
  • Can sign in with Clerk
  • Dashboard layout renders with sidebar navigation
  • All placeholder pages load without errors
  • Convex schema deploys successfully
  • App deploys to Vercel and loads in browser
  • All tables exist in Convex dashboard (even if empty)

Do NOT

  • Add any AI functionality yet
  • Add R2 integration (Phase 2)
  • Add RunningHub API calls (Phase 2)
  • Build forms that submit data โ€” just shells
  • Add any provider code
  • Spend time on landing page โ€” redirect / to /dashboard
2 R2 Storage + Provider Interfaces + Spend Foundation
Builds the plumbing that every future feature depends on. After this: files can be stored, external APIs can be called, and every dollar spent is tracked automatically.
๐Ÿ’ก Why is this phase so important?
Phase 2 builds the infrastructure layer โ€” the plumbing that every feature uses. Without R2, there's nowhere to put generated images/videos. Without provider interfaces, every feature would need its own code to call RunningHub. Without the job queue, there's no way to track long-running AI jobs. Think of this as building the highway system before you start driving cars on it.
๐Ÿ”— How Phase 2 connects to everything else
Every single phase from 3-11 depends on Phase 2. Character creation (Phase 3) uses the image provider + job queue. Video generation (Phase 4) uses the video provider + TTS provider + R2 storage + job queue. Conversations (Phase 6) use all of the above. The spend tracking built here automatically records costs for every feature without any extra work in later phases.

Build Steps

  1. 1
    R2 presigned URL utilitysrc/lib/r2.ts โ€” Three functions: getUploadUrl() (get a URL to upload a file), getDownloadUrl() (get a URL to view/download a file), deleteObject() (delete a file).
    ๐Ÿ“Œ What is R2? Cloudflare R2 is like Google Drive or Dropbox, but for apps. It stores files (images, videos, audio) in the cloud. You access files via URLs.

    ๐Ÿ“Œ What is a presigned URL? A temporary, secure link that lets someone upload or download a file without needing your API passwords. Like giving someone a one-time key to a storage locker โ€” it expires after a few minutes.

    ๐Ÿ“Œ Why not store files in Convex? Convex is a database โ€” great for structured data (text, numbers, IDs), but terrible for large binary files (images, videos). R2 is built specifically for file storage. It's also much cheaper โ€” pennies per GB.

    ๐Ÿ“Œ What this solves: Every image, video, and audio file in the app goes through R2. Character portraits, talking head videos, TTS audio โ€” all stored here. The file naming pattern ({userId}/{type}/{timestamp}-{random}.{ext}) keeps files organized and prevents collisions.
  2. 2
    Convex media helpersconvex/media.ts โ€” Functions that Convex can call to generate upload/download URLs. Bridges the gap between the frontend and R2.
    ๐Ÿ“Œ What this solves: The frontend can't call R2 directly (it doesn't have the API keys). It asks Convex "give me an upload URL" โ†’ Convex generates a presigned URL โ†’ frontend uses that URL to upload directly to R2. This keeps your secret keys safe on the server.
  3. 3
    Provider type definitionssrc/lib/providers/types.ts โ€” TypeScript interfaces that define what every provider must do.
    ๐Ÿ“Œ What is a "provider"? A provider is a standardized wrapper around an external API. Instead of calling RunningHub's API directly in 10 different places, you create one RunningHubProvider class that implements a standard interface (submitJob, getStatus, getResult). Every feature calls the same interface โ€” doesn't matter if it's RunningHub or a different service.

    ๐Ÿ“Œ What is a TypeScript interface? A contract that says "any class that implements this must have these methods with these inputs and outputs." Like a job description โ€” "Video providers must be able to submit jobs, check status, get results, and cancel."

    ๐Ÿ“Œ What this solves: Three interfaces: VideoProvider (for video generation), ImageProvider (for image generation), TTSProvider (for text-to-speech). Every AI service in the app implements one of these. If you swap RunningHub for a different service later, you only change the provider โ€” no feature code changes.
  4. 4
    Base provider with spend trackingsrc/lib/providers/base.ts โ€” An abstract base class that wraps every external API call with automatic cost recording.
    ๐Ÿ“Œ What is a "base class"? A template that other classes inherit from. The base provider has one key method: whenever any provider calls an external API, the base class automatically records how much it cost. Subclasses (RunningHub, OpenAI TTS) inherit this for free โ€” they never need to write spend tracking code.

    ๐Ÿ“Œ What this solves: Without this, you'd have to manually add spend tracking to every single API call in every feature. With the base class, it happens automatically. Feature code just calls provider.submitJob() and the spend is recorded behind the scenes.
  5. 5
    RunningHub providersrc/lib/providers/runninghub.ts โ€” Implements both VideoProvider and ImageProvider. Talks to RunningHub's API: submit a job, check its status, get the result.
    ๐Ÿ“Œ How RunningHub works: You send a POST request with a workflow ID + inputs (image URLs, prompts, settings). RunningHub assigns a GPU, runs the ComfyUI workflow, and returns a job ID. You then poll (check every 10 seconds) "is job X done?" When done, you get the output URLs (images or videos stored on RunningHub's servers). You download them and upload to your own R2 for permanent storage.

    ๐Ÿ“Œ What this solves: Every image generation (character portraits) and video generation (talking heads) goes through this provider. It's the bridge between your app and the GPU cloud.
  6. 6
    OpenAI TTS providersrc/lib/providers/openai-tts.ts โ€” Implements TTSProvider. Sends text to OpenAI's API, gets back audio, uploads to R2.
    ๐Ÿ“Œ What is TTS? Text-to-Speech โ€” converts written text into spoken audio. OpenAI's TTS API has 6 voices (alloy, echo, fable, onyx, nova, shimmer). You send text + voice name, it returns an audio file.

    ๐Ÿ“Œ What this solves: Every video needs a voice. Whether it's a character talking, a narration, or a conversation โ€” the script text goes through this provider to become audio. The audio is then fed to the talking head models (Hallo2, etc.) to animate the character's mouth.
  7. 7
    Provider registrysrc/lib/providers/registry.ts โ€” A lookup table: give it a provider name ("runninghub"), it returns the right provider instance.
    ๐Ÿ“Œ What this solves: When the job queue processes a job, it needs to know which provider to use. The registry maps names to instances: getVideoProvider("runninghub") โ†’ returns the RunningHub provider. This is the "dependency injection" pattern โ€” features don't import providers directly, they ask the registry.
  8. 8
    Generic job queueconvex/jobs.ts โ€” Two key functions: submitWorkflow (send a job to RunningHub, save a record) and pollJob (check if the job is done, process the result).
    ๐Ÿ“Œ Why can't you just await the result? AI generation takes 1-10 minutes. If you await it, your server would hang for minutes waiting. Instead: submit the job โ†’ save a renderJobs record with status "submitted" โ†’ schedule a check in 10 seconds โ†’ check again โ†’ when done, update the record and process results.

    ๐Ÿ“Œ What is "polling"? Checking repeatedly: "Is it done yet? Is it done yet?" Every 10 seconds, the job queue asks RunningHub "what's the status of job X?" When it says "completed," the queue downloads the output, uploads to R2, updates the database, and notifies the user.

    ๐Ÿ“Œ What this solves: Every AI generation in the app (images, videos, audio) goes through this queue. It's the universal "submit and wait" mechanism. Features just submit a job and subscribe to status updates โ€” they don't manage the polling themselves.
  9. 9
    Spend trackingconvex/spend.ts โ€” recordSpend (insert a cost record) + queries for recent spend, spend by date/provider/category, spend per entity.
    ๐Ÿ“Œ What this solves: Every API call that costs money (RunningHub GPU time, OpenAI TTS characters, LLM tokens) creates a spendEvents record. The spend page shows: how much today, this week, this month. Per-provider breakdown. Per-category breakdown. Per-video cost. This is operational visibility โ€” you always know what you're spending.
  10. 10
    Workflow config seed dataconvex/workflowConfigs.ts โ€” Inserts initial rows: Flux Dev, Hallo2, MuseTalk, OpenAI TTS, InstantID, GFPGAN.
    ๐Ÿ“Œ What is "seed data"? Pre-populated database records that the app needs to function. Like pre-installing apps on a new phone. The workflowConfigs table is the "model registry" โ€” each row is an AI model with its name, provider, category, cost estimate, and enabled flag. New models are added by inserting rows, not editing code.
  11. 11
    Basic spend page/dashboard/spend/page.tsx โ€” Table of recent spend events + totals. No charts yet.
    ๐Ÿ“Œ What this solves: The first page that shows real data. After this phase, every time an API call is made (TTS, image gen, video gen), a spend event appears here. This is how you verify the spend tracking is working.

Questions You Might Have

โ“ Common Questions
  • "Why presigned URLs instead of uploading through my server?"
    If a user uploads a 500MB video through your Next.js server, it blocks the server and has file size limits. With presigned URLs, the file goes directly from the browser to R2 โ€” your server never touches it. Faster, no size limits, less server load.
  • "What's the difference between a Convex action, mutation, and query?"
    Query = read data (like SELECT in SQL). Mutation = write data (like INSERT/UPDATE). Action = call external APIs (RunningHub, OpenAI) โ€” can't write to DB directly, so it calls mutations. The job queue uses actions to call RunningHub, then mutations to update the database.
  • "Why check every 10 seconds? Why not have RunningHub notify us when done?"
    RunningHub supports webhooks (push notifications) in some plans, but polling is simpler and more reliable. 10 seconds is a good balance โ€” fast enough to feel responsive, slow enough to not spam their API.
  • "What happens if a job fails?"
    The pollJob function detects the "failed" status, updates the renderJobs record with the error message, and the UI shows the error to the user. The spend event still records the cost (even failed jobs use GPU time).
  • "Why a provider registry instead of just importing the provider directly?"
    The registry lets you swap providers without changing feature code. If you switch from RunningHub to a different GPU service, you create a new provider class, update the registry, and every feature automatically uses the new one.

Done When

  • Can generate presigned R2 upload URL from Convex
  • Can upload test file to R2 and retrieve via download URL
  • RunningHub provider class exists and compiles
  • OpenAI TTS provider can generate audio from text
  • Job queue can submit, poll, and complete a mock job
  • Spend event auto-created when TTS generates audio
  • /dashboard/spend shows recent spend events
  • Workflow config seed data loads into Convex
  • All provider interfaces defined and implemented (at least stubs)

Do NOT

  • Build UI forms for characters or videos (Phase 3+)
  • Call RunningHub with real workflow IDs yet
  • Add ElevenLabs (just OpenAI TTS)
  • Build model picker UI (just data layer)
  • Process/generate images or videos in this phase
3 Character Creator (Text โ†’ Candidates โ†’ Select)
First feature that actually calls RunningHub. You describe a person in words, AI generates portrait photos, you pick your favorite.
๐Ÿ’ก What is a "character" in this app?
A character is a persistent AI-generated person that can star in videos. You describe them ("young woman with dark curly hair, professional look"), the AI generates portrait images, you pick one, and that portrait becomes the character's identity. Later, when you make a video, this same character appears โ€” same face, same look โ€” thanks to InstantID identity injection.
๐Ÿ”— How Phase 3 connects to later phases
Characters are the building block for almost everything. Phase 4 uses them for talking head videos. Phase 5 refines their portraits. Phase 6 puts two characters in a conversation. Phase 8 has characters hold products. You need at least one "ready" character before any video can be made.

The Flow

User types: "young woman with dark curly hair, warm skin, professional look" โ†“ Optional dropdowns: Gender (female), Age (late 20s), Style (realistic) โ†“ LLM (GPT-4o-mini) expands this into a detailed portrait prompt โ†’ "Portrait photo of a young woman in her late 20s, dark curly shoulder-length hair, warm medium-brown skin with natural pores..." โ†“ RunningHub generates 4-8 candidate portraits (Flux Dev model) โ†“ User sees a grid of portraits โ†’ clicks "Select" on their favorite โ†“ That portrait becomes the character's identity โ†’ Seed, prompt, model, settings all saved for reproducibility

Build Steps

  1. 1
    LLM prompt expansionsrc/lib/prompts.ts โ€” Takes a short description and expands it into a detailed portrait prompt using GPT-4o-mini.
    ๐Ÿ“Œ What this solves: AI image models work better with detailed prompts. "Young woman" gives generic results. "Portrait photo of a young woman in her late 20s, dark curly shoulder-length hair, warm medium-brown skin with natural pores and subtle freckles, oval face, soft jawline, dark brown eyes, cream knit sweater, soft natural lighting, clean background, editorial portrait style, photorealistic, 8k detail" gives much better results. The LLM bridges the gap between casual description and AI-optimized prompt.
  2. 2
    Character creation form/dashboard/characters/create/page.tsx โ€” Text area + 3 optional dropdowns + "Generate Candidates" button.
    ๐Ÿ“Œ What this solves: The user interface for creating characters. The text area is the main input (free-form description). The dropdowns (gender, age range, style) help the LLM generate better prompts. The button triggers the whole pipeline.
  3. 3
    Character CRUDconvex/characters.ts โ€” Create, read, update, delete operations for characters + the generation action.
    ๐Ÿ“Œ What is CRUD? Create, Read, Update, Delete โ€” the four basic database operations. createCharacter inserts a new row. listCharacters reads all characters for the current user. generateCandidates is the action that calls the LLM + RunningHub.

    ๐Ÿ“Œ What this solves: The backend logic for the entire character feature. The frontend form calls these functions โ€” it doesn't talk to RunningHub or the LLM directly.
  4. 4
    Candidate gridsrc/components/candidate-grid.tsx โ€” Displays 4-8 generated portraits in a grid with Select buttons.
    ๐Ÿ“Œ What this solves: After generation, the user needs to see all candidates and pick one. The grid shows them side by side. Each has a "Select" button and a visual highlight when selected. "Regenerate" button generates a fresh batch if none are good enough.
  5. 5
    Character selectionconvex/characterVersions.ts โ€” Marks one version as "selected", updates the character's identity.
    ๐Ÿ“Œ What this solves: When you generate 4-8 portraits, each is a "characterVersion." Selecting one marks it as the character's identity โ€” this is the portrait that will be used in all future videos. The seed and settings are saved so you can reproduce the exact same portrait later.
  6. 6
    Character detail page/dashboard/characters/[id]/page.tsx โ€” Shows the selected portrait, metadata, all versions, generation settings, and a link to create a video.
    ๐Ÿ“Œ What this solves: The character's home page. Shows everything about the character: portrait, name, description, all generated versions (selected, rejected, generating), and the technical settings (seed, prompt, model) used to create the selected portrait. The "Create Video" button links to Phase 4.
  7. 7
    Character gallery/dashboard/characters/page.tsx โ€” Grid of all characters with status badges.
    ๐Ÿ“Œ What this solves: Overview of all characters. Each card shows the portrait thumbnail, name, and status (draft, generating, ready, error). Click to go to detail page. "Create Character" button goes to the creation form.
  8. 8
    Verify spend trackingAfter creating a character, check the spend page โ€” you should see events for both the LLM expansion and the image generation.
    ๐Ÿ“Œ What this solves: This is a verification step, not a build step. It confirms that the spend tracking from Phase 2 actually works with real API calls.

Questions You Might Have

โ“ Common Questions
  • "What's a 'seed' and why save it?"
    A seed is a random number that AI models use as a starting point. Same seed + same prompt = same image. Saving it means you can reproduce the exact same portrait later. Different seeds with the same prompt give different variations.
  • "Why generate 4-8 candidates instead of just 1?"
    AI image generation is probabilistic โ€” the same prompt gives different results each time. Generating multiple gives the user options. Some might look great, some might not. The user picks the best one.
  • "What happens to rejected candidates?"
    They stay in the database with status "rejected" โ€” they're not deleted. This is useful for reference: "I didn't like version #3 because the eyes were wrong." They're just hidden from the main character view.
  • "Can I change the character's portrait later?"
    Yes โ€” you can generate new candidates and select a different one. Or use Phase 5's refinement to enhance the existing portrait.

Done When

  • User can type description and click "Generate Candidates"
  • 4-8 candidate portraits appear in grid (from RunningHub)
  • User can select one candidate as character identity
  • Selected character appears in gallery
  • Detail page shows portrait, metadata, generation settings
  • Seed, prompt, model, settings stored and visible
  • Spend events appear for LLM expansion + image gen
  • Second character doesn't interfere with first
  • Character status updates real-time during generation

Do NOT

  • Build reference pack generator (Phase 5)
  • Build refinement pass (Phase 5)
  • Build face embedding extraction (Phase 5)
  • Add photo upload โ€” synthetic only
  • Generate video from character (Phase 4)
4 Single-Person Talking Head Video (Yap Style)
The first end-to-end video. Pick a character โ†’ write a script โ†’ generate audio โ†’ animate the character to "speak" the script.
๐Ÿ’ก How does a still photo become a talking video?
Two AI models work together. First, InstantID takes the character's portrait and generates a new image in a "yap" pose (casual, upper body, neutral background). Then, Hallo2 takes that pose image + the audio file and animates the mouth, eyes, and head to match the speech. The output is a video file โ€” the character appears to be talking.
๐Ÿ”— How Phase 4 connects to later phases
This is the core video generation flow that Phase 5 extends (more styles), Phase 6 uses per-line (conversations), and Phase 8 wraps with product placement. The TTS โ†’ animate pipeline is the foundation of every video in the app.

The Flow

Select character from gallery โ†“ Write script (plain text for single speaker) โ†“ Pick a voice (alloy, echo, fable, onyx, nova, shimmer) โ†“ "Generate Audio" โ†’ OpenAI TTS โ†’ audio file โ†’ stored in R2 โ†“ Pick style: "Yap/Casual" (only style in Phase 4) โ†“ "Generate Video" โ†’ RunningHub: InstantID (character in yap pose) โ†’ RunningHub: Hallo2 (animate from audio) โ†’ Job polls until complete โ†’ Video stored in R2 โ†“ Watch the video, download it

Build Steps

  1. 1
    Script editorsrc/components/script-editor.tsx โ€” Text area for the script with character count, estimated duration, voice selector dropdown, "Generate Audio" button, and audio preview player.
    ๐Ÿ“Œ What this solves: The script is the text the character will "say." The character count estimates duration (roughly: characters รท 15 = seconds). The voice selector lets you choose from 6 OpenAI voices. The audio preview lets you listen before committing to video generation.
  2. 2
    Video creation page/dashboard/videos/create/page.tsx โ€” 4-step wizard: (1) select character, (2) write script + generate audio, (3) pick style, (4) pick resolution.
    ๐Ÿ“Œ What this solves: Breaks the complex video creation process into manageable steps. Each step has clear inputs and outputs. The user can't proceed to the next step until the current one is complete.
  3. 3
    TTS integrationconvex/videos.ts โ€” generateAudio action: calls OpenAI TTS โ†’ uploads audio blob to R2 โ†’ saves audioUrl on the video record.
    ๐Ÿ“Œ What this solves: Connects the script text to actual spoken audio. The audio file is stored in R2 and its URL is saved on the video record. This URL is later passed to the talking head model so it knows what audio to animate to.
  4. 4
    Video generation orchestrationconvex/videos.ts โ€” generateVideo action: loads character portrait โ†’ submits to RunningHub โ†’ creates renderJob โ†’ starts polling.
    ๐Ÿ“Œ What this solves: This is where the magic happens. The character's portrait URL + the audio URL are sent to RunningHub. RunningHub runs two workflows: (1) InstantID puts the character in a yap pose, (2) Hallo2 animates that pose with the audio. The polling loop checks every 10 seconds until done.
  5. 5
    Video playersrc/components/video-player.tsx โ€” HTML5 video element with play/pause, download button, duration display.
    ๐Ÿ“Œ What this solves: Displays the finished video. Uses a standard HTML5 <video> tag with the R2 URL as the source. No special library needed โ€” browsers know how to play videos.
  6. 6
    Video detail page/dashboard/videos/[id]/page.tsx โ€” Video player, job status, script text, character used, style, resolution, cost breakdown, download button.
    ๐Ÿ“Œ What this solves: The video's home page. Shows everything: the video itself, what character was used, what script was spoken, how much it cost, and a download button. If still generating, shows a real-time progress indicator.
  7. 7
    Video list page/dashboard/videos/page.tsx โ€” Grid of all videos with thumbnails, status badges, duration, date.
    ๐Ÿ“Œ What this solves: Overview of all generated videos. Each card shows a thumbnail (or placeholder if still generating), the title, status (generating, ready, error), and duration. Click to go to the detail page.
  8. 8
    Job status componentsrc/components/job-status.tsx โ€” Real-time progress display using Convex subscription: Queued โ†’ Submitted โ†’ Running โ†’ Completed (or Failed).
    ๐Ÿ“Œ What is a Convex subscription? Instead of the page refreshing every few seconds, Convex pushes updates to the browser via WebSocket. When the renderJob's status changes in the database, the UI updates instantly. This is why Convex was chosen โ€” real-time updates without writing any polling code on the frontend.

    ๐Ÿ“Œ What this solves: AI generation takes minutes. Users need to see progress. This component shows: current state, progress percentage (if available), estimated time remaining, and error messages if something fails.

Questions You Might Have

โ“ Common Questions
  • "What if the RunningHub workflow isn't ready yet?"
    Build the full flow with a mock/stub that returns a test video URL after a delay. The architecture must be correct โ€” the polling, the status updates, the R2 storage โ€” even if the actual workflow ID isn't configured yet. Fill it in later.
  • "Why two separate RunningHub calls (InstantID + Hallo2) instead of one?"
    They do different things. InstantID is an image model (generates a still pose). Hallo2 is a video model (animates a still image with audio). They're separate workflows because they use different AI models. The app chains them: InstantID output โ†’ Hallo2 input.
  • "What resolution should I pick?"
    16:9 = landscape (YouTube), 9:16 = portrait (TikTok/Reels), 1:1 = square (Instagram). The resolution affects which RunningHub workflow is used and how the output looks.

Done When

  • Can select character and write script
  • Can generate TTS audio and preview it
  • Can generate video (or mock if RH workflow not ready)
  • Job status updates real-time during generation
  • Completed video plays in player
  • Video can be downloaded
  • Video list shows all generated videos
  • Detail page shows cost breakdown (TTS + video gen)
  • Spend page reflects TTS and video gen costs
  • Failed jobs show error message

Do NOT

  • Build multiple styles โ€” only Yap/Casual
  • Build conversation/multi-speaker (Phase 6)
  • Build UGC or product video (Phase 8+)
  • Build model picker UI โ€” hardcode default config
  • Skip job polling โ€” critical infrastructure
  • Store video files in Convex โ€” they go to R2
5 Character Refinement + Multiple Styles
Characters get a quality boost, reference packs for consistency, and the video creator gets multiple style options.
๐Ÿ’ก What is "character refinement"?
The initial portrait from Phase 3 is good, but can be better. Refinement takes the selected portrait and runs it through another AI pass that enhances skin pores, hair strands, lighting, and eye detail โ€” while keeping the exact same person (same face, same proportions). Think of it as touching up a photo.

Build Steps

  1. 1
    Character refinementconvex/characters.ts โ€” refineCharacter action: takes selected portrait โ†’ submits to RunningHub with a refinement prompt โ†’ creates a new "refined" version.
    ๐Ÿ“Œ What this solves: The initial Flux Dev portrait might have minor issues (slightly blurry hair, flat lighting). Refinement is a targeted enhancement that says "preserve the exact identity but improve the photorealistic detail." The refined version becomes the new default.
  2. 2
    Reference pack generationOptional. Generate the same character from different angles: front smile, three-quarter, profile, speaking expression. Uses InstantID for identity injection.
    ๐Ÿ“Œ What this solves: Different video styles need different poses. A "podcast" style needs a front-facing upper body. A "walking" style needs a full body. Reference packs give the AI more information about what the character looks like from different angles, improving consistency.
  3. 3
    Multiple video stylesAdd 5 new styles to the video creation page: Podcaster, In the Car, Office/Business, Walking, Street Interview. Each maps to a different RunningHub workflow or prompt.
    ๐Ÿ“Œ What this solves: Phase 4 only had "Yap/Casual." Now users can create videos in different settings โ€” a podcast studio, a car interior, an office, walking down a street. Each style has a different RunningHub workflow that puts the character in the appropriate setting/pose.
  4. 4
    Model picker componentsrc/components/model-picker.tsx โ€” Dropdown that lists enabled workflow configs for the current category.
    ๐Ÿ“Œ What this solves: Lets users choose which AI model to use. For image generation: Flux Dev (best quality) vs a faster alternative. For video: Hallo2 (best quality) vs MuseTalk (faster). The dropdown reads from the workflowConfigs table โ€” adding a new model is just adding a row.
  5. 5
    Settings > Models page/dashboard/settings/models/page.tsx โ€” Table of all workflow configs with edit capabilities.
    ๐Ÿ“Œ What this solves: The admin interface for managing AI models. See all models, enable/disable them, update cost estimates, fill in workflow IDs. This is where you'd add a brand new model โ€” insert a row, and it becomes available in the model picker everywhere.

Done When

  • Character detail page has "Refine" button
  • Refined portrait is noticeably more detailed
  • Refined portrait preserves identity (same person)
  • Video creation shows multiple style options
  • At least 2 styles produce different-looking videos
  • Model picker appears on video creation page
  • Settings > Models lists all workflow configs
  • Configs can be enabled/disabled from settings

Do NOT

  • Train any LoRAs โ€” use reference images + InstantID
  • Build face embedding extraction unless InstantID requires it
  • Over-engineer model picker โ€” dropdown is enough
  • Add styles without a corresponding RH workflow
6 Cut-Based Multi-Character Conversations
Two characters having a conversation. Cut between close-ups of whoever is speaking โ€” like a real podcast or interview.
๐Ÿ’ก Why "cut-based" and not both characters in one scene?
Having two characters in one continuous scene (both visible, both moving) is extremely hard for current AI models. But cutting between close-ups of each speaker? That's just running the single-person talking head model separately for each line, then stitching the clips together. It's how real podcasts and interviews are edited โ€” you cut to whoever is speaking. This approach works reliably today.

The Flow

User creates a conversation script: Speaker A: "Welcome back everyone." Speaker B: "I've been looking forward to this." Speaker A: "Three things. First, the speed..." โ†“ Each speaker โ†’ assigned character + distinct TTS voice โ†“ For EACH line (in parallel): 1. Generate TTS audio for that line 2. Generate talking-head clip (character + audio segment) โ†“ ALL clips complete โ†’ concatenate with crossfade transitions โ†“ Output: single video that cuts between speakers

Build Steps

  1. 1
    Conversation script editor/dashboard/conversations/create/page.tsx โ€” Two-column layout: Speaker A | Speaker B. Each column has a character selector, voice selector, and text inputs for lines.
    ๐Ÿ“Œ What this solves: The UI for writing multi-speaker scripts. Each line is assigned to a speaker. Lines can be reordered by dragging. The preview shows the alternating dialogue.
  2. 2
    Conversation generation orchestrationconvex/conversations.ts โ€” Submits all clip jobs in parallel, polls each independently, assembles when ALL are complete.
    ๐Ÿ“Œ Why parallel? If you have 10 lines and generate them one at a time (2 min each), that's 20 minutes. In parallel, it's 2 minutes (limited by the slowest clip). Each clip is an independent job โ€” they don't depend on each other.

    ๐Ÿ“Œ What this solves: The most complex orchestration so far. Manages N independent jobs, tracks each one's status, and only assembles when all are done. Assembly concatenates the clips with brief crossfade transitions between speakers.
  3. 3
    Video assemblyConcatenate clips with 0.3-0.5s crossfade transitions. Merge audio into one continuous track.
    ๐Ÿ“Œ What this solves: The final step โ€” stitching individual clips into one coherent video. Crossfades make the cuts smooth (not jarring hard cuts). Audio is merged so there are no gaps or overlaps between lines.

Done When

  • Can create a 2-speaker conversation script
  • Each speaker assigned to different character with distinct voice
  • All clips generate independently (per-clip progress visible)
  • Final assembled video plays as coherent conversation
  • Cuts are smooth (crossfade), audio is continuous
  • Spend page shows itemized costs per clip + assembly

Do NOT

  • Generate shared scene with both characters (Phase 7)
  • Support more than 2 speakers
  • Allow overlapping dialogue โ€” speakers take turns
  • Assemble clips until ALL are confirmed complete
7 Shared-Scene Two-Person Video QUALITY GATE
Both characters visible in one continuous scene, each speaking their lines. This is experimental โ€” if quality isn't good enough, it doesn't ship.
๐Ÿ’ก What's a "quality gate"?
A quality gate is a hard pass/fail test. If the output doesn't meet ALL criteria, the feature is NOT shipped. Phase 6's cut-based editing is the fallback โ€” it always works. This phase tries something harder (both characters in one scene) but only ships if it looks good. Time-boxed to 3-4 days max.

Quality Gate Criteria (ALL must pass)

  • Person A's mouth moves during Person A's audio
  • Person B's mouth moves during Person B's audio
  • The silent person does NOT visibly speak
  • Both faces remain stable (no identity drift/morphing)
  • Characters don't swap identities
  • Background consistent (no flickering/warping)
  • Result looks like conversation, not two stitched animations
  • No extreme visual artifacts

Done When

  • Tested with at least 3 different character pairs
  • Results documented (pass/fail per criterion per test)
  • If passed: "Shared Scene" option available in conversation creation
  • If failed: phase marked complete with failure documented, Phase 6 cut-based remains only mode
8 UGC Product Placement
Characters hold, present, or use products in videos. Think: an influencer unboxing a product.
๐Ÿ’ก What is "UGC"?
UGC = User-Generated Content. In marketing, it's content that looks like a real person made it (not a brand ad). An AI character holding a product and talking about it looks like organic UGC โ€” but it's fully generated. This is a huge use case for AI video.

The Flow

Upload product photo โ†“ SAM2 (AI model) removes background โ†’ clean transparent PNG โ†“ Select character + pose (holding / presenting / using / unboxing) โ†“ Pipeline (4 sub-tasks, each a separate renderJob): 1. Segment product (remove background) 2. Generate character in pose (InstantID + ControlNet) 3. Composite product into scene (inpainting) 4. Animate with audio (Hallo2/MuseTalk) โ†“ Each sub-task has independent status tracking

Build Steps

  1. 1
    Product image upload + segmentationUpload product photo โ†’ SAM2 removes background โ†’ clean transparent PNG stored in R2.
    ๐Ÿ“Œ What is SAM2? Segment Anything Model 2 โ€” an AI that can identify and separate objects from their backgrounds. It takes a product photo and outputs a clean cutout with a transparent background. This is needed so the product can be composited into a new scene.
  2. 2
    UGC generation pipelineconvex/ugc.ts โ€” 4-stage pipeline: segment โ†’ pose โ†’ composite โ†’ animate. Each stage is a separate renderJob.
    ๐Ÿ“Œ What this solves: UGC is the most complex pipeline โ€” 4 sequential AI steps. Each step is tracked independently so the user can see exactly where the process is ("Currently compositing product into scene..."). If any step fails, the error is specific and actionable.

Done When

  • Can upload product image and get clean background-removed version
  • Can generate UGC video with character holding product
  • Sub-task progress visible during generation
  • Final video shows character naturally interacting with product
  • Spend page shows per-sub-task costs
9 Product URL โ†’ Video
Paste an Amazon (or other) product URL โ†’ system extracts info and images โ†’ auto-generates a showcase video.
๐Ÿ’ก What is "scraping"?
Scraping means programmatically reading a webpage and extracting structured data. You give it a product URL, it reads the page, and pulls out the title, price, features, and images. The app uses Jina Reader (a clean text extractor) and falls back to Amazon's official API for Amazon URLs.

Build Steps

  1. 1
    Product scrapersrc/lib/scraper.ts โ€” Jina Reader for clean text โ†’ LLM parses out title, price, features, image URLs โ†’ downloads images to R2.
    ๐Ÿ“Œ What this solves: Automates the boring part โ€” manually copying product info. Paste a URL, get structured data. The LLM is smart enough to parse messy HTML into clean fields.
  2. 2
    Auto-script generationLLM generates a 30-second video script from product data. User can edit before generating.
    ๐Ÿ“Œ What this solves: The script is the voiceover for the product video. The LLM knows how to write compelling product descriptions. User reviews and edits before committing to video generation.
  3. 3
    Multi-resolution outputGenerate in 3 aspect ratios: 16:9, 9:16, 1:1. Store all three.
    ๐Ÿ“Œ What this solves: Different platforms need different formats. YouTube = 16:9 landscape. TikTok/Reels = 9:16 portrait. Instagram feed = 1:1 square. Generating all three at once means the user has content ready for every platform.

Done When

  • Can paste Amazon URL and see extracted product data
  • Product data can be edited and saved
  • Video script auto-generated from product data
  • At least one video style generates watchable product video
  • Multi-resolution works (at least 2 of 3 ratios)
  • Products saved and reusable
10 Timelapse Videos
Write a description โ†’ AI generates 3 keyframe images โ†’ interpolates between them for a smooth timelapse video.
๐Ÿ’ก What is "interpolation"?
Interpolation means generating smooth transitions between images. If you have a "start" image (empty desk) and an "end" image (desk with computer and books), interpolation creates the in-between frames โ€” like a smooth timelapse of the desk being set up. The AI fills in the missing frames.

Build Steps

  1. 1
    Keyframe generationLLM expands description โ†’ 3 scene prompts (start โ†’ middle โ†’ end). Flux generates 3 keyframe images. User can preview and regenerate individual keyframes.
  2. 2
    Video interpolationH3 FL2VA or Wan 2.1 generates smooth transitions between the 3 keyframes. NOTE: H3 workflow has a resolution mismatch bug that must be fixed.
  3. 3
    Post-processingSpeed ramp for timelapse feel, optional text overlay, merge with music track if provided.

Done When

  • Can write description and see 3 generated keyframes
  • Keyframes can be individually regenerated
  • Smooth timelapse video generated from keyframes
  • Video duration controllable
  • Spend page shows all generation costs
11 Dashboard Polish + Spend Analytics
Polish the overall experience, add spend charts, clean up rough edges. The "make it nice" phase.

Build Steps

  1. 1
    Dashboard overview/dashboard/page.tsx โ€” Recent activity feed, quick stats (characters, videos, monthly spend), quick action buttons.
  2. 2
    Spend analyticsBar chart (daily 30d), pie charts (category + provider), most expensive jobs table, cost trend over time.
  3. 3
    Daily spend aggregationConvex cron job at midnight: sum all spendEvents by provider + category for the day โ†’ insert into spendSummaries.
  4. 4
    General polishLoading states, error states, empty states on all pages. Responsive layout (tablet minimum). Consistent status badges. Toast notifications for job completion and errors.
  5. 5
    Settings pageView connected accounts, environment status, link to Models page.

Done When

  • Dashboard overview shows meaningful recent activity
  • Spend page has at least 2 charts
  • Daily aggregation job runs and populates spendSummaries
  • All pages have proper loading, error, and empty states
  • Full end-to-end flow: create char โ†’ create video โ†’ watch โ†’ see cost
  • No console errors in normal usage
๐Ÿ“Directory Structure
๐Ÿ’ก How to read this
This is every file and folder in the project. convex/ = backend (database + logic). src/app/ = pages (what the user sees). src/components/ = reusable UI pieces. src/lib/ = utility code (providers, R2, scraper). The file path tells you what each file does.
video-studio/ โ”œโ”€โ”€ convex/ โ† BACKEND (database + logic) โ”‚ โ”œโ”€โ”€ schema.ts โ† Database blueprint (ALL 12 tables) โ”‚ โ”œโ”€โ”€ auth.ts โ† Clerk authentication integration โ”‚ โ”œโ”€โ”€ characters.ts โ† Character CRUD + generation triggers โ”‚ โ”œโ”€โ”€ characterVersions.ts โ† Version management + selection โ”‚ โ”œโ”€โ”€ scripts.ts โ† Script CRUD โ”‚ โ”œโ”€โ”€ videos.ts โ† Video CRUD + generation orchestration โ”‚ โ”œโ”€โ”€ conversations.ts โ† Multi-character clip orchestration โ”‚ โ”œโ”€โ”€ products.ts โ† Product scraping + management โ”‚ โ”œโ”€โ”€ ugc.ts โ† UGC job orchestration โ”‚ โ”œโ”€โ”€ jobs.ts โ† Generic render job queue + polling โ”‚ โ”œโ”€โ”€ spend.ts โ† Spend recording + aggregation โ”‚ โ”œโ”€โ”€ workflowConfigs.ts โ† Model/workflow registry CRUD โ”‚ โ”œโ”€โ”€ media.ts โ† R2 presigned URL generation โ”‚ โ””โ”€โ”€ http.ts โ† HTTP endpoints for webhooks โ”œโ”€โ”€ src/ โ”‚ โ”œโ”€โ”€ app/ โ† PAGES (what the user sees) โ”‚ โ”‚ โ”œโ”€โ”€ layout.tsx โ† Root layout (wraps everything) โ”‚ โ”‚ โ”œโ”€โ”€ page.tsx โ† / โ†’ redirects to /dashboard โ”‚ โ”‚ โ””โ”€โ”€ dashboard/ โ”‚ โ”‚ โ”œโ”€โ”€ layout.tsx โ† Sidebar + topbar (wraps all dashboard pages) โ”‚ โ”‚ โ”œโ”€โ”€ page.tsx โ† /dashboard โ†’ overview โ”‚ โ”‚ โ”œโ”€โ”€ characters/ โ† gallery, create, [id] โ”‚ โ”‚ โ”œโ”€โ”€ videos/ โ† list, create, [id] โ”‚ โ”‚ โ”œโ”€โ”€ conversations/ โ† list, create โ”‚ โ”‚ โ”œโ”€โ”€ products/ โ† list, add โ”‚ โ”‚ โ”œโ”€โ”€ spend/ โ† spend dashboard โ”‚ โ”‚ โ””โ”€โ”€ settings/ โ† account, models โ”‚ โ”œโ”€โ”€ components/ โ† REUSABLE UI PIECES โ”‚ โ”‚ โ”œโ”€โ”€ ui/ โ† shadcn/ui (Button, Card, Dialog, etc.) โ”‚ โ”‚ โ”œโ”€โ”€ character-creator.tsx โ”‚ โ”‚ โ”œโ”€โ”€ candidate-grid.tsx โ”‚ โ”‚ โ”œโ”€โ”€ video-player.tsx โ”‚ โ”‚ โ”œโ”€โ”€ job-status.tsx โ”‚ โ”‚ โ”œโ”€โ”€ script-editor.tsx โ”‚ โ”‚ โ”œโ”€โ”€ style-picker.tsx โ”‚ โ”‚ โ”œโ”€โ”€ model-picker.tsx โ”‚ โ”‚ โ”œโ”€โ”€ spend-chart.tsx โ”‚ โ”‚ โ””โ”€โ”€ product-card.tsx โ”‚ โ””โ”€โ”€ lib/ โ† UTILITY CODE โ”‚ โ”œโ”€โ”€ providers/ โ† AI service adapters โ”‚ โ”‚ โ”œโ”€โ”€ types.ts โ† Interfaces (what providers must do) โ”‚ โ”‚ โ”œโ”€โ”€ base.ts โ† Base class with auto spend tracking โ”‚ โ”‚ โ”œโ”€โ”€ runninghub.ts โ† RunningHub implementation โ”‚ โ”‚ โ”œโ”€โ”€ openai-tts.ts โ† OpenAI TTS implementation โ”‚ โ”‚ โ””โ”€โ”€ registry.ts โ† Provider lookup by name โ”‚ โ”œโ”€โ”€ r2.ts โ† R2 file storage helpers โ”‚ โ”œโ”€โ”€ scraper.ts โ† Product URL โ†’ structured data โ”‚ โ””โ”€โ”€ prompts.ts โ† LLM prompts (description expansion) โ”œโ”€โ”€ workflows/ โ† RunningHub workflow JSONs (reference) โ”œโ”€โ”€ package.json โ† Dependencies list โ”œโ”€โ”€ next.config.ts โ† Next.js configuration โ”œโ”€โ”€ tailwind.config.ts โ† Tailwind CSS configuration โ”œโ”€โ”€ tsconfig.json โ† TypeScript configuration โ”œโ”€โ”€ .env.local.example โ† Environment variable template โ””โ”€โ”€ .gitignore โ† Files Git should ignore
๐Ÿ”‘Environment Variables
๐Ÿ’ก What are environment variables?
Secret values (API keys, passwords, URLs) that the app needs to function but should NEVER be in the code. They're stored in a .env.local file (gitignored โ€” never uploaded to GitHub). The .env.local.example file is a template showing what's needed.
# Clerk (authentication) NEXT_PUBLIC_CLERK_PUBLISHABLE_KEY= โ† Public key (safe for browser) CLERK_SECRET_KEY= โ† Secret key (server only) # Convex (database + backend) NEXT_PUBLIC_CONVEX_URL= โ† Your Convex project URL # RunningHub (AI execution) RUNNINGHUB_API_KEY= โ† API key for RunningHub # Cloudflare R2 (file storage) R2_ACCOUNT_ID= โ† Cloudflare account ID R2_ACCESS_KEY_ID= โ† R2 API access key R2_SECRET_ACCESS_KEY= โ† R2 API secret key R2_BUCKET_NAME= โ† Bucket name (e.g., "video-studio") # TTS (text-to-speech) OPENAI_API_KEY= โ† OpenAI API key # Optional โ€” add when needed # ELEVENLABS_API_KEY= โ† For premium voice cloning # AMAZON_ACCESS_KEY= โ† For Amazon product scraping # AMAZON_SECRET_KEY= # AMAZON_PARTNER_TAG=
๐ŸšซNever Build (Unless Explicitly Asked)
  1. 1
    No local ML inferenceAll AI runs on RunningHub's cloud GPUs โ€” you don't install or run AI models locally.
  2. 2
    No video editorThis app generates videos from scratch. Editing existing videos is a different product.
  3. 3
    No mobile appsWeb only. The responsive layout works on tablets.
  4. 4
    No team featuresSingle user. No sharing, collaboration, or permissions.
  5. 5
    No payments / billing / creditsThis is an internal tool. Spend tracking is for visibility, not billing.
  6. 6
    No LoRA trainingUse reference images + InstantID for character consistency. No model training.
  7. 7
    No public sharing / social / publishingGenerated content stays in the dashboard.
  8. 8
    No landing page or marketing siteInternal tool โ€” no public-facing website.
  9. 9
    No Supabase, Firebase, n8n, or external job queuesConvex replaces all of these.
  10. 10
    No wrangler.toml or Cloudflare Pages deploymentThe app deploys to Vercel, not Cloudflare Pages.
  11. 11
    No Redux, Zustand, React Query, or SWRConvex subscriptions handle all state management and data fetching.
๐Ÿ”—How Everything Connects
๐Ÿ’ก The big picture
Here's how a video gets made, end to end, showing which components are involved at each step:
1. USER clicks "Create Video" in the Next.js UI โ””โ†’ UI is in src/app/dashboard/videos/create/page.tsx 2. USER selects a CHARACTER from the gallery โ””โ†’ Characters come from Convex query (convex/characters.ts) โ””โ†’ Character was created in Phase 3 via RunningHub Flux Dev 3. USER writes a SCRIPT and generates AUDIO โ””โ†’ Script stored in Convex (convex/scripts.ts) โ””โ†’ Audio via OpenAI TTS provider (src/lib/providers/openai-tts.ts) โ””โ†’ Audio file uploaded to Cloudflare R2 (src/lib/r2.ts) โ””โ†’ Spend event recorded automatically (convex/spend.ts) 4. USER clicks "Generate Video" โ””โ†’ Convex action (convex/videos.ts) submits to RunningHub โ””โ†’ RunningHub provider (src/lib/providers/runninghub.ts) calls API โ””โ†’ renderJob created in Convex (convex/jobs.ts) โ””โ†’ Spend event recorded automatically 5. JOB QUEUE polls RunningHub every 10 seconds โ””โ†’ Scheduled Convex action (convex/jobs.ts โ†’ pollJob) โ””โ†’ When complete: downloads output, uploads to R2 โ””โ†’ Updates renderJob status in Convex 6. UI updates in REAL-TIME via Convex subscription โ””โ†’ job-status.tsx component shows progress โ””โ†’ No manual polling on frontend โ€” Convex pushes updates via WebSocket 7. VIDEO is ready โ””โ†’ Video stored in R2, URL saved on video record โ””โ†’ video-player.tsx displays it (HTML5 <video> tag) โ””โ†’ Download button gives direct R2 URL โ””โ†’ Spend page shows total cost (TTS + video gen)
๐Ÿ”— Key architectural patterns
Provider pattern: All external API calls go through providers (src/lib/providers/). Never call RunningHub or OpenAI directly from feature code.

Spend tracking: Automatic via the base provider class. Feature code never records spend manually.

Job queue: All long-running tasks go through convex/jobs.ts. Never use await for AI generation โ€” always submit + poll.

Real-time updates: Convex subscriptions push data changes to the UI. No manual polling, no refresh buttons, no Redux.

Models as config: All AI models stored in workflowConfigs table. Adding a model = adding a DB row. No code changes.