AI Video Content Studio — Master Build Report

Core principle: Models are config, not code. All AI models are selectable from the UI via the workflowConfigs table. Adding a new model = adding a DB row.

Image Generation (Portraits)

Single-Person Talking Head

Multi-Person Talking Head (Phase 7+)

TTS

Build Steps

1. 1
   Initialize Next.js 15 projectApp Router, TypeScript, Tailwind CSS
2. 2
   Install + configure Convex
3. 3
   Install + configure ClerkClerk → Convex auth integration
4. 4
   Create the complete Convex schemaconvex/schema.ts — ALL 12 tables defined here
5. 5
   Install shadcn/uiBase components: Button, Card, Input, Select, Dialog, Tabs, Badge, Table
6. 6
   Create dashboard layout/dashboard/layout.tsx — sidebar nav + topbar
7. 7
   Create placeholder pagesAll routes: just page shells, no functionality
8. 8
   Create .env.local.example
9. 9
   Deploy to VercelEven if just the empty shell

Sidebar Navigation

Done When

• npm run dev starts without errors
• Can sign in with Clerk
• Dashboard layout renders with sidebar navigation
• All placeholder pages load without errors
• Convex schema deploys successfully
• App deploys to Vercel and loads in browser
• All tables exist in Convex dashboard (even if empty)

Do NOT

• Add any AI functionality yet
• Add R2 integration (Phase 2)
• Add RunningHub API calls (Phase 2)
• Build forms that submit data — just shells
• Add any provider code
• Spend time on landing page — redirect / to /dashboard

Build Steps

1. 1
   R2 presigned URL utilitysrc/lib/r2.ts — getUploadUrl, getDownloadUrl, deleteObject. Keys: {userId}/{type}/{timestamp}-{random}.{ext}
2. 2
   Convex media helpersconvex/media.ts — generate upload URLs, get download URLs
3. 3
   Provider type definitionssrc/lib/providers/types.ts — VideoProvider, ImageProvider, TTSProvider, SpendContext interfaces
4. 4
   Base provider with spend trackingsrc/lib/providers/base.ts — abstract base wraps every external call with spend event. Subclasses inherit auto-tracking.
5. 5
   RunningHub providersrc/lib/providers/runninghub.ts — implements VideoProvider + ImageProvider. Base URL: runninghub.cn. Endpoints: create/status/result.
6. 6
   OpenAI TTS providersrc/lib/providers/openai-tts.ts — implements TTSProvider. 6 voices. $0.015/1K chars. Audio → R2 → URL.
7. 7
   Provider registrysrc/lib/providers/registry.ts — getVideoProvider, getImageProvider, getTTSProvider lookup
8. 8
   Generic job queueconvex/jobs.ts — submitWorkflow + pollJob (scheduled actions, NOT await — RH jobs take minutes)
9. 9
   Spend trackingconvex/spend.ts — recordSpend, getRecentSpend, getSpendByDate/Provider/Category/Entity
10. 10
    Workflow config seed dataconvex/workflowConfigs.ts — seed Flux Dev, Hallo2, MuseTalk, OpenAI TTS, InstantID, GFPGAN
11. 11
    Basic spend page/dashboard/spend/page.tsx — table of events + totals (no charts yet)

Done When

• Can generate presigned R2 upload URL from Convex
• Can upload test file to R2 and retrieve via download URL
• RunningHub provider class exists and compiles
• OpenAI TTS provider can generate audio from text
• Job queue can submit, poll, and complete a mock job
• Spend event auto-created when TTS generates audio
• /dashboard/spend shows recent spend events
• Workflow config seed data loads into Convex
• All provider interfaces defined and implemented (at least stubs)

Do NOT

• Build UI forms for characters or videos (Phase 3+)
• Call RunningHub with real workflow IDs yet
• Add ElevenLabs (just OpenAI TTS)
• Build model picker UI (just data layer)
• Process/generate images or videos in this phase

Flow

Build Steps

1. 1
   LLM prompt expansionsrc/lib/prompts.ts — expandCharacterBrief() → detailed portrait prompt
2. 2
   Character creation form/dashboard/characters/create/page.tsx — text area + 3 optional dropdowns + "Generate Candidates"
3. 3
   Character CRUDconvex/characters.ts — createCharacter, generateCandidates, listCharacters, getCharacter
4. 4
   Candidate gridsrc/components/candidate-grid.tsx — 4-8 images, Select button, Regenerate button
5. 5
   Character selectionconvex/characterVersions.ts — selectVersion, rejectVersion mutations
6. 6
   Character detail page/dashboard/characters/[id]/page.tsx — portrait, metadata, versions, generation settings
7. 7
   Character gallery/dashboard/characters/page.tsx — grid of cards, status badges, Create button
8. 8
   Verify spend trackingLLM expansion + image gen events visible on spend page

Tables Used

characters · characterVersions · characterAssets · workflowConfigs · renderJobs · spendEvents

Done When

• User can type description and click "Generate Candidates"
• 4-8 candidate portraits appear in grid (from RunningHub)
• User can select one candidate as character identity
• Selected character appears in gallery
• Detail page shows portrait, metadata, generation settings
• Seed, prompt, model, settings stored and visible
• Spend events appear for LLM expansion + image gen
• Second character doesn't interfere with first
• Character status updates real-time during generation

Do NOT

• Build reference pack generator (Phase 5)
• Build refinement pass (Phase 5)
• Build face embedding extraction (Phase 5)
• Add photo upload — synthetic only
• Generate video from character (Phase 4)

Flow

Build Steps

1. 1
   Script editorsrc/components/script-editor.tsx — text area, char count, duration est, voice selector, Generate Audio, audio preview player
2. 2
   Video creation page/dashboard/videos/create/page.tsx — 4 steps: select char → script+audio → style → resolution → Generate
3. 3
   TTS integrationconvex/videos.ts — generateAudio action: OpenAI TTS → blob → R2 → audioUrl
4. 4
   Video generation orchestrationconvex/videos.ts — generateVideo + onVideoJobComplete. Character image + audio → RH workflow → poll → R2
5. 5
   Video playersrc/components/video-player.tsx — HTML5 video, play/pause, download, duration
6. 6
   Video detail page/dashboard/videos/[id]/page.tsx — player, job status, script, character, style, cost, download
7. 7
   Video list page/dashboard/videos/page.tsx — grid with thumbnails, status badges, Create button
8. 8
   Job status componentsrc/components/job-status.tsx — real-time via Convex subscription: Queued→Submitted→Running→Complete/Failed

Tables Used

videos · scripts · renderJobs · spendEvents

Important

If the exact RunningHub workflow isn't configured yet, build the full flow with a stub/mock that returns a test video URL after a delay. Architecture must be correct even without real workflow ID.

Done When

• Can select character and write script
• Can generate TTS audio and preview it
• Can generate video (or mock if RH workflow not ready)
• Job status updates real-time during generation
• Completed video plays in player
• Video can be downloaded
• Video list shows all generated videos
• Detail page shows cost breakdown (TTS + video gen)
• Spend page reflects TTS and video gen costs
• Failed jobs show error message

Do NOT

• Build multiple styles — only Yap/Casual
• Build conversation/multi-speaker (Phase 6)
• Build UGC or product video (Phase 8+)
• Build model picker UI — hardcode default config
• Skip job polling — critical infrastructure
• Store video files in Convex — they go to R2

Build Steps

1. 1
   Character refinementconvex/characters.ts — refineCharacter action: take selected portrait → RH refinement prompt (preserve identity, enhance detail) → new refined version
2. 2
   Reference pack generationOptional. Generate front_smile, three_quarter, profile, speaking_expression via InstantID identity injection
3. 3
   Multiple video stylesAdd 5 new styles: Podcaster, In the Car, Office/Business, Walking, Street Interview. Each maps to different RH workflow or prompt/settings
4. 4
   Model picker componentsrc/components/model-picker.tsx — dropdown of enabled workflow configs for current category
5. 5
   Settings > Models page/dashboard/settings/models/page.tsx — table of all workflow configs, edit enabled/cost/workflow ID

New Styles

Done When

• Character detail page has "Refine" button
• Refined portrait is noticeably more detailed
• Refined portrait preserves identity (same person)
• Video creation shows multiple style options
• At least 2 styles produce different-looking videos
• Model picker appears on video creation page
• Settings > Models lists all workflow configs
• Configs can be enabled/disabled from settings

Do NOT

• Train any LoRAs — use reference images + InstantID
• Build face embedding extraction unless InstantID requires it
• Over-engineer model picker — dropdown is enough
• Add styles without a corresponding RH workflow

Flow

Build Steps

1. 1
   Conversation script editor/dashboard/conversations/create/page.tsx — 2-column layout, char per speaker, voice per speaker, drag-to-reorder lines
2. 2
   Conversation script storageconvex/scripts.ts — type: "conversation", lines array: { speakerId, text, order }
3. 3
   Conversation generation orchestrationconvex/conversations.ts — generateConversation (parallel clip gen) + assembleConversation (concatenate after ALL complete)
4. 4
   Clip trackingconversationClips table — each clip: pending → generating_audio → generating_clip → ready
5. 5
   Conversation list + detail pagesList: all conversation videos. Detail: script, per-clip status, final video
6. 6
   Video assemblyConcatenate clips with 0.3-0.5s crossfade. Merge audio into continuous track. RunningHub workflow or FFmpeg.

Tables Used

videos · scripts · conversationClips · renderJobs · spendEvents

Done When

• Can create a 2-speaker conversation script
• Each speaker assigned to different character
• Each speaker has distinct voice
• All clips generate independently (per-clip progress visible)
• Final assembled video plays as coherent conversation
• Cuts between speakers are smooth (crossfade)
• Audio is continuous (no gaps or overlaps)
• Conversation list shows all videos
• Spend page shows itemized costs per clip + assembly

Do NOT

• Generate shared scene with both characters (Phase 7)
• Support more than 2 speakers
• Allow overlapping dialogue — speakers take turns
• Skip per-clip status tracking
• Assemble clips until ALL are confirmed complete

Build Steps

1. 1
   Two-person anchor scene generationGenerate single image: Character A left, Character B right, chest up, shared lighting, podcast studio. Use InstantID for both faces. This is the hardest image gen task so far.
2. 2
   Multi-audio video generationSubmit anchor scene + 2 audio files to multi-person model. Primary: LongCat Avatar 1.5 → Fallback: InfiniteTalk → 2nd: MultiTalk
3. 3
   Quality gate evaluation8 criteria — ALL must pass. If ANY fails: feature does NOT ship, fall back to Phase 6 cut-based.

Quality Gate Criteria

• Person A's mouth moves during Person A's audio
• Person B's mouth moves during Person B's audio
• The silent person does NOT visibly speak
• Both faces remain stable (no identity drift/morphing)
• Characters don't swap identities
• Background consistent (no flickering/warping)
• Result looks like conversation, not two stitched animations
• No extreme visual artifacts

Time Limit: 3-4 days max. If it doesn't work → move on.

Done When

• Quality gate tested with at least 3 different character pairs
• Results documented (pass/fail per criterion per test)
• If passed: "Shared Scene" option available in conversation creation
• If failed: phase marked complete with failure documented, Phase 6 cut-based remains only mode

Do NOT

• Spend more than 3-4 days on this phase
• Try to fix quality with post-processing hacks
• Ship if quality gate fails
• Block other phases on this

Flow

Build Steps

1. 1
   Product image uploadUpload or paste URL. SAM2 RunningHub workflow → clean transparent PNG. Store original + clean in R2.
2. 2
   Pose selection4 poses: holding, presenting, using, unboxing. Each has reference template/prompt.
3. 3
   UGC generation pipelineconvex/ugc.ts — generateUGC: segment → pose → composite → animate. Each sub-task = separate renderJob.
4. 4
   UGC creation pageSelect char → upload product → select pose → script+audio → generate. Show sub-task progress.
5. 5
   Product managementconvex/products.ts — save products for reuse. Store: title, images, clean images.

Tables Used

products · ugcJobs · renderJobs · spendEvents

Done When

• Can upload product image and get clean background-removed version
• Can generate UGC video with character holding product
• Sub-task progress visible during generation
• Final video shows character naturally interacting with product
• Spend page shows per-sub-task costs

Do NOT

• Build product URL scraper (Phase 9)
• Support video product demos — still images composited only
• Over-engineer pose options — 4 is enough

Build Steps

1. 1
   Product scrapersrc/lib/scraper.ts — Jina Reader first, LLM parse, fallback Amazon PA-API 5.0. Download images → R2.
2. 2
   Product add page/dashboard/products/add/page.tsx — URL input → Scrape → preview (title, price, features, images) → edit → Save
3. 3
   Auto-script generationLLM generates 30-second video script from product data. User can edit before generating.
4. 4
   Video style optionsProduct Showcase (images + avatar voiceover) or UGC Review (reuse Phase 8 pipeline)
5. 5
   Multi-resolution outputGenerate in 3 aspect ratios: 16:9, 9:16, 1:1. Store all three.

Done When

• Can paste Amazon URL and see extracted product data
• Product data can be edited and saved
• Video script auto-generated from product data
• At least one video style generates watchable product video
• Multi-resolution works (at least 2 of 3 ratios)
• Products saved and reusable

Do NOT

• Build full e-commerce integration
• Scrape sites that block scraping — Jina + PA-API only
• Auto-generate without user script review

Build Steps

1. 1
   Keyframe generationLLM expands description → 3 scene prompts (start → middle → end). RunningHub Flux → 3 keyframes. User can preview + regenerate individual keyframes.
2. 2
   Video interpolationRunningHub: H3 FL2VA or Wan 2.1 img2vid → smooth transitions. NOTE: H3 workflow has resolution mismatch bug (Stage 1 = 9:16, Stage 2 = 16:9) — must fix before use.
3. 3
   Timelapse creation pageText input → keyframe preview grid (3 images) → regenerate individual → duration control → optional music upload → Generate
4. 4
   Post-processingSpeed ramp for timelapse feel, optional text overlay, merge with music if provided

Done When

• Can write description and see 3 generated keyframes
• Keyframes can be individually regenerated
• Smooth timelapse video generated from keyframes
• Video duration controllable
• Spend page shows all generation costs

Do NOT

• Build complex timeline editor
• Support more than 3 keyframes
• Fix H3 bug by guessing — match both stages to same resolution

Build Steps

1. 1
   Dashboard overview/dashboard/page.tsx — recent activity feed, quick stats (chars, videos, monthly spend), quick action buttons
2. 2
   Spend analytics/dashboard/spend/page.tsx — bar chart (daily 30d), pie charts (category + provider), expensive jobs table, cost trend
3. 3
   Daily spend aggregationconvex/spend.ts — Convex cron job at midnight: sum spendEvents → spendSummaries
4. 4
   General polishLoading states, error states, empty states on all pages. Responsive (tablet min). Consistent badges. Toast notifications.
5. 5
   Settings page/dashboard/settings/page.tsx — connected accounts, environment status, link to Models page

Done When

• Dashboard overview shows meaningful recent activity
• Spend page has at least 2 charts (daily trend + category breakdown)
• Daily aggregation job runs and populates spendSummaries
• All pages have proper loading, error, and empty states
• Full end-to-end flow: create char → create video → watch → see cost
• No console errors in normal usage

Do NOT

• Build a landing page — this is internal
• Add team features, sharing, collaboration
• Add API access for external consumers
• Add mobile-specific layouts — tablet-responsive is enough
• Add analytics beyond spend tracking

1. 1
   No local ML inferenceAll generation through RunningHub
2. 2
   No video editorGeneration only — editing is a future product
3. 3
   No mobile appsWeb only
4. 4
   No team featuresSingle user
5. 5
   No payments / billing / creditsInternal tool
6. 6
   No LoRA trainingUse reference images + InstantID
7. 7
   No public sharing / social / publishing
8. 8
   No landing page or marketing site
9. 9
   No Supabase, Firebase, n8n, or external job queues
10. 10
    No wrangler.toml or Cloudflare Pages deployment
11. 11
    No Redux, Zustand, React Query, or SWR