Skip to content

Generate short video clips from the dialog - #299

Merged
iamwavecut merged 2 commits into
mainfrom
feat/fastframes-video
Nov 18, 2026
Merged

iamwavecut merged 2 commits into
mainfrom
feat/fastframes-video

Conversation

@iamwavecut

Copy link
Copy Markdown
Owner

Summary

Adds a make_video dialog tool. The bot turns a request into a 5-second silent FastFrames clip (832×480, 16 fps) on draw-api and replies with it as a Telegram animation. Clients show an MP4 without sound sent through sendAnimation as a GIF.

  • Prompt optimizer (prompts/video/optimizer.prompt, openplotva-media::video): rewrites the request into one English Wan-style shot description (style → setting and light → subject → "First… Then…" action → camera move → look), 80–120 words, with no negations or readable text. It also classifies the request as safe, adult or forbidden. It runs on the existing media_prompt_optimizer route.
  • Queue: taskman VideoGen jobs go to a new video queue. Each user can have one active clip. The usual enqueue rate limit and a delivery obligation apply, so a lost clip still gets a failure notice.
  • Worker (video_jobs.rs) runs these steps:
    1. checks the chat's send_video permission;
    2. optimizes the prompt;
    3. answers a forbidden request with the blocked message;
    4. calls draw-api fastframes_generate through Discovery;
    5. sends the MP4 with sendAnimation, behind a spoiler when the verdict is adult.
  • Entry points: the agent picks make_video for requests for a video, clip, GIF or animation. The !video, !видео and /video@bot shortcuts enter the agent with make_video as the explicit intent.
  • Switch: VIDEO_GENERATION_ENABLED (default true) disables both the tool and the worker.

No schema migration: the job reuses ImageJobData (original_text, prompt = brief, meta.caption).

Verification

  • cargo fmt --all
  • cargo clippy --workspace --all-targets -- -D warnings: clean
  • cargo test for openplotva-app (1458 passed), -dialog (110), -taskman (68), -media (63), -llm (236), -config (57), -storage (187), -telegram (237), -prompts (10)
  • New tests cover:
    • optimizer render and decode;
    • draw-api video contract (endpoint, idempotency key, body, decode);
    • queue limits and permission refusal;
    • worker delivery, block and failure paths;
    • shortcuts;
    • dispatch fallback to the message text.
  • draw-api fastframes_generate already runs on the farm: about 10 s per warm clip.

Adds a make_video dialog tool that turns a request into a 5-second silent
FastFrames clip on the farm's draw-api and delivers it as a Telegram
animation, so clients show it like a GIF.

- video/optimizer prompt: rewrites the request into a Wan-style English
  shot description (style, setting and light, subject, "First... Then..."
  action, camera move) and classifies it safe/adult/forbidden.
- taskman VideoGen job on its own "video" queue with one active clip per
  user, a delivery obligation and the image-generation timeout.
- video worker: optimizes the prompt, blocks forbidden requests, calls the
  draw-api fastframes_generate endpoint through Discovery, and sends the
  MP4 with sendAnimation (spoiler for adult clips).
- !video, !видео and /video shortcuts enter the agent with make_video as
  the explicit intent.
- VIDEO_GENERATION_ENABLED (default true) turns the tool and worker off.
@github-actions

github-actions Bot commented Nov 18, 2026 •

Copy link
Copy Markdown
Warnings
⚠️ Prompt files are cache-sensitive contracts; confirm prompt shape and role markers were preserved intentionally.
⚠️ Large PR (2112 changed lines). Consider splitting unless this is generated/mechanical.
Messages
📖 Danger checked 24 changed file(s).

Generated by 🚫 dangerJS against 00e754d

The delivery obligation watcher treats a completed ticket without a result
message as orphaned and sends the failure notice. The video worker now
stores the id of the sent animation, or of the safety refusal, on the
ticket before it completes it.
@iamwavecut

Copy link
Copy Markdown
Owner Author

Re Danger warnings:

  • Prompt files. prompts/aifarm/system.prompt changes one line: make_video joins the tool list on the side_effect_end line, next to draw_image and generate_song. Role markers, partials and the rest of the cached system text stay as they were. prompts/video/optimizer.prompt is a new template with a system and a user role. It renders only for the video optimizer, so no existing prompt cache changes.
  • Large PR. About half of the diff is tests. The rest is one feature that works only end to end: tool spec → dispatch → scheduler and queue → worker → Telegram delivery. A split would merge parts that have no caller.

@iamwavecut
iamwavecut merged commit cb081c8 into main Nov 18, 2026
12 checks passed
@iamwavecut
iamwavecut deleted the feat/fastframes-video branch November 18, 2026 23:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant