Innovate Futures @ Benji
patreon
(ComfyUI Workflow) DiT Talking Avatar - LTX+KokoroTTS+LateSync Ver.20250123
🕑 Added 2025-01-22 19:54:26 +0000 UTCHey guys!
When you have try LTX, you know that this model need long form of video types of prompt in order to make some good motion, no matter what.
And if you use image caption for LTX img2vid , for the generated video, 90% possibility the character are stay still.
But this can be a benifit for creating talking avatar, becasue the character don't move too much.
So, I skip the part using LLM to convert image prompt to video prompt; I tested using caption from Florence 2 as the input text prompt to generate the character portrait video.
As the result, it's very suitable for lipsyncing.
So I combine KokoroTTS and LatenSync for the character talking part.
Any I got this...
Although I have not organized in groups, kind of messy. For functionality, it works.
So for the input:
A close-up portrait image. (real person or AI character whichever you like.)
Example Text Prompt from Florence 2:
"The video is a close-up portrait of a young woman with shoulder-length brown hair. She is wearing a shiny, metallic purple crop top with a silver zipper on the front. The top has two black straps on either side of her chest. The woman is looking directly at the camera with a serious expression on her face. The background is blurred, but it appears to be a futuristic setting with a circular light shining down on her. The overall mood of the video is mysterious and futuristic.The scene is captured in real-life footage."
We don't need to put action, movment in prompt, or if you want the character running while he talk, of course you can :)
And I put 2 upscale models at the end, just to enhance the blurry mouth from LatentSync 512px resolution.
Check it out if you want to try this, attached below. Have fun :)
Comments
BBCB4S1C
install in your comfyui/venv/scripts folder
Tom Sepe
I had to open powershell as an administrator and run pip install pylatexenc but the TTS plugin still would not install!
