Nick Fisher
gumroad
Blender LiveLinkFace Add-On
🕑 Added 2024-08-22 09:46:35 +0000 UTCBlender add-on for ARKit blendshapes animation with the LiveLinkFace app
An add-on that lets you use the iPhone LiveLinkFace app with ARKit blendshapes inside Blender!
The free version supports:
- importing recorded animations from a LiveLinkFace-formatted CSV file;
- streaming ARKit blendshapes live from the iPhone LiveLinkFace app to any mesh with matching shape keys and/or bones.
In addition, the full version also supports:
- recording animation streams direct to Blender actions
- one-click setup for driving head/neck rotation from the HeadPitch, HeadYaw and HeadRoll blendshapes
- filtering to smooth out jerky animations and improve the look and polish of your captures
- blendshape remapping, allowing you to easily bind animations to a mesh with ARKit blendshapes named differently from the format used by the LiveLinkFace app
Both versions require:
- an iOS device with a TrueDepth camera (iPhone X or higher)
- Live Link Face on the App Store (apple.com)
- Blender 3.1 to 3.6, or 4.0-4.2 (earlier versions may work but this is untested)
- your own mesh rigged with ARKit shape keys/armatures.
If you do not have your own mesh, you can download one from the following link:
MetaHuman Head - 52 blendshapes - ARkit ready (Free) (gumroad.com)
(though be aware that this mesh is not rigged for head pitch/yaw/roll, so those ARKit blendshapes will obviously not function).
IMPORTANT - this is an unofficial add-on and I am not associated with LiveLinkFace, Unreal Engine or Epic.
Join the discussion on Discord via https://discord.gg/vpugcXQFRg
General Setup
1) Download the add-on:
For Blender >= 4.4:
The above version should also work with Blender < 4.4, but if you encounter any issues, try the following version first:
2) Start Blender and open a file containing an object rigged with ARKit shapekeys
3) Click Edit->Preferences, then Add-ons in the left pane
4) Click "Install", then select the file you downloaded in (1) and click "Install Add-on"
5) Click the checkbox next to "3D View: LiveLinkFace Add-on" to enable the add-on and close the Preferences panel
6) In the 3D view for your scene, select the "LiveLinkFace" tab on the right
7) Select the mesh you want to animate, then click the + button in the "Target" box. The name of your mesh must appear here; if this is empty, nothing will be animated.
Streaming
1) Start the LiveLinkFace app on your iPhone and click the Settings icon in the top-left
2) Tap the "Live Link" row, and if the IP adddress of your host machine (the one that you are running Blender and the LiveLinkFace add-on) does not appear under "Targets", tap "Add Target" and add the IP address.
3) Back in Blender, most people should be able to leave the Host IP/Port as the defaults (0.0.0.0/11111). If this does not work, set the IP address and the port to the exact values you specified in the LiveLinkFace app.
4) IMPORTANT - make sure that "Protocol" is set to "4.25 and Later".
5) Click "Connect", and you should see your mesh in the 3D preview sync with the LiveLinkFace stream.
CSV Import
1) When you record a session via LinkLinkFace, a CSV is saved alongside the raw video in your phone.
2) This CSV can be imported into Blender by clicking the "Load from CSV" button and selecting the file. Ensure that the correct target mesh is selected (step 7 under "General" above).
3) Two new actions will be created (one for shape keys, one for bones) to animate the target mesh based on the values in the CSV. You may need to change the source framerate for your Blender project to match the framerate of the LiveLinkFace app (60fps).
Bone animations
Some LiveLinkFace/ARKit blendshapes may not correspond directly to shape keys on your mesh, but rather to bone rotations (e.g. HeadPitch/HeadYaw/HeadRoll).
One option for matching the two is to create a custom property on your target mesh that matches the name of the ARKit value (e.g. "HeadPitch"), then create a driver for the bone (e.g. "neck") that matches the orientation you wish to animate. Streaming to the plugin or loading from CSV will then animate the bone orientations according to the range (if you choose to create a driver using quaternions, you can probably use the ARkit value directly, since -1/1 for the ARkit values will correspond to a -90/90 degree rotation on the bone).
Updated instructions (10 June 2024)
The current version should automatically add data properties for HeadPitch, HeadYaw and HeadRoll to every mesh that you add as a target and click "Stream". You then need to use these values to drive your head/neck bone.
Go to pose mode, select your head/neck bone, change to XYZ Euler, then add a driver for each X/Y/Z.
The exact order will depend on the orientation of your head/neck bone, but if your bone has Y-up and Z facing towards camera, then X should correspond to HeadPitch, Y should correspond to HeadYaw, and Z should correspond to HeadRoll.
The value from LiveLinkFace will be between -1 and 1, so you need to convert that to rotation degrees for your bone. Strictly speaking that's between -90 degrees and 90 degrees (so a value of 0 is at rest with head pointing forward), so the "expression" field would be var * 90, but I've found 90 degrees is a bit too extreme. Around 60 degrees is better, so var * 60. If you want to play around manually after the drivers have been applied, you can select the mesh, go the custom properties tab, then slide the values for HeadPitch etc up and down.
FAQ
Streaming isn't working.
There seems to be an issue with the LiveLinkFace app at the moment, a recent update may have broken streaming non-Unreal Engine blendshapes. No ETA on a fix unfortunately - the only workaround is to load via CSV.
This should be fixed as of v0.0.5 of the plugin. If it's still not working for you, please let me know.
I get an error "internal error setting the array" when trying to load via CSV.
Change to "Blender File" in the Display Mode dropdown, delete the LiveLinkAction_shapekey action, and import the CSV again.
There's some combination of stashing/deleting actions in the NLA stack that deletes the action but somehow leaves it attached to the target objects, which then breaks the import (it also seems to affect GLTF export separately too). I haven't figured out exactly what steps can reproduce this yet, but the workaround should work for imported CSVs.
