Blend Systems

Blend Systems are an important component of LipSync. They are responsible for updating the visual representation of the character in the scene.

Overview

Blend Systems are classes that inherit from the built-in RogoDigital.Lipsync.BlendSystem class. They exist as hidden components on a GameObject that has LipSync or Eye Controller added to it, and they are managed automatically.

The purpose of Blend Systems is to allow LipSync to be as flexible as possible by not actually doing the animation itself. In terms of the Blend Shape Blend System, instead of LipSync directly storing references to Skinned Mesh Renderers or setting the values of blend shapes, it simply passes an index and desired value to the Blend System to handle however it wants.

Blendables

A Blendable is any aspect of the Character's representation that the Blend System can alter. In the Blend Shape Blend System they are Blend Shapes, in the Sprite Blend System they are layer/sprite combinations stored in the Sprite Manager component.

Included Blend Systems

A few Blend Systems are included out of the box, with others available for download from the Extensions Window.

• Blend Shape Blend System This is the most common Blend System. It uses Blend Shapes on Skinned Mesh Renderers. It requires one main Mesh Renderer, and also supports any number of additional ones so long as they have the same set of Blend Shapes.

• Advanced Blend Shape Blend System Expanding on the Blend Shape Blend System, this blend system allows you to use any number of completely unrelated meshes together. Whereas the normal blend shape system requires all meshes to have the same set of blend shapes, in the same positions, the advanced one adds a Blend Shape Manager component that stores references to different blend shapes on different meshes and groups them together. Just add the meshes you want to use with the Skinned Mesh Renderer field at the top, then press the Build From Names button to apply the meshes. You can then use any of the blend shapes in LipSync as normal.

• Sprite Blend System This Blend System is designed to work with 2D Sprites. When selected, a Sprite Manager component will also be added to the GameObject. You can use this component to define one or more 'layers' (Sprite Renderers) and as many different sprites as you wish. You can then select layer/sprite combinations to use in poses. Each layer will display the sprite with the highest blendable value, and will simply swap sprites when one value becomes greater. For example, you could use this functionality to create one foreground layer for the mouth and a second layer for the rest of the face. Emotion poses could then use the background layer and Phoneme poses would use the foreground one, to allow both to display independently.

• Texture Offset Blend System Works much like the Sprite Blend System, but instead sets the Texture, Texture Offset and Texture Scale of a texture on a material. If correctly planned-out, you could use this technique to animate an anime-style face on a character.

• Bones Only Blend System This is more of a pseudo Blend System, as it doesn't actually do anything. It simply disables all blendable related functions to make the UI clearer if you only need to use bone transforms.

Writing New Blend Systems

Part of the power of Blend Systems is the fact that anyone can write their own. This allows LipSync to work with almost any character imaginable - anything from basic sprites and 3D models, to crazy light or particle-based systems.

In order to function as a Blend System, your script needs to inherit from RogoDigital.Lipsync.BlendSystem. This base class has a number of virtual methods you can override in order to implement your Blend System's functionality. The most important parts of a Blend System class are as follows:

While you can write all this yourself, the simplest way to get started is to use the provided sample code. Just right-click somewhere in the Project panel, and choose Create > LipSync Pro > Empty BlendSystem.

OnEnable()

Unlike a standard MonoBehaviour, you need to override OnEnable like so:

public override void OnEnable()
{
    base.OnEnable();
}

In this method, you can do any initialization your Blend System requires. You should also set the value of blendableDisplayName, blendableDisplayNamePlural, noBlendablesMessage , and notReadyMessage here. The two DisplayName variables are used in place of 'Blendable'/'Blendables' in the inspector, e.g. 'Blend Shape' or 'Sprite'. You can also optionally set the values of blendRangeLow & blendRangeHigh to re-map the minimum and maximum values on the blendable sliders in the Pose Editor. These are set to 0 and 100 respectively by default.

It is important to call base.OnEnable() after setting these variables to ensure that the Blend System works correctly.

GetBlendables()

This method is used to poll a Blend System for the blendables that can be selected in the editor. It should return a string[] containing the names of each blendable in its corresponding index. By convention, the index itself is appended in parentheses, though this is not strictly required.

In this method, you should get a list of available blendables, register them internally using AddBlendable(int index, float startingValue) if necessary and then return the list.

Below is the GetBlendables() method from BlendShapeBlendSystem.cs for reference:

public override string[] GetBlendables ()
{
		if (!isReady || characterMesh == null)
				return null;

		bool setInternal = false;
		string[] blendShapes = new string[characterMesh.sharedMesh.blendShapeCount];
		if (blendableCount == 0) setInternal = true;

		for (int a = 0; a < blendShapes.Length; a++)
		{
				blendShapes[a] = characterMesh.sharedMesh.GetBlendShapeName(a) + " (" + a.ToString() + ")";
				if (setInternal) AddBlendable(a, characterMesh.GetBlendShapeWeight(a));
		}

		return blendShapes;
}

SetBlendableValue(int blendable, float value)

This method is called whenenver a blendable's value needs updating. E.G. in the editor when a slider in the pose editor is moved, or during an animation at runtime. It should do whatever is required to actually update the character's visuals, and also call SetInternalValue(int index, float value).

It's important to keep the internal values synchronised with the real ones, as this is the value that will be reported when GetBlendableValue(int blendable) is called. This helps with preventing conflicts when multiple scripts attempt to change the same value.

Again, here is what this method looks like in the Blend Shape Blend System:

public override void SetBlendableValue (int blendable, float value)
{
	if (!isReady || characterMesh == null)
			return;
		
	characterMesh.SetBlendShapeWeight(blendable, value);
	SetInternalValue(blendable, value);
	
	foreach (SkinnedMeshRenderer renderer in optionalOtherMeshes)
	{
		if (blendable < renderer.sharedMesh.blendShapeCount)
			renderer.SetBlendShapeWeight(blendable, value);
	}
}