Animation system 3 – Automatic Lip Syncing
For Automatic Lip Syncing one has to extract the Phonemes from a recorded speech and map it to particular Visemes in real time or pre process it and store it in some way so application can use it later in real time.
Phonemes is the smallest unit sound of a word that we can hear. And Viseme is the smallest unit of speech that we can see or it is the shape that our mouth makes when making a particular phoneme.
Deaf people have learned to take great advantage of this concept when they do lip reading
So there are couple of techniques to analyze speech giving different results. One of them involves analyzing the amplitude of speech sound and that’s what I have used for this demo. Beauty of this technique is that it can be used on different languages without any extra work.
Here’s the Demo video of what I achieved with my system –
I also tried some songs instead of speech just to see how does my system responses (a fun exercise) –
So quality mayn’t as good as other techniques requiring Fourier Transforms to analyze speech data but for that can’t work alone. If we have to go for that quality we should also implement Face bones system (something like Factial Animation Coding System) and we also need very good artwork also made specific for our requirements. Well that’s something I am looking forward to do sometime in future but for now this system is good enough for what i needed for upcoming tech demo of my engine.
For people who are also interested in Lip Syncing must check these 2 articles –