Well, ok, rhubarb works for linux, they just don't mention that in their README.md. That's good. I tried it out. It works... but I think I'll still invest some time in trying to understand how to use pocketsphinx. Seems like a useful skill to have.
However, to keep moving forward with the Open Movie Project (OMP), I'll see how I can use rhubarb to do lip-synching. There is already a blender plugin, so I need to see how that'll work and what kind of results I can get out of it. I would like to have tighter integration with MB-Lab though, to streamline my workflow.
Here are the tasks I foresee
- Create an expression control rig.
- Once drivers are added to the shape keys the MB-Lab expressions sliders don't work
- Create a Phoneme Rig.
- This will basically control the different mouth shapes.
- Create a pose library for the different phonemes.
- I have to see how that will work with rhubarb
Been a while since I updated this blog. This is going to be a long haul project.
I finished a draft of the script. And I wanted someone to go over it. Honestly, it's hard finding someone to collaborate with. I decided to simply higher someone with writing experience to go over the script and polish it. I met one guy and we got to chatting. He had a few good questions, which motivated me to go back and re-examine the story. Sort of go back to the basics. I used the story I had already developed as basis for a new version of the story. I'm planning to spend a couple of days with that writer to fine tune it and make sure it's complete and captivating.
In the meantime, I ran into a really interesting show called, "Undone". It's been developed in a really interesting way. They call it rotoscope animation. Basically they film the actors and then paint over them, and create the rest of the elements in 3D or 2D animation. "Scanner Darkly" was done in the same way I believe. Here is a trailer
After watching that show it got me thinking about creating a new visual style for the show. After going through the animation test I've done, I came to realize the importance of real actors to get the emotions right. Especially, if I'll be using character generation tools like MB-Lab, I won't have the fine control to express the emotions. The tool just doesn't have the rig to do that.
What would be ideal is a mix of real actors and 3D animation. But doing something photo-realistic is simply beyond my means. I'm currently asking myself this question: Is there a visual style which can capture the actors emotions, but allow me to build a grand world without having a billion dollar budget?
Investigating...
Alrighty then... I've reached the end of a milestone. Took a bit longer than I expected, but I got there. I set off to determine what I need in order to actually make an animated short (which turned into a series). I knew that I wouldn't be modelling my characters from scratch, so I wanted to use an open source character generator of some sorts. The ones available were makehuman and MB-Lab. After comparing both, I decided to generate most of my characters with MB-Lab. Not saying I'll never use Makehuman; I will, especially for younger aged characters. For example MB-Lab doesn't generate characters younger than 18 years. However, I decided to concentrate my effort on seeing how to modify MB-Lab to fit my needs. I ended up creating a few additions to MB-Lab which I need:
- Face Rig to make it easier to animate facial expressions
- Phoneme Rig to make it easier to lip-sync
To do both of the above, I needed to do so some coding. This took quiet a bit of time. You can look through the history in this blog to see what I did. Suffice it to say, it required me to learn how to create blender add-ons, learn how to create drivers for shape keys in blender and a bunch of extra mambo jumbo, to be able to make the face and phoneme rig work.
I also wanted to automate the lip-syncing as much as possible. I looked at existing add-ons to do that. But I decided to create my own: Yet Another Speech Parser (YASP). This has two components. A C program which uses pocketsphinx library to parse audio clips and generate a phoneme description in JSON format. Basically a JSON file describing the phonemes and their timing. I looked at a couple of speech parsing libraries out there, and I settled on pocketsphinx. The second part is a python blender add-on which takes as input the audio file and text transcript of the audio file and creates the animation. This actually worked more or less well. It creates a first pass animation, which I then fine tune. One of the weaknesses of MB-Lab is its lack of finer control over the face expressions. I think Makehuman has a better face rig, but I digress. It is what it is.
Along the way I got side tracked by trying to create an automated face expression tool, which uses an open source library called OpenFace to analyze footage and reflect the facial expressions from real footage on the animated character. I did try to use it, but I found that I can get better facial expressions through animating by hand. It's actually more fun too.
Once I started doing some simple animation, I quickly realized that the MB-Lab Rig isn't very nice. Thankfully, there is an add-on which generates a Rigify rig from the MB-Lab rig. Of course I had to tweak it and change it to work the way I wanted it to. That took a while. But I really like Rigify. It's a nice rig.
It's worth mentioning that I adopted Blender 2.8 while it's still in alpha stage. So I got involved in porting some of the add-ons to Blender 2.80 as well as opening tickets and submitting a couple of python patches. If you fish for my name you'll find me buried somewhere in the commit log.
Once I was at a point where the tools I created were mature enough, I decided to apply them and my skills to create an animated scene. I flip-flopped a bit with this. I started off thinking I'll do an independent short, then I changed my mind and decided to create an existing movie scene. After a bit of thinking I settled on the latter, but which scene should I do? I was first thinking of animating a Captain America Civil War fight scene, and I actually started. I created a run cycle, and then connected that to a car jump, but then I stopped going down that path. I just wasn't that into Marvel movies to justify spending a lot of time animating one of their fight scenes. I finally decided on a Star Trek First Contact scene. I'm a trekkie at heart and I'm not ashamed to say it. I love TNG (that refers to Star Trek: The Next Generation, for you non-trekkies out there). Anyway, I sorta settled on working on that and I got down and dirty with setting up the scene and doing the character animation, etc. Below is a video walk through of the final scene I ended up with.
The TNG scene I animated was training ground to see how I can use the tools I have at hand to create my series. As far as I can tell, these tools are sufficient to create game like animated scenes, sorta like this. The quality isn't gonna be extremely cinematic, but I really just want to get to create my story and do what I love to do, which is story telling and film-making
The next step for me is to start creating the animatics for my script. I'm still toying with the idea of pushing the TNG scene a bit further and creating some proper locations, polishing the lighting and atmosphere. The jury is still up on that.
Still working on the practice animation scene. I'm further along now. One thing I'm starting to realize with that scene is the cuts. There are some unnatural cuts that break up the action of the scene. They were done to insert the reactions of the actors. But when I'm trying to translate that into one continuous scene, the pauses feel unnatural, so I have to keep adding some actions to smooth the transitions.
Been looking at open source Automatic Speech Recognition (ASR) Engines. What I want to do is integrate with an ASR engine so I can run speech audio through it and generate timing information on when words (more particularly phonemes) were spoken. I can then take this information and create a first pass lip-sync animation in ManuelBastioniLab.
I looked at a couple of ASR engines one called "DeepSpeech" and the other called "pocketsphinx". The former is written in C++ and the latter is written in C. Man pocketsphinx is a whole lot easier to understand. And it has the functionality I want. Deepspeech doesn't produce timing information. Well they do, but it's not exposed in the API, which means if I need to use it, I have to take the initiative and expose this information in the API. I actually thought about it, but it's a lot more work than I'm willing to undergo at the moment.
pocketsphinx is used in rhubarb-lip-sync. However this application runs only on windows and macs (as far as I can see) and I'm not a windows user. I have seen the light and abandoned windows. In other words, I need to have something working for linux. I'm cool if I only support linux.
rhubarb-lip-sync is also a generic application designed to work with multiple different application, like after effects. Anyway, the end result for me is to develop an application which works very similarly to rhubarb-lip-sync, but is directly integrated in blender.
We'll see how that goes.
P.S. If you're interested in learning more about pocketsphinx, here is there wiki.
I thought I'd share the updates that I've been doing to the script. I use git to keep track of the different versions, as I find the color coding version is not enough. I do a lot of changes and I'd like to be able to roll back to them. Also git allows me to enter a commit message per change so I know what I changed. Anyway, here is the story update log:
I finally have a script. I'm still working on polishing it, but I thought I'd share the storyboard.
So I think I'm drawing to a close on the script. I'm sure there are ways to improve it, but at this point I want to move to doing some pre visualization of the first scene of the script.
I've been debating whether I should hire someone to do a script analysis and suggest areas of improvement. My problem is this type of service is usually a hit and miss service. It would be nice if I have a bunch of people who would be willing to read the script and provide their independent feedback. This would be highly useful, but it's hard finding such an audience. Guess we'll see.
Just a quick update. I finally decided on which scene I'll animate for practice, and it's this one. As I work through the scene, I figure out what works for me and what doesn't and how to improve my workflow. Here is what I have so far:
2018 passed pretty quickly. It amazes me how fast time goes. The passage of time has its advantages and disadvantages. One of its advantages is it motivates people to accomplish goals. If I had all the time in the world to get something done, it's likely I won't get it done with any urgency. But I know my time is always running out. A fact which motivates me to work and get things done.
Putting philosophy aside, I think this week has been somewhat beneficial. I worked through the story with Nicole a bit more. I think we're working through the plot holes. I've also figured out something new about the ManuelBastioni Lab. It has a very extensive set of shape keys, which in theory should be a replacement for a facial rig. You know what that means, I'm using it in my proof of concept short. It's superior to Makehuman. The control it gives you over modifying the character is better, the skin shader is better, the weight painting is a lot better, the muscle system and rig are awesome. Only disadvantage is it can't be used to create small children, so I'll need to use Makehuman for that.
The next step is to fit the ManuelBastioni character with some clothes, then we're set to start building the location and moving forward with the 1 minute short. I'm excited. Hopefully, I'll have something completed in the first month of 2019.
I'm disappointed Manuel Bastioni has decided to stop supporting the lab. I understand where he's coming from. No one has showed him any support and it does take a lot of time to get this lab working properly.
I put some time into porting it to 2.80. I have taken some clothing assets from Makehuman, and I'm planning to make them available for the ManuelBastioni Lab. Maybe that'll encourage others to contribute assets, which will be all around useful.
I do think having a facial rig is useful, but I hate weight painting. I need someone to help out with that. Hopefully, we'll be able to get that project completed in 2019.
YAAM seems to be gaining some traction. I put a post on blendernation: https://www.blendernation.com/2018/12/28/free-download-asset-manager-add-on/ Got 45 shares... which ain't bad. And another dev is also working on it, so I think it could get some cool features implemented in 2019, which will make it even more useful.
I also made some small contribution to the Blender add-on code base, mainly around porting add-ons from 2.79 to 2.80. Not too shabby. Looking forward to contributing in more significant ways in the future, if I get an opportunity.
Well, signing off 2018. Happy New Year to all. And hope you accomplish your dreams in 2019.
Yup! Can you believe it? I not only have one script I have an entire season, 10 episodes. Granted each episode is about 3-5 minutes long, but hey, it's a full story. The way I figure it what I wrote can be a Youtube/online season, or it can be a 45 minute pilot episode. Since I'm making it, I'll go with the former.
After months of brooding and moaning and chest beating, I sat down and wrote all 10 episodes in two days. Weird, eh?
Here comes my next dilemma. I want to share it publicly, but like any artist I have some insecurities that it'll get torn to shreds. Of course, there isn't that many people (if any) following this site, but I'm still not sure if I should share it publicly.
Here is what I'm thinking. I'll keep it under wraps for now and I'll share the output of the actual production cycle publicly, including all 3D resources, files, etc.
But I'm looking for feedback, so if you're interested in reading the scripts and providing constructive and detailed criticism ping me here.
It's been a while since I updated this blog. I've hit a bit of a lull. Work has been taking all my time. I've been thinking though, where to go from here. There are two directions. I can keep going down the technical path. Try to enhance my facial expressions system. Or I can start revisiting my story. I really want to produce something creative. So I decided to head down the creative road. I'm working on a short film idea that has some potential. Once I have it well formed, I'll share it on here. I still like the direction the story I have outlined here is taking, but I'll take a break from it and explore a slightly different world.
My dad used to tell me write what you know. I never understood what that means. If I only write what I know, I'll write really boring stories. But I think I have modified his advice to something that makes more sense to me. Write the characters you know. The most important aspect of a story is the character. If you analyze any movie you like you'll come to the same conclusion as I did. Characters are what draws you into the story and keeps you glued to the screen or the pages. They are the ones who make you feel something. You relate to them, and you care what happens to them. The challenges and the "plot" is what brings out the personality of the character. They are the events which show what the character is made out of; their courage; their compassion; their fears; their love. And these aspects is what makes the viewer identify with your character.
As a writer the character, I believe, has to come from a personal place. In essence they are a part of you. If you don't write from what you know, the character you write will feel contrived and fake. But if you write from your own experiences, then they might, just might, be relatable. For this reason, I decided to write a character which I have personal experiences with. The world I'm thinking of is not a real world. It's a futuristic world, which I haven't seen done before. The character will leave the world she knows behind and be thrust into the other one with no way back. She will need to navigate this new world and rediscover the purpose she thought she lost.
YAAY... I have something working. Now I know you might think it ain't great animation. But in order to get to great, you need to pass this post first. Keep in mind, this is all automated. In a few clicks, you can get a lip-sync. There are still some improvements I'm planning in two areas. 1) The poses. I need to clean up the poses for each phoneme a bit more. 2) Keyframing. I'm going to introduce a polish step to clean up the keyframes. This step will require some trial and error to figure out a bunch of lip-sync heuristics. As an example, I don't have to turn off a pose completely if it'll morph a few frames later. Things like that, I believe will make it look a bit better.
Of course at the end, it'll require some manual clean up to really polish it.
I'm also planning on adding some emotional meta data. This information will translate into corresponding facial expressions and head movements. I'm thinking the input will be a wave clip and an XML file, which would include the transcript and emotional meta information, sorta like this
<transcript>
<panic>We have to get out of here</panic>
<sad>but with a broken leg he can't come with us</sad>
</transcript>
Anyway, still thinking on it.
It's a lot of work, especially when I have to learn how to interface with the Blender Python API. It's a lot of visits to the API documentation, and looking at existing add-ons to understand how they did things. Not to mention a lot of trial and error... But I'm getting there.
I'll have it done by Christmas 2018... It'll be my Christmas gift for myself! And for whoever wants an asset manager.
I decided what to call it...
YAAM: Yet Another Asset Manager
I think it's kinda clever... yet not completely original :) It's a spin off YAML: Yet Another Markup Language.
As I was working through my pipeline, I ran in the first obstacle. I need an asset manager built into blender. An asset manager is a key piece of the pipeline needed to stay organized. Otherwise, I'll be forgetting where everything is.
I looked around. I found one, called Asset Flinger. I looked through it and it looked cool. But it had two problems: 1) it only worked with .obj files and 2) It only works for Blender 2.79. I'm building my entire pipeline around Blender 2.80. So I decided to convert it to blender 2.80. After converting it, I decided to spring forward and actually write my own Asset Manager, which I can use for the open movie. The idea is I should be able to work with different types of assets; obj, 3ds, blend, images, materials, etc.
I forked the Asset Flinger repository. The 2.80 compatible add-on is available here.
To be honest I found one for 2.80 on blender market for $40, but I decided against buying it. I want to build my own. I'm now 100% in Software Development mode.
I had only dabbled with blender add-ons, but nothing serious. Writing this add-on has been educational for sure. It's been 3 days working on it. I got the interface there and currently working on the functionality. Here is a PDF of my design document. And below are a couple of screen shots. Once I have it complete I'll upload the code on github.
Although I started with the Asset Flinger code, what I'm writing is its own add-on. I'll create a new repository for my Asset Manager. Felt like I should clarify this, since the images below carry the name "Asset Flinger", but the final add-on will not be named that.
