The goal can be simply put:  I want to automatically turn my 3d scan in to an avatar.  For Jaunt’s yearly Hacakthon in 2018, I decided to try and tackle this.

I found out right off the bat, it turns out that there’s a library called “Pinocchio” out of MIT that did the hardest part for me — it takes an oriented mesh and spits out bone positions and skin weights.  It has some drawbacks though; it requires a pristine mesh (manifold, closed, only one part), it requires careful calibration, and it often gets confused around the shoulder area.  So in creating a pipeline that could be made to run automatically, I had to deal with these issues.

By the end of the Hackathon, I had iterated to a pipeline:

1. Person gets scanned in a ‘T’ pose on a scanner.
2. The 3d model is produced by the scanner and handed to the rigging pipe via UI.
3. The model is passed through a Meshlab .mlx script using Meshlabserver to clean the mesh.
4. The model is passed through a Meshlab .mlx script using Meshlabserver to generate a poisson reconstruction of the mesh.  (This was necessary because the source meshes from our scanner were moderately dirty and didn’t play well with MIT’s Pinocchio library)
5. The poisson-reconstructed model is passed to the Pinocchio library, which returns some bone positions and skin weights.  This result is displayed with OpenGL and some open source mocap animations.

Planned steps that were implemented by the end of the Hackathon, but not yet functional:

1. The results of the rigging are projected on to the original mesh, adding bones and skin weights.
2.  The now rigged original mesh is exported to an FBX file.  (This was mostly done but getting materials added to the FBX file and correctly inverting the transforms applied by the Pinocchio library proved too much of a time sink in the quickly dwindling latter half of the Hackathon)

 

This was all wrapped up nicely inside a JUCE application, and the source code is available here.

The results speak for themselves:

 

Of course, if the mesh isn’t aligned correctly when being loaded in, the results wind up more amusing or horrifying than useful.  But making it misbehave in this way was one of the most popular activities during the Hackathon demos.