iPhone 12 Pro 3D scan --> Fusion

I just got an iPhone 12 Pro and did a quick 3D scan and wanted to pass on some results for anyone else who is interested.

The iPhone 12 Pro (and iPhone 12 Pro Max) both have a lidar scanner in them, so can more accurately gauge depth in an image. The free app 3D Scanner App is (https://www.3dscannerapp.com) is available on the iPhone App Store. With it, you can scan a scene/object.

After scanning, you can select the scan and pan around in it, then you have a More menu that has some critical options. I'm not sure what Refine Scan does yet. But I was able to use Simplify Mesh followed by Smooth Mesh to come up with a fairly small and useful result. It warned me that these operations would remove the texture, but after doing them I was able to use the Color option to reintroduce the texture back onto the mesh.

You can then Share an OBJ file that will have an accompanying JPEG texture image and a .mtl file to tie the two together. I chose to share as a single Zip and used Air Drop to send it to my laptop. There are other options.

You can import the OBJ file into Blender (I'm using the latest, 2.91) and turn on Viewport Shading and you should see your scene, with texture. (All three files should be in the same directory so Blender sees the two texture-associated files when you load the OBJ.)

Blender View

Blender View.jpg (750.36 KiB) Viewed 217 times

Then you can export an Alembic File (.abc) which can be imported into Fusion. I exported the whole "animation" -- nothing was animated even though the default Blender timeline is 250 frames or something like that.

Fusion View

Fusion View.jpg (716.94 KiB) Viewed 217 times

The texture by itself looks weird, but the final product looks the same as in 3D Scanner App and Blender. Scroll down in the attached image to see the Fusion Nodes I used. There may be a better way to do this, but it's what worked for me. I loaded the Alembic file with the 3D node for Alembic files, and then loaded the texture JPEG and applied the texture.

Just a quick scan, not rigorous. You'll notice a strange bump on the chair's arm, and lots of missing background -- the background was mainly included by chance as I went around the chair. You could really polish things, but I figured this might be of general interest to folks. With a phone, you can go all Hollywood and make a reasonable, textured 3D scene and bring it into Fusion.

I didn't get down on the floor to allow it to scan under the chair, so the feet and underlying carpet are part of the texture on the boxy "chair" itself. Further refinement in 3D Scanner App would help a lot -- I can't find any documentation, so it's a bit of a poke-around-n-experiment process, but the tools I've figured out are very nice. I used Low Resolution because of dire warnings that High Resolution should only be used on small objects. Further cleanup could be done in Blender, then bring it into Fusion and work away.

Note that the texture is just an image of the actual object, so has the lighting of the actual scene baked into it. So if you really wanted to relight it, you'd probably want to light the object/scene as diffusely as possible. Maybe even a ring light around the phone's camera lens -- not blocking the lidar, of course -- would be helpful. Or just light the scene in 3D as it's lit when you scan it.

Fine, need to learn how to remove speculars and reflections from textures. So that you can make them yourself in 3D. This is very interesting and promising.

Good point The baked-in nature of the texture includes not just lighting/shadows, but also specular highlights, reflections, etc. So to place the object in a totally different lighting scenario what I have shown is not good enough.

For official use, you'd probably want to do a retopology of the object to dramatically simplify the polygons and to fix the features that are shown in the texture but not properly expressed in the geometry due to the limited resolution of the lidar. For example, in the scene I posted, if you move closer to the floor you can see that "underneath" the chair is actually filled in but appears to show under the chair because of the texture. The scan wasn't thorough enough to detect that there's greater depth under the chair. Some of that may be the limited resolution of the iPhone lidar and some is my haste and the "low res" setting in the application.

Here is the UV editor in Blender to show what the texture looks like.

Blender UV View

Blender UV View.jpg (504.05 KiB) Viewed 126 times

I don't know if there are tools to clean up a UV map like that into more contiguous sections. If so, I imagine that you could export it and use a clone brush to clean up specular highlights, etc. Of course, some of those specular highlights give the impression of 3D-ness, like the wrinkles in the leather which are not really reflected in the polygons.

Though if you need something quick and based on the actual scene -- say you wanted to insert something next to the chair, or a person standing behind it, I think you could do something fairly quickly with some cheats and things like reflection/shadow catchers to make it fit in that environment. Worst-case scenario, it serves as a pretty authoritative reference of the scene.