Best Practices for Filming for Animate 3D? (Will Tracking stickers work?)
I saw that wearing something very white and very black will not show up well.
So I'm wondering about best ways to get the best result.
I was thinking of wearing Grey spandex for upperbody and legs (defines knee joints, shoulder, elbow, etc.) And then putting tracking stickers (small blue or pink circle stickers) on the: shoulders, elbows, hands, knees, feet, hip, chest, head.
I wanted to try that against a plain white wall. Then trying some basic punches, walks, jumps.
Surprisingly the animate 3D app can track my twisting and flips better than other apps can. However, it's a bit of hit or miss. Maybe 1/10 entries come out working great. And the ones that come out great, I'm not sure what about the video (angle, lighting, background color vs subject color) creates a better result.
I really want to try the grey spandex and color stickers. If there are any other new advice from Deep Motion's own testing, would love to know! And anyone out here who is trying to see the best results and done other experiments as well, please share your experiences :)
-
These are the in product guidelines for uploaded videos. We actually just added "Do not wear all white or all black clothing " that corresponds to your findings exactly ! It's not necessary to go all the way to attach labels to your physical joints, they will not help directly. However making your feet and hands and face standing out in the picture in general will help. Cloths with some pattern or textures help too. For style of clothing in general tight cloths or shorts with your feet and hands exposed work well. Also try to minimize self occlusion from the camera angle you shoot the video: e.g. a side view camera shoot from your right side may have your entire left side occluded by your torso. For such situations a 45 degree camera angle may do better than a straight on side view. For most casual dance kind of motion a front view shoot works well too. Clean background without a lot of random objects cluttering the view will help. You don't need a green screen or white wall, however a good contrast in color/pattern between the background and your cloth will help.
Camera: Should be stationary and parallel to your subjectCharacter Placement: The entire body from head to toe should be visible, located 2-6 meters (6-20 feet) from the camera
Lighting: Neutral lighting with high contrast between the subject and the background is recommended
Size: Max video length allowed is 60 seconds and max size is 50.00 MB
Occlusion: The subject should not be occluded by any objects and there should be a single subject in the motion clip
Clothing: Do not wear all white or all black clothing or loose clothing that covers key joints like knees and elbows
-
Twisting and flips are among the challenging motions that we still need improve the AI for. The current version is tuned more for landed motion. Aerobatic motion improvement is on our TODO list though. If you have to give it try currently, shooting it at 60-120FPS with at least HD resolution from a camera on the tripod may help.
-
Thank you!!!
Yes, the black and white clothing thing was from your guidelines on the main Animate 3D. Usually I would wear tighter black spandex, but as I've done some uploads with Animate 3D, I can see that pure black and pure white are confusing for the app.
Yeah 60-120 FPS makes sense! I had a few videos where the camera was parallel and with tripod. What seemed like a good video to use, I realize I was wearing black pants and it confuses the motion. These are older videos I've been using to upload, but I'm very set on doing a shoot where I use grey spandex and use 60fps. I think with 60 fps there's less Motion Blur and it's easier to trace the legs and arms that swing.
Wow, this is interesting, not using stickers for physical joints. But it makes sense. For example, if I use green stickers on hands and feet, it would be confusing to the app what's an arm or leg. This is what I'm guessing, not sure how it all works. It reminds me of when I tried photogrammetry to create a 3D model: a busy, unique pattern helped the software get a better model because it's easier to locate exactly what each distinct feature is.
I was also thinking to help the app process profile movement (ex: walking from side view,) start every video facing directly towards the camera with T pose to help the program figure out the basics and then turn side ways to test profile/ side view movement.
You mentioned textures or having feet or hands exposed helps. I'm now thinking about using the grey spandex, but pulling the spandex up so knees, shin, feet are exposed. And pulling up spandex sleeves so elbow, forearm, and hands are exposed. It seems one of my better mocap results was where I was wearing shorts or had grey pants with lines on them. These were all older videos from a year or so ago, not made with the intention of using for mocap. However, I am very surprised and happy with the results deep motion gives for complex moves. I still think it's the best and you don't have to buy a suit. I can also see, in the future, the algorithm / app becoming more and more optimized. It's really cool and fascinating.
Twisting and flips are for sure very challenging. Yet I am surprised how well Deep Motion works! I had some good results with some videos (most videos however were very hard to get good results.) I had a few videos that came out really well and I will post them. What surprised me is that the type of flip twisting movement I was doing, it was rotating on Z, Y, X axis. Spinning like a pirouette is hard for an AI app to process I'm sure. And regular backflip with head over heels movement, is probably hard to process too. But the type of movement I was doing that combines twist and flip together (X,Y,Z axis rotation) would probably be the most confusing. However, I was lucky and had a few videos with great results! I'll make sure to post them in Share Your Animate 3D thread. Very interesting and powerful app. What types of advancements or tests are being done to further the flip / twisting accuracy with the App?
-
VoiceOver Mocap Thank you for sharing with us the detailed feedback. I'm sure the feedback on video best practice help us and other users to improve the quality of the AI animation results. In terms of what types of advancements or tests can help flip / twisting accuracy, we do need more training data of "twists & flips" to teach the AI to handle these movements better. It's harder to find performers for these challenging moves than the normal daily walking & dancing moves.
-