Building augmented reality experiences in Hawaii.
In 2019 I got a call from Steve and he wanted to create few immersive experiences and validate a customer need on visual augmented storytelling. So, we set to Hawaii to build and get reactions.
The task was to test the viability of augmented reality experiences in Kuala Ranch Hawaii. Premise was that people go there to take pictures at scenic Jurassic Park filming locations with dinosaur toys. We wanted to know if there's an feasibility to up a notch the experience with augmented reality and immerse people to the scenes by bringing dinosaurs alive in ar.
For starters I knew that in order to create compelling scenes we needed to scan the real life locations and use point clouds from the scans to create a mesh and to use it as a base anchor position for all digital assets and arrange their paths so that they are contextualized to the environment realistically. Meshing the same point clouds gave us occlusion as well. (This was prior to iPad/iPhone lidar).
The setup I used was Immersals SDK for point cloud and localization, meshlab to turn the point cloud to a mesh. Then Unity to tie everything together to build and script the scene. The assets were also bought from the Unity Asset Store.
There was a lot of trial and error making the scene to align well with the minimal references we got out of the point clouds. Nevertheless we made it work. Having a full mesh will help especially with the new lidar to build and create super contextualized scenes minimizing the need to try out the experience at the location and see if it works.
For the Unity stuff I used bezier curves to add paths for the Gallimimus from the back, middle and front of the scanned log for another level of scene depth. There is also a trigger collider in the log that initializes the animation sequences of the dino's jumping as well as the T-Rex coming out of the trees. Part of the animations are predone and some are procedural. The T-Rex for instance focuses on the human on the scene getting the anchor from human body tracking that comes with ARKit. Unfortunately you cannot see it that well in the scene but these things, these minor details will help make immersive ar experiences come alive.
In the second scene I am using image targets to arrange the Pteranodon on the sign and also to occlude the other dino’s behind it. I also used the human occlusion so the visiting star can be seen in front of the dino's. This scene uses horizontal plane detection and the image target to align itself. It does not use pre saved point clouds to localize. This also shows you that you can get creative while making contextualized ar scenes. Sometimes an image anchor is enough or a vertical, or horizontal plane. All really depends on the environmental context you are making the ar scene at.
On the picture above, the green is the imaget target used as ground truth for the landing of Pteranodon and occlusion for the Brontosaurus that will been on the video below. The yellowish floor is the horizontal plane used for the right elevation for the Brontosaurus dino’s as well.
Above you can see Pteranodon landing on top of the sign excactly at the debug position seen in the previous picture. The dino also keeps looking at the user with the phone to create more immersion.Â
As these prototypes were designed to have consistency. This scene as well has two buttons that enables basic interaction with the scene. Replay and Action. The idea behind this was you could record and then share the immersive experience to others. While act out a "part" in the immersion.
Tbh, I got really excited while building this out. For the possibility building a augmented theme park at a park is just mindblowing. Obviously with handheld devices there are caveats and one of them is that looking through a screen is less exciting than being hands free with glasses that can cover your whole fov, let's hope those drop out soonish.
Nevertheless the reactions we got for demoing the experience were priceless, it delighted and brought wonder to the visiting people at the scene and they asked to have the recordings as souveniers.
Bonus!
We also did Pearl Harbour as an immersive experience. The process to set up anchoring to the real world follows the process similar of the first log scene.
For Pearl Harbour the validation point was the same. A lot of people go there to listen and learn of the historical events that took place in WW2 and we wanted to see how it would resonate to let people witness the historical event in ar.