Dream Syntax: 10 years on

I've been working on a Blender/Unity/VR remake of the dream maps in my book Dream Syntax. Because I had changed computers so many times since then (and have not installed any fonts on the new laptop), this is what my original illustrator working file for Dream Syntax looked like when I opened it:



Pictured is the map for a dream I had 10 years ago, in 2008.

How to set up SteamVR with the HTC Vive

ITS THE NEW YEAR!

These last few weeks Meshminds gave me the opportunity to work with the HTC Vive and I've decided to build a small VR experience using Unreal Engine. I had a "VR Ready" laptop (Gigabyte Aero 15) but yet I must confess that the setup for the Vive is not as easy or straightforward as it seems. Here are some notes on the process...

Setting up the Play Area

Recommended setup
First, you'll need to clear the play area. In our case, having just moved back to Singapore, our boxes haven't arrived from London just yet, so we simply moved all the furniture out of the way. Technically speaking, all you need to clear is 60m2 (3x4x5) metres. But let's be realistic: not everyone is going to have this luxury of floor space in their home or rental flat. And even if you think you've cleared everything (and even when you see the Chaperone boundaries) you'll probably still violently thwack something with the controllers or stub your toes on the furniture you stowed away in the corner at some point.



[Note that the average HDB flat living room floor area will exceed the volume required so take care that the second lighthouse shouldn't exceed a distance of 5m from the other lighthouse]

Vive Installation

The Vive came in a reasonably compact box, but once I took out the items from their neatly packed state, the cables somehow seemed to gain volume and a wilful desire to take over the entire room. After moving the equipment from one room to another (whilst trying to decide which was the room to use), I found myself spending time detangling cables over and over again.


Cable explosion after I took it out. Even with the diagram its difficult to put it all back in after you uncoil everything...
If you are renting, then you probably don't want to do anything so drastic as to drill a permanent mount into the wall. After cracking our brains on what to do, we found two cheap $5 selfie sticks with 1/4" camera screw mounts that also fit the Vive Lighthouses's screw base. We took the selfie stick apart and mounted the lighthouse on the flat stick which gave us an adjustable mobile rig that we could stick to the wall with a combination of 3M Command tabs and duct tape. (Note that the base station generates some vibration so you definitely want to pile on the duct tape for safety)






DIY Vive Lighthouse Mounting - NOT PRETTY BUT IT WORKS

Troubleshooting

After spending countless boring hours installing new drivers and updates for Windows and the graphics card, we were able to run SteamVR intermittently. One recurring issue was relating to the VR Compositor. Every time we restarted the entire system, on the first run it always returns "Shared IPC Compositor Connect Failed (306)" and "Compositor is not available (400)" even when everything is already set to use High Performance graphics (set to default to the better GPU in the laptop). It seems to be a common issue - if you search online you will find several forums where many lost people are also asking the same question "HOW DOES I COMPOSITOR???"





Apparently this error is caused either by too many display devices (monitors, etc) connected or the Intel graphics card somehow taking precedence over the Nvidia GTX1060 in this laptop. The fix for it is to go to SteamVR Settings and Disable Direct Mode and then Re-enable Direct Mode. SteamVR will restart each time you do that. Sometimes this works, SOMETIMES THIS DOESN'T. Okay, most of the time it will fix it.



At this point it seems worth asking, so what is the VR Compositor? If you read Valve's documentation, it says:

The Compositor simplifies the process of displaying images to the user by taking care of distortion, prediction, synchronization and other subtle issues that can be a challenge to get operating properly for a solid VR experience.

If I understand it right, the compositor runs the VR display, and when an application wants to use the HMD (Head Mounted Display) it asks the compositor for access to a buffer and begins to render into that buffer. The eye buffers are a bit like layers that are ontop of one another. Vive’s Chaperone is an example of one such layer that already asked the compositor for access at the very start - so you see the room boundary in the HMD when you walk too close to the edge.



The standard or normal rendering of the 3D world is rendered into each of the eye buffers (one for each eye!) and then the warp pass/spatial as well as chromatic distortion is executed over the eye buffers to allow the final image to match the curved lenses on the HMD so that you see it as a 3D image with the headset.

[Another issue we had was hardware related - one of the lighthouses seemed faulty - one LED does not light up (instead of 17 only 16 LEDs are lit). In the meantime, we simply used the working Lighthouse as the front facing one and tried not to turn 180 degrees in a hurry so it wouldn't grey out. Apparently it still mostly works with one Lighthouse so long as if you are still kinda facing the single Lighthouse]

ANOTHER FUN READ: Teardown of the HTC Vive

To take a screenshot in SteamVR, press System+Right Trigger at the same time. The image in SteamVR is quite confusing as the prompt on screen is not very clear - I read it as a suggestion that you press System and THEN press Right Trigger. So I thought the screenshot feature was broken until George told me to press BOTH AT THE SAME TIME. As for where these screenshots are being saved, its really not very obvious either. For example, you can't set the folder for it to be saved. It took me a while to figure this out, but the screenshots are being saved into the userdata folder in Steam. For me, by default the screenshots were being saved here:

C:\Program Files (x86)\Steam\userdata\760\remote\250820\screenshots\

Besides the 250820 folder, there were other numbered folders for different apps or sessions. Look in all of the folders inside remote to find your screenshots.

Also note that the screenshots are PRE-warp/distortion. So you'll get screenshots that all look like two normal views without the overlay of the other buffer layers such as the Vive Chaperone, etc.



Now for the fun... testing out various VR apps!

---

The Lab

On Steam: http://store.steampowered.com/app/450390/The_Lab/



First stop is surely The Lab. Pet your robotic dog on a mountain side and play with other very solid VR experiments in The Lab. This free title is deservedly highly rated - you'll definitely find that there are quite a number of VR games appearing on the market which are simply clones / rip offs of the devices from this portal-themed collection.

VR Museum of Fine Art

On Steam: http://store.steampowered.com/app/515020/The_VR_Museum_of_Fine_Art/

Art? What art? I came here specially to look at the potted plants used to decorate the dusty corners of the museum. 10/10.

Google Earth VR

Google Earth VR is super impressive in 3D cities like London where you can fly around in a warped clone universe...



and with a wave your hand you can instantly change the time of day into THE TIME OF THE APOCALYPSE





There is also another frightening feature...



AIN'T NOBODY WANTS TO FIND THEMSELVES FLOATING OVER EARTH AT THIS ANGLE!!! URGHHH. Which brings us to the issue of sim sickness. Its not great when the motion in the graphics suddenly does not correspond with the user's own head movements...

Job Simulator

In Job Simulator, you play a human being assigned to do some artisanal jobbing for robots who have developed a refined taste for their services and products being made sloppily and badly by humans in a world of limitless machine perfection. I liked this game too much, because I'm not very good with following instructions in an open roaming world. I haven't tried all the jobs yet, but here I was being a chef - burning lemons on the grill, drinking virtual wine on the job, and throwing raw meat at the frowning robot customers.

SUPERHOT

Superhot works on the premise that time only moves when you move. The graphics are simple but the rapidity with which things move when you accidentally make a wiggle of your head in VR is very effectively. It also however leads to a lot of unintentional arm or hand flapping when you somehow need to make things go faster, so it was quite amusing to watch George playing this.

Blender & Unity: Manually Rigging Blender Humanoid Characters for use with Unity Mecanim

I'm definitely no character animator by trade, but there comes a time when you end up with a Unity project that somehow requires it. There are obviously many automatic rigging methods available (Blender does actually have an auto-Rigging system called Rigify for biped humanoids) and you could even try to download other rigs made by other people and plonk them into your scene, but I found that so many of the rigs including the rigify one seems to involve so many complicated bones you don't need, so you end up having to sift through the bones, deleting so many unwanted bones, renaming bones, perhaps even having the impression of the impossibility of rigging up them bones.

Although it may seem terrifying at the beginning (I'm not an animator or rigging specialist!), I found that surprisingly, it is not that difficult to manually rig up all your bones if what you have is a very simple humanoid character. You just need to be orderly and to stick with the admittedly tedious bone naming process. (Although our character is blobby, we're sticking with a humanoid as we're going to use it with the Kinect to sync it with the movement of the human user, and our human user is going to return a humanoid set of values that we'll need to rig up our character to...)



According to the Unity Blog's post on Mecanim Humanoid:

"The skeleton rig must respect a standard hierarchy to be compatible with our Humanoid Rig. The skeleton may have any number of in-between bones between humanoid bones, but it must respect the following pattern:"
Hips – Upper Leg – Lower Leg – Foot – Toes
Hips – Spine – Chest – Neck – Head
Chest – Shoulder – Arm – Forearm – Hand
Hand – Proximal – Intermediate – Distal



This is the list of all the bones you need (I found it useful to copy and paste in these names directly)

head
neck
collarbone.L
collarbone.R
upperArm.L
upperArm.R
lowerArm.L
lowerArm.R
hand.L
hand.R
chest
abdomen
hips
upperLeg.L
upperLeg.R
lowerLeg.L
lowerLeg.R
foot.L
foot.R
toes.L
toes.R

Optional: eye.L and eye.R

For starters: Ensure that your character model is positioned at origin and that its pivot point is also at origin (0,0,0). Make sure you reset the scale to 1 just in case (Ctrl+A, Select Scale). The hip bone is the key bone in all this, so start by creating one big bone starting from the bottom of hip to top of the chest. Hit Space and start typing "Subdivide Multi" (Armature) and give it 2 cuts so you get 3 bones. These will form the hips, abdomen and chest bone.



After you've done the main spine bones, you can turn on x-axis mirror.



- Select the ball on top of the bottom bone (hips bone). Make sure Options>Armature option>X-Axis Mirror is selected, then press Shift-E to extrude mirrored bones. When you're in mirror mode, every time you create a new bone, you'll have a second one mirrored on the other side of the X-Axis. Remember that you'll have to rename BOTH bones later on - if you are facing your model face-on, also remember that L is actually to the right and R is to the left, and name it accordingly.

- Arrange the leg bone into position (you may need uncheck "Connected" in order to let the leg bone go into the right position). Reposition the leg bones away from the hip. Subdivide Multi (1 cut) this leg bone into two bones, forming upperLeg and lowerLeg.

- Shift-E to extrude two more foot and toe bones, and also add in the collarbone, arms and neck+head bone. Do make sure you keep it all in a standing T-pose (as if the character is standing in the shape of the letter t).

- Ensure that all of your bones are renamed correctly as per the list. If there is an L bone there must always be a R bone.

- Go into Object Mode and Select first the character and then Shift select the armature. Press Ctrl+P and select Set Parent To - Armature Deform - With automatic weights. Your computer might lag for a second before its all connected up.



From there, you're in the home stretch. Export your Blender model in FBX format and then import it into Unity, and in Unity set the rig to humanoid (instead of generic) and at the bottom of that, hit Apply.

Let the wild rigging begin!



See also:
Animate Anything with Mecanim

From Compassvale to Coney Island: Casuarina Pines, Chicken of the Woods, and CCTV

Here is a visual documentation of a weekend walk we did to Coney Island. Starting from the... er... 'charmingly named' Compassvale, we made a beeline to Punggol Jetty and then crossed over to Coney Island by foot. On first glance it looks like a popular haunt for groups of school children and families going for a picnic, but if you step off the path and to the beach, you'll find the crowds thinning down... to practically nothing. The island is quite sizeable and most visitors seem to cycle on the paths rather than to stumble over driftwood and sand. A rather beatifically peaceful tropical park with strangely neat and new paved paths, huge airy casuarina pine trees, tall grasses, marred only by the brightly coloured plastic pollution from the oceans. At the boundaries of the island, CCTV cameras and other metallic monitoring devices stick out incongruously between sea apple trees and coastal shrubbery, silently watching the waters between Singapore and Malaysia at all times.































































Shelf Mushroom aka Chicken of the Woods!