This week we started a 5-week group project. We don’t even really have a brief yet, but we were told to form groups and prepare a presentation for Friday (tomorrow) indicating what we’ll be doing for out group project (so one week pre-production, 4 weeks production/post-production). In this post I’ll outline the work that has gone into the pre-production of the project so far this week, and I’ll make additional posts in the future as we progress.
Lockie said he’d like to try a 360 video and had already done some tech tests (and proved it to be very feasible) and that’s something I’ve wanted to try out for a while now, so we banded together and brainstormed various potential subject matters.
We ultimately agreed on making a futuristic (sorta cyberpunk) prison. It’ll be visually interesting whilst being very easy to create in a limited timeframe. If you think about it any given prison block is made of basically identical cells, and most of the detail will be in textures as there won’t be many fiddly small items to sell the space as believable. We’ll only have to make a few assets and duplicate them around, then make a few texture variants for variation. We’ll probably add some asymmetry in the form of pipes or something to break it up a bit too.
We spent a good amount of time looking for good references and inspiration images (some of which I’ll include below) and found one image in particular which showed a tram overhead. We figured that’d be a pretty interesting way to show off the space, and gives us more freedom to create more than just the one section. We also recruited two more team members, one of whom is Michael (with whom I worked on the dinosaur rig). The other (Tamaki) is very interested in doing some roto work, so we decided we’ll look into integrating live-action footage into a 3D rendered 360 video.
We narrowed down the video to a 30-45 second video spanning 3 sections. I really liked the standard sorta prison layout (see ref 5 and 6 below) and Lockie is a fan of a rotary prison (see ref 3 and 4), so we figured using a rotary prison as an elevator and general population area, and then moving into a high-security prison section would be effective. We weren’t too sure how to start it off at this stage but everyone agreed that some sort of opening section was required (perhaps a station, or a security lobby). After that, a lot of the day was spend looking up more references – both for specific elements we liked as well as the overall mood for each section.
We found that Foundry has made a Nuke plugin called Cara VR which Lockie did some research on. I decided to jump into unity and do some tests to see what kind of fidelity we could get on an asset we already had, as well as looking into the overall workflow. I discovered very quickly that Unity doesn’t natively support UDIMs, and that the standard shader handles spec maps differently than we’re used to (it takes spec color from the RGB channels of the spec map, and uses the alpha of the spec or albino map’s alpha channel for the actual specular value). The result of this test follows. We’re pretty happy that this is a good level of fidelity, and what little quality might be lost compared with a full render in Vray will be more than made up for by being able to render in realtime. Heck, we might even be able to add some interactivity into it, depending on how we decide to ultimately distribute the experience.
We also decided that the live action portion could be a hologram, acting as a sort of guide for the viewer. Obviously we’d have to find a way to integrate that in Unity, which I started researching. More on that for Wednesday below.
Michael also started shoving together a mockup of how the overall viewing experience might look (modelling placeholders for each section and of the tram out of primitives).
Finally, we started thinking about viewer’s role in the experience. The main ideas we came up with were: prisoner, inspector, museum-goer. Ideally recording the footage for this and getting it set up for use in Unity will be done very early on next week.
Michael finished up his mockup (I’ll link to it as soon as I can; Lockie is currently slapping the meta data on it so we can upload it to Youtube as a proper 360 video) and it looks pretty great! I looked up some more information on improving the look of a scene in Unity and found a ton of settings that really helped out, as well as some nice post effects. I also found a lot of resources and different methods for importing a video with alpha for transparency. Unfortunately the codecs that Unity supports with an Alpha channel are not codecs that we have at school, so doing it natively isn’t an option for us. I didn’t want to use an animated texture either as the image sequence would probably take up more space than a well encoded video. Ultimately I found a bunch of people talking about creating the alpha as a part of the video itself – you make a video that’s twice as wide (or high) than the original source and you add the alpha channel right next to (or below) the source footage. Then you use a custom shader that grabs the information from >50% X or Y and uses that as an alpha. I couldn’t find a shader on the Unity asset store that does this, but I ran out of time to look into it further. Good news is it’s at least possible to do.
Lockie did some more research into Cara VR and it sounds like that’s also a way we can go if Unity falls through (or if we choose to render it after the assignment anyway for the added fidelity). Tamaki was jamming some 360 concepts in photoshop.
We also narrowed down a few details, of note deciding that the first section will be a station area, and the viewer’s role will be that of an inspector (we chose not to be specific about what kind of inspector or what they’re inspecting for exactly so that the audience can project a bit onto their “character”). The hologram will be talking about the prison in a generic and informative way, but won’t be telling them what they’re there for beyond greeting them as “Inspector.”
We talked more specifically about the shapes of some of the sections and decided based on our inspiration images what direction artistically we’d take them in. We really liked one image of a surface-level train station (previously we’d been thinking the entire complex would be underground) and so we’ll be doing something very reminiscent of ref 1 and 2 below. Based on this we’re intending to have the lighting grow dimmer (and grimmer) and the colours less saturated the lower down you go, which should really drive the hopelessness of prison home.
Based on that discussion Lockie made a more specific mockup the complex, but as it’s modelled in such a way that is hard to really visualise without manipulating 3D space (and hasn’t at this stage been prepared as a video) I won’t be uploading images of it here.
Finally, I found an awesome resource that actually includes the shader code!
I wasn’t able to get their finalised shader to work (I got an “unexpected token: if” error that I wasn’t able to rectify quickly), but an earlier version worked well enough for a proof of concept! My test is here below. Nothing exciting visually, but the words there are actually a video with transparency as described in Wednesday’s writeup. Based on that test we should be able to get a video of a person (edited somewhat to look hologram-y) into a unity scene without any issues at all.
Tomorrow’s the approval presentation, so wish us luck!
Reference/inspiration images that I think are exemplary of what we’re going for: