Ten years ago I made the best spontaneous decision of my life, but let’s start at the end…
My wife and I and our two cats are moving back to Germany. No need to worry though, because I’ll continue to make video games with the talented and nice folks at Double Fine.
When I first moved to San Francisco in order to join LucasArts I thought I would only stay for a year or two before going back to Europe, but here I am many years later. In hindsight it’s no surprise though, because the Bay Area is an amazing place to work and live and the best part about moving to San Francisco is that I was able to meet amazing people and that I’m fortunate enough to call many of them my friends today.
Never would I have dreamed to work with Tim Schafer on a brand new adventure game or that I would be able to help to resurrect my favorite game of all time: Monkey Island. I’m certainly very grateful for these opportunities! Looking back at it now it’s hard to believe that none of this might have happened since I originally didn’t even plan to move abroad.
While I was finishing my studies in Dresden (Germany) I started to look for jobs as a programmer in games or movies (e.g. visual effects). The job market at that time was very limited and it was tough finding a company that was willing to hire somebody straight out of school especially within Germany. I started expanding the search radius and eventually I found a small company in Scotland that offered me a position.
I made the spontaneous decision to give it a try, packed a bag (yep just one), booked a flight to Scotland and reserved a room in a hostel. I still remember sitting in the train to the airport thinking to myself “This is crazy! Why am I doing this? What if this doesn’t work out?” One of the most important lessons I learned since then is that it doesn’t really matter whether or not a plan works, since there is always a way forward. Being able to experience life somewhere outside of my comfort zone was definitely worth the risk and I wouldn’t want to miss a single second of it.
I didn’t end up staying in Scotland for too long though and after working in London for a while I decided to move to San Francisco by accepting a job offer at LucasArts where I joined a group called Team 3 (we didn’t work on Star Wars or Indiana Jones and hence we were the third team) to work on a really cool sci-fi game. Unfortunately management was very unstable at the time and our project was cancelled just months after I started to work at LucasArts.
Team 3 at LucasArts – Bonus points if you can spot the programmers.
Our team (see photo above) decided to use this time of uncertainty in a productive way by digging through the archives. We found the source code and some assets for the classic point & click adventure game Monkey Island and started porting it to modern hardware; mostly because we had nothing better to do. I’m still super proud of what we achieved and I have to say it was fantastic to be able to work on the favorite game of my childhood.
The reoccurring layoffs at LucasArts became very exhausting after a while and I realized that it was time to move on. A friend of mine who worked at Double Fine invited me to check out the studio and I immediately fell in love with the creativity and the general vibe in the office and so I came back for an interview. The rest is history…
My time at Double Fine was wonderful. I mean just look at that awesome bunch! Everyone is incredibly talented, creative and passionate about what they do and I’m very proud of all the games I was able to contribute to. Now that Broken Age is done though it’s time for me to go home and while I’m sad that I won’t be in the office anymore I’m happy that I’ll be able to continue to work with these amazing people in the future.
Coming back to Germany after being away for a decade will be strange for me and I think I’ll probably feel like a foreigner for a while, but I’m definitely looking forward to be closer to my family and friends who I haven’t seen that much in the last couple of years.
In a way I feel like Bilbo Baggins returning to the Shire at the end of the Hobbit… only with a laptop and the internet… Alright maybe this analogy doesn’t work all that well, but I’m excited about the future nonetheless. Let’s make some awesome games you guys!
Now that we wrapped up this year’s Amnesia Fortnight (which is the name of Double Fine’s public game jam) and released the prototypes I wanted to share my thoughts on adding Oculus Rift support to ‘Mnemonic’. Some of you might know that I’m personally very excited about the current developments in VR which is why I pitched a sci-fi exploration game called ‘Derelict’ that would have been developed for the Oculus Rift from day one.
Even though it didn’t get picked in the end I was still lucky enough to work on VR for Derek Brand’s film noir inspired exploration / adventure game ‘Mnemonic’ and I think it turned out great! In this blog posts I will shed some light on some of the technical and non-technical aspects of adding support for the Oculus Rift to the prototype.
Obviously I didn’t do all the work myself and the VR version of ‘Mnemonic’ would not have been possible without major contributions from Brandon Dillon (@Noughtceratops) and Matt Enright (@ColdEquation)!
The following quick presentation introduces the topic in five short minutes:
Game design
In ‘Mnemonic’ you have to discover your past by entering the surreal world of your memories. As you explore different events you can restore more memories by solving adventure-game style puzzles which will eventually lead you to a dark secret. I think the team definitely nailed Derek’s vision of a film noir art style. The game is rendered (almost) entirely in black and white and looks stunning and that the prototype was created in only 2 weeks still blows my mind.
The design of ‘Mnemonic’ is a great fit for the Oculus Rift since the game is slow paced and doesn’t require fast or unnatural player movement (e.g. strafing, jumps). The core mechanic is to look at and interact with interesting things, which works very well even with current generation of VR headsets. The fact that you are exploring memories also helps the VR experience, because the brain does not expect 100% realistic behavior from the game world (e.g. ‘why can’t I push that barrel’).
One thing I wish we could have done is to give the main character a (virtual) body since it would have helped to increase the feeling of immersion by creating a stronger connection between the player and the virtual alter ego. In one of the memories you are sitting in a car and it really feels weird to see the seat instead of a body when looking down.
I think the finished result proves the point that you have think about VR from the beginning in order to create a great experience. Porting the design of a game to VR after it is done will never be as successful as incorporating it from the start.
Mnemonic in VR mode
Adding VR support to our game engine
I originally integrated Oculus Rift support to ‘Autonomous’ a while ago in my spare time, because I wanted to know how complicated it would be to add such a feature to a preexisting and relatively complicated code base.
‘Autonomous’ is a first-person game that lets you build and program autonomous robots in a cool 80s inspired sci-fi world. The game was Lee Petty’s pitch for Amnesia Fortnight in 2012 and I contributed to the prototype as a graphics programmer. If you have a Leap controller you can check out the game here: http://autonomousgame.com/
Autonomous
In order to add Oculus Rift support to our proprietary ‘Buddha’ game engine I had to first solve the problem of rendering the current scene for each eye. Since ‘Buddha’ uses immediate mode rendering I was able to draw the scene twice by simply duplicating the frame data and offsetting the camera for each eye.
I ran into an interesting problem with our implementation of directional sun-light shadows though. Since ‘Buddha’ was originally developed for ‘Brutal Legend’ it uses cascaded shadow maps in order to provide high quality shadows at varying distances. The cascades are computed by splitting the view frustum into multiple slices, but since the frustum for each eye were slightly different the resulting shadow maps caused disparity between the left and right eye.
This may seem like a minor problem, but the differences were big enough to create discomfort when playing the game. It took me a while to figure out what was going on, but in the end I was able to identify the discrepancy by performing the ‘left-eye-right-eye-test’: Close the right eye and look at the scene, then open the right eye and close the left eye and compare the rendered results. Any visual difference that is not directly connected to the camera offset is a VR bug!
My solution for this problem was to perform all shadow calculations in the space centered between both eyes. Due to complications, which are beyond the scope of this blog post, it wasn’t possible to cache the shadow maps for the second frame, so there is definitely some room for future improvements.
The rest of the integration was relatively straight forward and in the end ‘Autonomous’ looked like this in VR mode:
Autonomous in VR mode
Integrating VR into Mnemonic
Adding to support for the Oculus Rift to ‘Mnemonic’ was straight forward using the VR integration mentioned above (especially after Brandon cleaned up my experimental code). We simply retrieve the orientation from the HMD and apply it to the FPS camera. The user is also able to rotate the camera left and right with the mouse (or game-pad), in order to make it possible to reorient the main character.
Initially there were plans to add cutscenes to the prototype that would take away camera control from the player, but we managed to convince the team that this would make people feel sick and therefore break the VR experience. In the final prototype the player has full control over the orientation of the camera at all times and I think it definitely helps to prevent nauseousness (aka VR sickness).
Here is a picture of Tim playing the ‘Mnemonic’ prototype in VR mode using the Oculus Rift:
User interface design
Traditional 2D UI doesn’t work well in VR, which is a lesson I had previously learned when adding Oculus Rift support to ‘Autonomous’. The main problem is that you can’t simply blit the UI on top of the scene. Drawing a 2D element at the same screen-space location for both eyes essentially means that the UI is infinitely far away. However since it is supposed to be on top of everything else the brain can’t really make sense of what it sees.
For ‘Autonomous’ I used the solution described in Joe Ludwig’s excellent paper about adding VR support to ‘Team Fortress 2’. The idea is to draw the UI as a camera-attached plane which ‘floats’ in front of the camera. Since the plane has a real distance to the camera each eye will see it at a slightly different screen-space location and the brain will therefore interpret it correctly. Readability of the UI is still problem, but that is a story for another day…
For ‘Mnemonic’ we decided to avoid 2D UI entirely. Fortunately the game doesn’t require menus and we only had to find a solution for the inventory. Items carried by the player are represented by real 3D models that are located on a ring around the camera. This way the inventory items are rendered as part of the regular scene and show up at the correct location in VR. This approach works very well and I would like to explore it further in a future project.
3D inventory in Mnemonic
Post effects and other screen-space problems
Image post-processing of the rendered scene is a pretty standard (and useful) technique in games these days. Typical post effects include color correction, edge darkening (aka vignetting), anti-aliasing, blooming or depth-of-field blurring. These operations are usually applied in screen-space which makes them problematic for the same reason that 2D UI doesn’t work well in VR.
Unless the operations are spatially independent (e.g. color correction) it is important to take the interpupillary distance into account when rendering the effect. In ‘Mnemonic’ we offset the texture coordinates of extra textures used during the image post-processing step.
In the prototype the player is able to return to the ‘memory hub’ at any point by pressing a button and the transition is represented by an animated Rorschach image. By applying a horizontal offset to the Rorschach texture the effect essentially gets rendered at a virtual depth (much closer than infinity), which is important since the effect is faded in (and out) on top of the scene.
While this works alright in the prototype a better approach will be necessary for a full game. Drawing these kinds of effects in 3D space (just like the inventory) seems to be the only real solution for this problem. I’d love to experiment with camera-attached particles or similar techniques.
Of course there are effects that can’t be represented by 3D geometry (e.g. vignette) and more research will be necessary to figure out how to do implement them in VR.
Conclusion
I think the potential of VR to create an immersive experience for the player is very exciting, but creating an excellent VR experience isn’t trivial. In his talk at the Steam Dev Days Palmer Luckey (the CEO of Oculus) said that you really have to design a game with VR in mind and I very much agree with him. Adding VR support later on is very difficult and will require quite a few changes.
The games industry is only at the beginning of figuring out how to effectively use VR and I think next few years will be very interesting and exciting. I’m very grateful that I was able to experiment with VR during this year’s Amnesia Fortnight and I really hope that I can come back to it and work on a full game.
I’m very proud of what we achieved with ‘Mnemonic’ and I hope you will check out the prototype (especially if you own an Oculus Rift). You can still get access to all of the games on Humble Bundle.
Post scriptum
If you made it this far and you are still not tired of my ramblings, then you might also want to check out the excellent documentary about Amnesia Fortnight made by 2 Player Productions. You can find the entire playlist on YouTube.
I’m talking a bit about adding VR support to ‘Mnemonic’ in the episodes about day 9 (starting at 17:18) and 10 (starting at 37:44):
It is certainly no secret that lighting can tremendously improve the visual quality of a game. In this blog post I’ll describe various techniques for dynamic 2D character lighting that are very easy to implement and don’t require any additional assets (e.g. normal maps). I have successfully used these techniques in various games like Monkey Island: Special Edition, Lucidity and most recently in Broken Age.
But why is character lighting important in 2D games? Very often artists will paint lighting (and shadowing) directly into the environment artwork, which is okay because the world is generally static. In fact it would be very difficult to compute the lighting for, let’s say, a background image in real-time, since no spatial information (e.g. z-depth, normal direction) is available. Characters on the other hand move through the world and will be visible in various locations with different lighting conditions, so it is generally not possible to paint the influence of light sources directly into the sprites (or textures). Therefore the only solution is to calculate the character lighting during run-time.
The video below demonstrates how Shay (who is one of the main characters in Broken Age) looks with and without lighting. I hope you’ll agree that lighting helps to integrate Shay into (the beautiful) world.
Ambient Lighting
The idea behind ambient lighting is to make a character look like they are part of the game world rather than floating on top of it. So if, for example, a player moves from a shadowed into a fully lit area the visual appearance of the character should change from dark to bright.
Thankfully this can be achieved very easily by tinting the sprite (or texture). Using a gradient tint allows independent control of the color of the lower and upper part of the body, which makes it possible to emulate ambient occlusion between the floor and the character.
Ambient gradient tinting is the backbone of Broken Age’s lighting system and is used in every scene for every character. The following picture demonstrates how the ambient gradient changes based on the location of Shay.
There are different ways to interpolate the gradient color across the body of a character. The simplest approach is to use the bounding box in which case vertices on the head sample from top of the gradient whereas the feet use the bottom of the gradient. This approach works great for sprite-based characters, but doesn’t offer much control over the shape of the gradient.
Broken Age applies a different strategy by computing normals as the vectors from the translated center of the bounding box to each vertex. The ambient gradient is then sampled based on the y-coordinate of the resulting normal. Not only does this technique work great with skinned geometry it also makes it possible to define how quickly the gradient interpolates across the body by changing the offset of the center. My GDC Europe talk ‘Broken Age’s Approach to Scalability’ contains more information about this approach, so please feel free to check it out if you want to know more about it.
Local Lighting
The goal of local lighting is to show the influence of nearby light sources. In other words it answers the questions where the characters are in relation to lights. The video of Monkey Island: Special Edition (MISE) below shows how the local lighting of the campfire illuminates the two characters. Please make sure to watch it fullscreen and in high definition.
If you look closely you’ll notice that the campfire only illuminates the parts of the characters that are facing towards it. There are different ways to achieve this look. The characters in Broken Age have hand authored normal maps that are then used to compute the reflected light. While painted normals offer a lot of artistic freedom they obviously require a lot of time to create.
In MISE we weren’t able to go down this route due to time constraints of the development schedule, so we needed a technique that required no extra assets and only a minimal amount of tweaking. Our solution utilizes a screen-space light map that contains the illumination of nearby light sources. The following image shows the light map for the lookout scene.
In order to calculate the light map we first render the alpha mask of the characters into an offscreen render-target (see image A below) before downsampling and blurring it using a Gaussian kernel (image B). The resulting image can be interpreted as a soft height field containing a blobby approximation of the characters. With this in place we can now compute the normals of the blob approximation as the gradient of the height field (see image C). Now that normal information is available we can finally calculate the light map by accumulating the reflected illumination from all nearby light sources (image D). In MISE we used a very simple Labertian lighting model (aka ‘n dot l’), but more a sophisticated formula can easily be used instead.
The resulting lighting data can be combined with the scene in different ways. The easiest solution is to blit the light map on top of the framebuffer using additive blending. We used this approach in MISE and it worked great, but has the disadvantage that reflected light is barely visible on bright pixels (e.g. Guybrush’s shirt).
Another compositing strategy would be to sample the illumination from the light map in the character shader. This way the light can be integrated in different ways with the texels of the sprite (or texture), so characters could effectively have varying reflectivity. In this case the light map essentially represents the results of a light pre-pass.
Light Sources
One thing I haven’t talked about up until now is how light sources are authored. It should be easy to associate different parts of the world with specific parameters in order to be able to match the character lighting with the illumination painted into the environment.
Representing light sources as radial basis functions works great for ambient as well as local lighting. The gradient tint for each character can easily be computed as the weighted average based on the distance from the light sources. In my experience it is also a good idea to expose some kind of global light, which is sampled if there are no light sources nearby. The following image illustrates the radial basis function blending for ambient gradient tints.
The light map can be calculated by drawing the light sources as approximated shapes (e.g. quad or circle) into the render-target. The color and intensity of each output fragment is computed by evaluating the lighting model of your choice. In MISE we calculated the reflected light basically like this:
vec l = light_pos - fragment_world_pos
vec n = tex2D(normal_map, fragment_screen_pos)
scalar nDotL = saturate(dot(normalize(l), n))
scalar dist = length(l)
vec result = light_color * nDotL / (dist * dist)
Using radial basis functions has the additional benefit that it’s very easy to animate a light source both by changing its transformation (e.g. position, scale) as well as parameters (e.g. color, intensity). A flickering campfire can easily be achieved by interpolating between two states using a (nice) noise function. Attaching this light source to the hand of a character effectively makes it a cool looking torch.
Conclusion
Dynamic character lighting helps to make a game world and its inhabitants look more believable and interesting. Thankfully it’s relatively easy to implement a lighting system that produces nice results and I hope that my blog posts could provide you with some inspiration as well as practical advice. Please feel free to post questions as comments below.
That’s it for today. The last one to leave turn off the (dynamic) lights! 🙂
I agree with my colleague David Gardner that adventure games aren’t dead. They just seemed to be standing still though. It is true that there were many excellent adventure games (with a large production value) in the last couple of years, but few tried to move the genre forward.
And then there was Resonance! It’s very traditional and really progressive at the same time. So if you enjoy the ‘old school’ adventure look, an excellent near-future Sci-Fi story or if you simply would like to experience the next evolutionary step in the adventure genre then you shouldn’t hesitate and get a copy of the game.
What impressed me most about Resonance is that developer xiigames succeeded to add something new and exciting to the proven adventure game formula. One of my favorite new features is the ‘short term memory’ which allows you to drag objects or characters into it, so that they can then be used as topics in conversations with other characters. It makes so much sense and feels very natural. In fact while playing the game I was wondering how I played other adventures without this feature.
In addition to moving the genre forward Resonance also has a fantastic story with multiple story twists. There are few games that surprise me in terms of plot development, but Resonance certainly managed to do just that.
I really have nothing bad to say about the game and I’m not alone. Rock Paper Shotgun, Kotaku and many other high profile game review sites praise the game.
It just occurred to me the other day (and I’m not sure why it took me so long to make that connection) that enjoy programming and adventure games because of similar reasons.
Adventure games are all about puzzles and the joy of solving them. Well so is programming. You are constantly confronted with questions like ‘Why does the system behave like that?’, ‘How can we get around these limitations and make feature X work?’
The other day I was trying to track down a weird issue were the parts of the screen would sometimes get corrupted. Initially I thought it had to do with how I was managing graphics memory, so I was poking around in that code but couldn’t find the problem. The issue turned out to be in a completely different system and I (eventually) identified the source by observation of the behavior of the affected code. I approached the problem just like a puzzle in an adventure game (I even tried to look up the solution on the internet) and felt quite good after I finally solved the problem.
But there are more similarities than solving puzzles. The taxonomy of adventures in my simple world view is defined by whether the puzzles follow designer logic or observational logic. For the former category you basically have to figure out what the designer was thinking when he was creating the puzzle. Very often there is only one solution and it might not be the most obvious/logical one (usually the solution is very creative and funny though). The latter category describes games were the puzzles can be solved by observing the environment and making logical conclusions. I’m not going to name examples here, but I’m sure you know what I’m talking about.
The same taxonomy can be applied to programming too! For example if you have to work with a closed-source API you’ll have to start thinking like the architect of the system in order to be able to use it properly. Very often there is only one right way of interfacing with the API and other approaches will introduce obvious (and worse than that non obvious) bugs. Unfortunately there barely is a funny pay-off though. Observational logic is also important, because debugging pretty much relies on reasoning based on the changing state of systems.
Not only that but sometimes you even have to do pixel hunting when trying to find and fix syntax problems. Also did you ever notice that branching is very similar to dialog trees… 😀
Maybe it’s far fetched, but it would explain why I like adventure games and coding a lot. 🙂
I find myself playing a lot of ‘old-school’ adventure games recently mostly because working on the Double Fine Adventure project has re-sparked my interest in the genre.
Recently I got realMyst for my iPad which is the real-time rendered version of the classic adventure game Myst. I contrast to the LucasArts adventures the game emphasis is on logical puzzles. Back in the day I didn’t like Myst at all. The puzzles all seemed artificial to me and I could never figure out what I am actually supposed to do.
The latter problem is still an issue, because you find yourself on the island without any clear goals. It’s up to the player to find out what to do. I generally like exploration as a game mechanic, but Myst is going a bit too far in my opinion.
Apart from that I really enjoyed playing the game though. Most of the puzzles are solvable with a bit of thinking and the progress is timed quite well. Whenever I felt like giving up a realization somehow appeared and I could progress.
My only complaint is the forced repetition in order to get both the blue and the red page from each ‘age’. I mean I can understand that I wouldn’t be able to carry a unrealistic amount of items, but why would I not be able to carry to book pages at the same time?
The iPad controls could use a bit tuning too, but apart from that a solid (logic) adventure game. I guess there is a reason why the Myst series is so successful… 🙂
