Steven Tattersall's Accreted Drivel

Personal Reading List, 2009

2009 saw a jump in the number of books read, from 25 to 43.

“The Way Some People Die”, Ross MacDonald (reread) (2nd Jan)
“The Blue Hammer”, Ross MacDonald (reread) (5th Jan)
“Selected Tales”, Edgar Allan Poe (19th Jan)
“Alice in Wonderland, Through the Looking Glass”, Lewis Carroll (24th Jan)
“Around the World in Eighty Days”, Jules Verne (29th Jan)
“Roseanna”, Maj Sjowall and Per Wahloo (5th Feb)
“The Man Who Went Up In Smoke”, Maj Sjowall and Per Wahloo (14th Feb)
“Getting Things Done: How to Achieve Stress-free Productivity”, David Allen (16th Feb)
“The Doomsters”, Ross MacDonald (26th Feb)
“The Picture of Dorian Gray”, Oscar Wilde (1st March)
“The Man on the Balcony”, Maj Sjowall and Per Wahloo
“The Laughing Policeman”, Maj Sjowall and Per Wahloo
“The Fire Engine that Disappeared”, Maj Sjowall and Per Wahloo
“Murder at the Savoy”, Maj Sjowall and Per Wahloo
“The Abominable Man”, Maj Sjowall and Per Wahloo
“The Locked Room”, Maj Sjowall and Per Wahloo
“Cop Killer”, Maj Sjowall and Per Wahloo (reread)
“The Terrorists”, Maj Sjowall and Per Wahloo
“The Heckler”, Ed McBain
“No Beast So Fierce”, Edward Bunker
“The Siege of Krishnapur”, J. G. Farrell
“The State of Africa”, Martin Meredith
“Lady, Lady, I Did It!”, Ed McBain
“I Married A Communist”, Philip Roth
“Ten Second Staircase”, Christopher Fowler
“The Secret War with Iran”, Ronen Bergman
“Cop Hater”, Ed McBain
“The Thin Man”, Dashiel Hammett
“Bad Blood: the Secret Life of the Tour de France”, Jeremy Whittle
“Rough Ride”, Paul Kimmage
“On Murder”, de Quincey
“The Art of the Idea”, John Hunt
“The Soul of a New Machine”, Tracy Kidder
“The Modern Con Man: How To Get Something For Nothing”, Todd Robbins
“A Most-Wanted Man”, John Le Carre
“White Corridor”, Christopher Fowler
“Brave New World”, Aldous Huxley
“The Drowning Pool”, Ross MacDonald (reread)
“Black Money”, Ross MacDonald (reread)
“Horse’s Arse”, Charley Own
“Der Fall Galton” (German translation), Ross MacDonald
“The Big Blowdown”, George P. Pelecanos
“Computer Vision: Principles and Practice”

About Steve

It’s no good running a web site unless you get a page to yourself, is it?

Steven Tattersall

I now work as Principal Programmer at Sony Computer Entertainment Europe, at its Cambridge studio.

Some past games and projects I have worked on are:

Medievil 2
(c-12)
Some work on Primal (link 1 2)
Ghosthunter.
Some of the SCE E3 2005 demos that launched the Playstation 3
24: The Game.
PlayTV.

Other bits:

Older drivel.
Freekick.

GDC 2009 Report

So, as usual, I’m so late with my conference notes that the next big conference (Siggraph 2009) is already rolling round.

“GDC is big. Really big. You just won’t believe how vastly hugely mindbogglingly big it is. I mean you may think it’s a long way down the road to the chemist, but that’s just peanuts to GDC.”

– with apologies to Douglas Adams

I really wasn’t prepared for just how many people go to GDC. When you’re used to the European version, going to a conference with 17,000 attendees is a real eye-opener.

As with most write-ups, I’ve got a bit tired towards the end. I’ll try to go back later to bolster the detail in the later talks.

Wednesday

Keynote

I missed this (I was in a meeting). So I missed the free copy of Rhythm Heaven that was given out. Which is a shame because my admiration for Nintendo grows every month. You can find an exhaustive write-up on Wired here

Media Molecule: ‘Winging It’ - Ups, Downs, Mistakes, Successes in the Making of LITTLEBIGPLANET

Speaker: Alex Evans (Technical Director, Media Molecule), Mark Healey (Creative Director, Media Molecule)
https://www.cmpevents.com/GD09/a.asp?option=C&V=11&SessID=9012

Thorough write-up

The Molecules managed to more or less fill a big hall with their engagingly high-tech yet pseudo-shambolic talk. Using some custom code that Alex Evans had written on his Mac, they used diagrams draw in real time, pre-built images and text to paste together “slides” in real time on a virtual graph paper display. (At one point A.E. pulled down the screen to reveal some of the app’s code. Tellingly the file was called “craplib.cpp”)

They called their talk “Winging It” but really that does a slight injustice to how flexible – or chaotic? – their development process is. MM seem to be ruthless at reworking and refining their game during its development. As part of their talk they showed a video of Milestone 6 of LBP (which appeared to be an early Greenlight demo). The core of LBP was clearly there in many ways, with the cloth-feel graphics and the same intro-level music. On the other hand, as they admitted, they had gone down several blind alleys. One was the attempt of a giant beanstalk in the game, not suited to a 2D platformer, and custom boss fights.

Perhaps the biggest change, though, was in terms of level creation. Their original strategy was to use in-game tools such as a paintbrush and shotgun to affect the world, something you could see in the demo, with the sackboy avatar with a long paint roller in his hand, desperately trying to paint the world competently. This was all ditched later for the custom editor and Pop-It tools. This was a very late decision in the project, previously there had been a tension between the “Level Designers” and “Pop-It Iterators”.

The culture of prototyping and the legacy of Lionhead was obvious. [If they want to try something]… “People just go ahead and do it” said Mark Healey at one point, and he showed videos mockups of some features that team members created on their own initiative. It made you want to go out and just do something. Very inspiring. This is helped by the fact that as Alex said, they have a very small codebase, and all their technology is under their own control. I think he said 5 minutes for a full build, but that might be my fevered imagination.

It clearly wasn’t all plain sailing though. Mark mentioned that the team had nearly come to blows over the creation of Sackboy’s design (they eventually had a “face off” to see who had the best design, then gave final say to one artist) and the integration of physics. It also took a long time to argue for the inclusion of the in-game mapped switches that LBP ended doing crazy things with, which seems such a fundamental component of the game that I’m amazed. So, in all, the transition to the final “Game Creation Package” took a long time to take hold fully. And even then, with the first private Sony beta, for MM families and Sony employees, the levels created were so poor that they thought they had made an enormous mistake…

In three keywords: inspiring – imaginative – ninjas

Level-5’s Techniques to Producing a Hit Game—-From PROFESSOR LAYTON to INAZUMA ELEVEN and THE ANOTHER WORLD

Speaker: Akihiro Hino (CEO/President, LEVEL-5 Inc.), Usuke Kumagai (Programmer, Level-5, Inc.)

https://www.cmpevents.com/GD09/a.asp?option=C&V=11&SessID=9357

This talk was real-time translated, and I suspect it lost a little in the process. As well as doing a franchise that is sweeping the world, Level 5 have at least two other games in production. There was also “Inazuma Eleven”, a combination of football game and RPG, and “Ninokuni” which is a collaboration with Studio Ghibli.

Of all the words in the title, “Hit” seemed to be the point of the talk. The subtitle was “how to make a fun game sellable”. It concentrated on why they thought they made their games sell. The gist was: they have a promotional strategy set up even before the game went into full development.

Through the power of a dodgy translation they cited two key elements that they used when selling a game

“Catch Copy Planning” Subtitled as “loading the weapons”, this appeared to boil down to “have key selling points in mind before you start”. When thinking of game ideas Hino-san said that he imagined a conference presentation from it, and how the audience would react to it. For Professor Layton the 3 key ideas were

Puzzles and story in the same game
The “mental exercise” theories of Dr Akiro Tago, who is extremely popular with children in Japan
Animation and voice acting beyond the quality of that seen before in a DS game (in Japan, the voice actors used are much more famous than those on other DS games).

These ideas don’t seem particularly revolutionary to us, but I think the key was that he had a clear idea of the products core strengths before setting out. In addition, they also produced a promo video before development started, setting out the key ideas to the development team, so that they also had a clear idea of what the project was meant to be. Very little gameplay was shown, even in videos which had been re-edited much later in development.

The second half of “Catch Copy Planning” was to identify the core market for the game (the female market was identified for Professor Layton) and to match the game content to the market and pitch the game directly to them.

“Boom Trigger” Subtitled as “Creating a big fire”. This was suggested ways to make people talk to one another about the game (what they termed the “Communication Gimmick”) and extending the lifetime of the game (“Extension Gimmick”). So for Prof L the Communication Gimmick was asking friends to solve puzzles for them (possibly using the wi-fi connection?) The Extension Gimmick was the sheer number of subgames, and the ability to unlock and download new puzzles after the release date.

So again, nothing too extraordinary, but it was noticeable that they applied these principles to all the other games they produced, including their football RPG.

There was much excitement in the audience about the Ghibli tie-in. Including some footage of the new game. Akihiro Hino admitted to being like a schoolboy in their presence, which was quite sweet.

In three keywords: hits – boomtrigger – cryptic

An American engine in Tokyo: The collaboration of Epic Games and Square Enix for THE LAST REMNANT

Speaker: Mark Cerny (Consultant/Developer, Cerny Games), Michael Capps (President, Epic Games Inc), Robert Gray (Technical Consultant, Square Enix), Hiroshi Takai (Director, The Last Remnant, Square Enix), Daniel Vogel (Lead Engine Programmer, Epic Games, Inc.)

https://www.cmpevents.com/GD09/a.asp?option=C&V=11&SessID=9350

Square Enix did something few Japanese developers would even consider: they switched to using Unreal for one of their games. So the panel had 2 very voluble Americans, a quiet German lead programmer, and a near-silent producer from Japan.

Reflecting on it now, the only thing that sticks in the mind was Mr. Square’s initial response when asked “what were the problems with working with Unreal 3”. The response was to pretend that the question hadn’t been asked at all and to stare into the distance until the question went away. Like my mother used to say, “if you can’t say something nice, don’t say anything at all”.

As with most panel discussions things were very disjointed, so I might have to resort to bullet points

The Squeenix guys got a bit of a shock when they first saw the code, their initial response was “how on earth can you ship a game with this”? They refused to buy a licence until Epic committed to translating all their documentation into Japanese. Unfortunately Epic hired cheap-ass translators who (a) translated the wrong documents, starting with Unreal 2 docs and (b) did things like translate “Actor class” into “man who plays in films class”.
Square sent engineers to the East Coast to “embed” with Epic. This is the first time any developer has done this.
Epic development process is very different from the Japanese. Square based around the idea of only bringing assets into the game when they are done. Unreal is based around iteration and effectively generating content in-game. This was a major cultural shift.
Their biggest asset was Rob Gray who acted as a “conduit” between the two companies. He could translate and act as a mediator. It sounded like the whole process would have failed without him.
Major problems (they answered eventually!) were:
- artist-driven development was new to them. They jumped in and made a lot of mistakes early on that made it very hard to fix things later.
- constant engine change. Updates took 1-2 months to come through to the game teams because of the upgrade time. Epic constantly asked “when will the engine be done?”
- working with the engine. The Unreal Lead Programmer effectively admitted that some of their code might not be great, but it’s there for a reason i.e. someone will be using it. So they never change for the sake of it. (Personally this sounds like a horrible situation for any programmer to be in, I wonder how long this can last).
- Square very used to a fixed, tight memory budget (“not 1 kilobyte wasted”). Garbage collection made them very nervous and Square reported “wastage” of up to 30MB. Epic didn’t really have an answer to this.
- Compromises had to be made. Square wanted lots of characters on screen, more than UE could do. They had to scale down as a result. There were “lots of areas” where it didn’t meet expectations.

When asked if it was a success, the Japanese producer described it as a “great experience”, which I think meant “no”. On the other hand, he did say that he got much happier as development went on (“we learned a lot”). Some of the team do want to use the engine again.

I notice Square have just licensed Gamebryo (albeit from a different internal studio). I wonder how much of that is based on the Unreal experience.

In three keywords: cultural – cagey – misunderstandings

Thursday

Valve’s Approach to Playtesting: the Application of Empiricism

Speaker: Mike Ambinder (Experimental Psychologist, Valve)

https://www.cmpevents.com/GD09/a.asp?option=C&V=11&SessID=8884

Valve make great games, but I’ve always been left disappointed when seeing their talks. Maybe I’m unlucky. Some people raved about this talk, but it seemed to me to be quite a dry, not too incisive runthrough of techniques on how to playtest a game. It suffered from trying to cover too many subjects rather than picking one and covering it well. So it ended up being a bit of a litany of starter information, with each technique discussed in turn. There were still a few nuggets in amongst the stones though.

Valve use:

Direct Observation
Verbal Reports (player speaking out loud while playing)
Q&A
Statistical Tracking
Design Experiments e.g. community polls
Surveys
Physiological Measurements (eye tracking, skin conductance, face sensors)

The advantages and disadvantages of each can be found quite readily with a bit of Googling, so again some snippets of things I thought were non-obvious or worth repeating.

Direct Observation (as most of these methods) needs a specific design goal to be useful i.e. “is this UI working”. Just watching without any aim isn’t worth it.
Of the pieces of footage shown, it’s obvious that Valve go a long way to make their UIs very, very, well… obvious. Big glows round characters, lots of icons were added if the player missed any important information. Trying to be too subtle didn’t work for them. Which is what we have found time after time :-)
What the player does, rather than what she says, is primary. For example, the player often gives the wrong explanation as to why they have done something after-the-fact, in order to self justify. It is too easy to bias the results with the wrong context. (I wish the presenter had given references for some of the psychology studies he mentioned. Some of the experimental results he stated were almost too good to be believed). But the bottom line is “people don’t know why they do what they do”.
When doing surveys, a common trick is to repeat the same question in different ways – and get different answers from the respondents. This is a way of correlating the underlying answer and checking the answer isn’t susceptible to innate (too much) bias or influence.
Valve’s play testing continues long after initial release. There is online stats gathering, polls, testing.
Some people have successful used an unusual marker to measure involvement: the rate a player is seen sipping his/her drink has a good correlation to their involvement!
When designing UIs, they try to limit long single eye movements (say between opposite corners of the screen), since this is known to be very tiring for the player.
Using skin conductance, it’s known that players get a second spike in their body alertness between 10 and 15 seconds after the initial shock. So designers use this to time secondary attacks in the game for maximum scariness!

It’s still reassuring to see effort in the direction of “design as hypothesis, so test it as scientifically as you can”, since that’s the direction we seem to be moving in as well.

In three keywords: dry – thorough – a-bit-dull-really

Solid Game Design: Making the ‘Impossible’ Possible

Speaker: Hideo Kojima (Head of Kojima Productions)
https://www.cmpevents.com/GD09/a.asp?option=C&V=11&SessID=9459

This talk has been done to death, and there are far better write-ups than I can manage (my favourite is on Wired. And I missed the start by being in a meeting with the Havok guys. The highlights were the Japanese MSG adverts, the one in the office and the jungle were hysterical. Unfortunately I can’t find them on YouTube :-(

In three keywords: isometric – adverts – dig-at-ps3

All About Noby Noby Boy

Speaker: Keita Takahashi (Bandai Namco)

Probably my favourite talk from the whole conference: it was just so… so… nice. It started out with a knitted scarf that looked like Noby Noby Boy himself and turned out to have been knitted by the designer’s mother, and ended with Takahashi gently inquiring if one of the questioners in the Q&A section maybe had a few, you know, issues?

More-or-less complete transcription is here It’s well worth reading to see the mental processes of someone who is clearly viewing the world from a different angle from most people. Sometimes I wondered if the games industry was sending him mad… he is certainly treading a separate path. No wonder he at one point declared that we should “ignore players and companies and make the games we like” and “do not fear failure”.

There was no real structure to the talk, so in many ways it matches the Noby Noby Boy “game” itself. It just gradually unfolded in a way that sort of made sense at the end. Similar to the Media Molecule talk, Takahashi-san used drawings and videos rather than a traditional presentation. Maybe the word we shouldn’t fixate on is “game”. Maybe it’s “play” instead. (Man, that’s deep). The presenter described NNB as being like “buying a ticket to a festival” and his goal as “creating something enjoyable”, which ties in with that somewhat.

In three keywords: charming – whimsical – inspiring

Experimental Gameplay Sessions

Speaker: various

another writeup

2 hours, 10 presentations, lots of indie games. How “experimental” they were seemed to vary (Flower seemed an odd choice, and Derek Yu almost seemed like he was chosen because he’s popular in the indie scene), but everything had at least one interesting element to it.

Website

You can download and play quite a few of them, and see videos. In order the games were

Unfinished Swan This looked like a really cool video that was still searching for a game to attach to it
Shadow Physics Steve Swink, Steve Anderson
Miegakure possibly the most bonkers of the lot
Spy Party - Chris Hecker
Daniel Benmergui “I Wish I Were the Moon” ,”Trials”, “StoryTeller”, “Today I Die”
Flower - Jenova Chen, Nick Clark
Achron - Hazardous Software (I have the word “MADNESS” written in big letters in my notepad next to this one)
Closure - Tyler Glail
Where is My Heart - Bernhard Schulenburg
Rom Check Fail - Farbs
Roguelikes - Derek Yu

At least 3 or 4 of the games (Miegakure, Achron, Closure, Shadow Physics) were influenced, or at least in the mould of, recent games where the concept of “mapping different dimensions to one another” is used, Braid being the most obvious. They ranged from the merely quite-bonkers (Shadow Physics), to the totally-bonkers (Achron, Miegakure).

In terms of new gameplay or mechanics, I thought Spy Party – although flawed in its implementation – was the closest to being something genuinely novel. The idea was for the player to watch on-screen characters interact (in a party situation), then try to work out who was the other real-life player, and who were the AIs. A sort of “reverse Turing Test” made more feasible by limiting the range of player “language” to something simpler than full natural written English. Unfortunately, the “gameplay” consisted of just watching for a tell-tale animation playing on a character rather than spotting things like patterns of behaviour.

The game that I was most interested in playing was “Today I Die”, a simple 2D game where the lyrics of a short poem were written on screen. By playing the game you could collect new words and swap them into the poem. And when the words swapped, the gameplay on the screen matched the new words, so new games opened up. It was the kind of thing you need to see – it becomes far simpler to explain. And quite charming in its execution.

In three keywords: mindbending – fun – exhausting

Friday

Lots of design talks yesterday, so today I put on my programming hat, and the unveiling of…

Rasterization on the Larrabee: A First Look at the Larrabee New Instructions (LRBni) in Action

Speaker: Michael Abrash (RAD Game Tools)

SIMD Programming with Larrabee: A Second Look at the Larrabee New Instructions (LRBni) in Action

Speaker: Tom Forsyth (Intel)

Note The slides for these presentations are available here on Intel’s site The slides are in PDF so don’t include the extra notes that Michael Abrash promised. Curses.

It’s probably best to lump these presentations together, although the style of presentation was somewhat different. Larrabee is Intel’s attempt to take on the hardware graphics cards vendors by gluing together lots of old Pentium chips and developing a software renderer. Actually that’s a horrible mis-representation, but the whole thing does remind me of a MacGyver-style “it’s crazy but it might just work!” skunkworks.

Abrash and Forsyth are two of the Smartest Guys in the Room, so if it’s possible they will be the ones to do it, but at the moment the jury is definitely out as to whether the numbers will work out. It looks like they are betting on using the inherent parallelisation of some algorithms involved in rendering, and load balancing to occupy the chipset’s time and spread out workload to constantly feed the processors, to give them enough of a gain to offset the inevitable performance decrease you will get by not using custom hardware to do your rendering for you. And those gains and losses are much too hard to define to just stick some numbers into Excel. And we haven’t gone into the power consumption either…

Michael Abrash’s talk was easily the most hardcore talk I went to. Not only did it feature extensive assembly language listings – which I love, but I know are an acquired taste – he also deliberately raced through some quite involved triangle rasterisation algorithms in order to fit into his alloted hour. And I can honestly say that the attendee sat next to me did, genuinely, fall asleep (he was a forum moderator on gamedev.net, I checked his pass). It was a really interesting concept though, so I need to wait till the full slides come out to go through it in-depth.

Tom Forsyth’s talk was much easier to follow, and still had a gratifying dollop of assembler. While Abrash’s talk focussed on how using clever algorithms could help you get a win over a GPU architecture, Tom’s talk went into more detail about the extra instructions they had add to the traditional Pentium instruction-set. With the addition of instructions with 3 operands, and the mask register usable on many parallel instructions, you can end up with

vmadd213ps vR{kL}, vR, vRstart ; I think this is vR * vR + vRStart (with mask), result in first operand (vR)

The new scatter-gather instructions look neat. They are meant to allow easily packing/unpacking between SoA and AoS data format, so you can easily shuffle them into formats that can be parallelised easily. However, the downside is that they use memory access just like any other instruction, so the cost of cache misses could be crippling. Again, the proof is in the pudding.

In three keywords: hardcore – assembler – abrashtalkstoofast

State-based Scripting in Uncharted: Drake’s Fortune and Uncharted 2: Among Thieves

Speaker: Jason Q Gregory (Naughty Dog)

This talk was packed 10 minutes before it started, although I have to confess that I didn’t find too much new in this talk. It generally talked about how they attach scripts to objects and use their scripting language to set up scripted sequences, and add a bit of variety to things like making prop objects wobble when the player is climbing on them, and fall off when the player jumps off.

All the usual issues such as making the scripts re-startable if the entity was streamed back in came up. There didn’t seem to be any solution other than “write your script correctly”.

ND are using a custom scripting language largely based around PLT Scheme which I guess you could broadly categorise as a Lisp variant with very strong typing. Partly this is because of ND’s heritage with Goal (a “kind of” Lisp, it seemed to be a lot more complex than a simple categorisation) on PS2. To me using Lisp’s syntax but not its amazing weak typing seems to be, at first sight, taking all of Lisp’s disadvantages and few of its advantages, but ND are the ultimate arbiter of that. I’d love to see how their scripters work, or indeed what background their scripters have.

One side-effect of the use of a Lisp variant was that their scripts ended up as a mix of code and data in “state blocks” (I think to represent their state machines).

Debugging seemed to be a case of using printf(). However, what Jason did seem to intimate, and what I’ve seen in the past, is that education, documentation and training are the most effective debugging tools in many cases.

In three keywords: scripting – standard – lisptastic

Real-time Research: An Experiment in Understanding Games

Speaker: various

Worst talk I saw :-( I regret not going to the Peter Molyneux talk instead, and it’s not often I say that.

The idea was that during GDC itself, small teams of people went around doing research on the attendees themselves. The “experiments” were things like conducting surveys, analysing Twitter and blogs for the “#GDC” tag or analysing the use of business cards (I think – it really wasn’t clear what that one was about).

It’s fair enough that the work wasn’t intended to be peer-reviewed or anything, but unfortunately a lot of the work was conducted so roughly as to not give even the semblance of a sensible result. For example, one experiment involved asking attendees to pick a point on the Richard Bartle player interest graph which map onto player styles, and build a profile of disciplines. However, the experiment involved sticking Post-Its on a largeish cardboard sheet, and as the Post-Its accumulated, people started to stick their entries further and further out to avoid obscuring others’ entries… oops.

David Tisserand of SCEE was in the presenting list for the Business Card experiment. I think he was a bit embarrassed about having been roped in, sorry David!

Side note: I don’t know if the poor standard of work was related to the fact that this talk had the highest factor of academics presenting and attending. Sample set is a bit too small.

In three keywords: disappointing – amateurish – wordle

Dynamic Walking with Semi-Procedural Animation

Speaker: Rune Skovbo Johansen (Unity Technologies)

Rune’s website

I saved a good one till last. Tom Forsyth’s blog had recommended this talk and I wasn’t disappointed. It was a tech talk about how to do runtime skeletal manipulation of a walking character to handle correct foot placement. And it also handled quadrupeds really well. Very impressive.

Gavin Clarke did a lot of very similar work for us on Ghosthunter on PS2 to get what was for its time real state-of-the-art foot IK. Many of the ideas here were the same, but it did some very good analysis to allow “hands-free” integration. So for example, it analyses the walk cycle animation to determine automatically the foot-down and foot-up times, length of stride and foot on the floor. It also goes into some depth about how to handle tricky cases, such as bending the trailing foot when stepping down from a ledge, which often looks very ungainly unless you get it right, or smoothing out the root transform of the character as you move up and down steep changes in height.

Good advert for the Unity engine, too. I’ve been guilty of thinking of it as “just a Flash-like browser game engine” in the past, I need to reconsider because they were doing some very good stuff.

In three keywords: animation – quadrupeds – impressive

Things To Remember Next Time

These are things I wish I’d know before I went:

take a laptop. There is lots of free wi-fi, but no pods with PCs
find a quick timetable – the provided one is useless
scout out the venue in advance so you can easily find where all the talks are held
don’t trust the titles of the talks! Generally they are either vague, trying to be too clever, or in some cases downright misleading. A better guide is stick to speaker’s reputations, or read an in-depth description of the talk

and finally, but possibly most importantly:

avoid the vegetarian option for the free GDC lunch. Or for that matter, probably the non-vegetarian option.

Personal Reading List, 2008

Inspired by Matt Webb sooper genius, I’m starting to keep a log of books read. It’s probably a bad idea to be inspired by someone who seems able to knock off several erudite editions per month. But I have to start somewhere.

It’s always hard to find time to read — or rather it seems to be. After a hard day’s ~~programmi~~ management, it’s too easy to flop down in front of the TV. Hence I am making an effort to structure my reading time. Faced by options paralysis in the bookshop yesterday, I decided to limit my choice of books to fiction authors only starting with the letter in the cabinet in front of where I stood: “F”, as it turned out.

I doubt this will be a hard-and-fast rule: I already know that I will break it for non-fiction reading, mainly depending on what my work requires. But it certainly made it easier to choose a set of books and push me to read books I’ve considered reading, but never quite got to. The aim is “structure”, rather than “constraint”. I guess I’ll see how it goes.

“Christine Falls”, Benjamin Black (a pseudonym of John Banville)
“The Death of Dalziel”, Reginald Hill
“White Heat: A History of Britain in the Swinging Sixties: 1964-1970”, Dominic Sandbrook. A monster, it took me several weeks; an intriguing attempt to blend the machinations of the Wilson government with the pop-culture of the Beatles and the Rolling Stones.
“Dark Full House”, Christopher Fowler. Really enjoyed this; had a playful sense of humour that I’ve not seen in crime thrillers for quite some time. (Finished 10th April)
“Designing For People”, Henry Dreyfuss. First published in 1955, an easy-to-digest blend of timeless good advice and delightful “oh what were they thinking” archaisms. The “predictions for the future” are more accurate in the 1955 version’s final chapter than the revised 1967 edition (Finished 12th April)
“The Merchant of Venice”. In preparation for our impending trip to Stratford to see the RSC production. I can see why it’s called a “problem play” now (13th April).
“Don’t Make Me Think!” (Second Edition), by Steve Krug. This has been sat on my desk at work, half finished, since January, and I have finally mustered the energy to polish off the second half. (14th April).
“A Clubbable Woman”, Reginald Hill. Comfort reading. (21st April).
“The Foreign Correspondent”, Alan Furst. (3rd May).
“Towards the End of the Morning”, Michael Frayn.
“The Water Room”, Christopher Fowler.
“Absalom, Absalom!”, William Faulkner (aborted — unreadable)
“Under The Frog”, Tibor Fischer
“A Death in Tuscany”, Michele Giuttari
“Independence Day”, Richard Ford (4th July - 3rd September)
“The Interpretation of Murder”, Jed Rubenfield
“Seventy-Seven Clocks”, Christopher Fowler
“The Return”, Hakan Nesser
“Ways of Seeing”, John Berger
“The Sportswriter”, Richard Ford
“Cos� Fan Tutti”, Michael Dibdin (reread)
“The Tailor of Panama”, John Le Carr�
“A Cure for All Diseases”, Reginald Hill
“Cop Killer”, Maj Sj�wall and Per Wahl��
“The Dip”, Seth Godin

So, round about a book every fortnight. This is pathetic in comparison to Mr. Webb. I think this can be improved in 2009. I am starting to find techniques to squeeze in more reading, and less television.

Around 50% of my consumption has been crime fiction, with the percentage increasing towards the end of the year, as my reliance on the letter F tailed off. And some of the first titles of 2009 will be the Ross MacDonalds I got for Christmas. I think it might be time to retire the letter F and switch to a new pattern — Penguin Classics, perhaps.

Memory Debugging on “24: The Game” (PS2)

Author’s Note

I originallly wrote this piece for an internal Sony discussion about memory debugging, about 2 years ago. In lieu of actual creating any original content for this website, I thought I’d resurrect this piece, and polish it to make it less dry than the original report. It’s unusual, since games developers rarely admit in public to anything that doesn’t sound like a shiny, new, perfect game engine. But game development is now more the art of the possible than the art of whizzy code.

I should also point out that a lot of the hard work was done by the two people who worked on the systems originally: Chris Sorrell and Gavin Bell. Kudos to them.

Background to “24”

I originally started working on “24” in mid-2005. By that point the game had had over 2 years of development on it. “24” was based on Cambridge Studio�s in-house PS2 engine, started in 1999. This is the fourth game based around the PS2 engine. It was originally developed for Primal, a third-person action adventure with on-foot action. “24” had a very different requirement set, as it attempted to incorporate free-roaming driving missions. The loading patterns for driving areas caused vastly more “problematic” memory usage than the slower on-foot sections. There were also quite a few subtle issues caused by persistent memory allocations, which we will see later.

With a couple of months left on the project we were still having serious memory and stability problems, so extra staff was added to work on diagnostics (i.e. me).

Some Terminology

Terminology for memory allocation seems to change with every engine I see, so here is the list for this piece. (This differs from the class names in our code, which reused “pool” in more than one context).

Heap: a large block of memory using a standard memory-allocation algorithm: for us, it was usually a “best-fit” algorithm;
Pool: chunk of memory acting more like an array of fixed-sized allocations for quick allocation. Its own internal array was allocated from a heap;
Allocator: a chunk of memory allocated on the heap, inside which another custom allocator type was used (usually first-fit).

The Standard Heap Memory Allocator

The core memory allocator had been custom-written from the ground up, six years earlier. It was, in essence, a best-fit allocator using boundary tags (as far as I could tell). It also had some custom bucketing code to try to boost performance — to skip any candidates that were known to be too small for a potential allocation, it tracked the free areas in “size buckets” and only looked in the bucket of sizes that were as big as, or bigger than, the amount of memory you wanted to allocate.

The basic heap allocator also offered the usual additions:

Alignment for the base allocation plus alignment for elements of array;
Deferred free processing (for render allocations). This is where the allocation would stay around for a couple of frames after the allocation was explicitly freed, to allow it to be read by the in-progress GS graphics rendering.
“Slave” allocations, where deallocation of one memory location would cause the slaved allocations to also be freed. (By the end of “24” this was unused, after a rewrite of the rendering system)

Memory layouts and patterns

There were several heaps in the game:

The main, or “global” heap: for all allocations over 8K in size, or any allocation that could be relocated. In the final shipped disk build, this was about 23MB in size, out of the PS2’s 32MB total.
The “bantam heap”: this was a separate heap, for any allocations under 8K in size. This ended up around 5MB in size. If the bantam heap overflowed, and it did, allocations would overspill into the main heap. (Those of a nervous disposition will get worried hearing that).
Debug heap. Debug allocations would go in the bantam heap on non-debug versions, or be blocked at compile time (debug allocations like this were prevented unless they absolutely had to be allowed).
Havok heap. Havok kept a completely separate heap of about 2.5MB, not managed by our systems.

All the heaps and their sizes were allocated at boot-up of the game. We couldn’t reinitialise heaps at runtime, as the engine design meant that some systems (particularly the render system, as it turned out), couldn’t cope with it.

The Hard Bits

The Really Hard Bit was getting rid of memory fragmentation.

The resource system and loading system on “24” made a separate allocation for each loaded resource (and possibly extra allocations on post-load processing). During runtime, there could be up to around 2500 allocations in the main memory heap, most of which would be resources.

These are the main root causes of memory crashes we had in 24.

Heap fragmentation: the main heap in particular would fill up due to fragmentation. Even with resource relocation (see later), we would often see out-of-memory crashes with around 1.5MB still free in the heap. Also fragmentation would be extremely unpredictable on different play-throughs.

Granular handle-based loading system. Our game resource system was completely handle-based and tried to reuse already-loaded resource data. Rather than have a hard-specific process for the loading of a chunk of assets, creating a handle for an object would register a request to load the data and it would appear some time later as a separate allocation. Certain “group” handle requests would pull in a block of files linked to that group (an informal form of WAD handling, although without compression). An example would be the set of geometry files for an area of map. Each file within that group would still count as a separate resource and have separate allocations. Creating a handle to an already-loaded object would simply reuse that resource and increment its reference count. Resources were deallocated only when all handle references to the object were lost.

While this was often a boon for programmers� productivity (although all our code had to check for handles pointing to loaded data), it meant that it was very difficult to explicitly control loading/unloading of resource allocations, and so have any control on the memory allocation patterns. Code would often almost invisibly trigger loads/unloads.

As a result, resources loaded in at the same time, and hence usually allocated close to one another, could have completely different deallocation times. Looking at research papers, this is a core root cause of fragmentation.

Leak tracking. This was particularly problematic. Until the end of development, the engine did not reset most systems between levels/game starts. This led to leaks going undetected for long stages of development. In the end, there was a lot of work recoding systems so we could reset them, and adding tagging information to reliably track these problems. In addition, some systems did not leak per se, but held on to allocations beyond level scope. See the “Conclusions” section for more about this.

Bantam Heap overflow. Due to historical reasons, we had to allow the bantam heap to overflow, particularly in areas with heavy rendering or lots of NPCs. Normally the situation would “right itself” when memory levels dropped again. However, some allocations would persist for the course of a whole level, or in some cases even longer (see “Leak tracking” above).

Render allocations. Memory allocation for rendering was highly dynamic and used granular allocations in the bantam, then main heap. This would cause spikes in the memory and frequent bantam heap overflow. However it would often try to reuse allocations to save processor time, meaning that it would try to hang on to allocations in the main heap. This exacerbated the fragmentation.

Load system differences. To try to improve load times on PS2, initially-loaded data for the start of a level was compressed and interleaved with the movies, and also supplied as a WAD (if the movie was skipped). However, this also caused the memory allocation patterns to vary wildly, as allocations for movie playback buffers affected the patterns. As a result, memory fragmentation was far worse after a movie was played back. This situation only ever happened on final disk builds, so having diagnostic data on disk builds on Test machines was essential for us.

Custom heap allocation types

Although best-fit allocation performs well in general tests, we ended up with some interesting problems innate to its approach. In particular, memory “islands” caused by persistent allocations in otherwise free space caused us serious problems later on.

A bit of reading up showed that best-fit allocators are very prone to this, which is intuitive when you start to think abou it. Guess what? We were using a best-fit allocator.

Very late in the project we added custom allocation strategies to force allocations to the top or bottom of memory heaps, or into heaps that they would not have used with the default strategy. An example would be that when movies with embedded WAD data were played back, movie buffers were forced to the top of the heap, and resource allocations to the bottom. Although this was an unpleasant hack, it gave us a measure of robustness to the “islands” problem. It was invaluable in actually getting the game to ship. (In essence we were switching to using a first-fit allocator for critical areas).

Runtime Relocation

Resource relocation was added to the engine before “24”. It helped fix any fragmentation problems we had seen on earlier titles. However, on “24” it wasn�t powerful enough to fix everything. This was mainly because the balance of the mixture of relocatable to non-relocatable objects in the heap shifted between projects � there weren�t enough movable objects in the heap any more, and we didn�t have time to recode systems to cope with relocation.

Relocatable types were:

Animations
Textures
Model instances
Localised text

Everything else (in particular, character model data) was non-relocatable. Analysing the patterns of how this caused fragmentation was essential for us.

The relocation allocator code just tended to push resources to the earliest big-enough free space in the top of memory. To keep overheads down, relocation was constrained to a maximum of 4 resources, or 64K, per frame. Generally, fragmentation wasn�t limited by this relocation rate; it was more that the memory patterns were unrecoverable by that point because of large, unrelocatable resources sat at inconvenient positions in memory.

In conclusion, although we had a very sophisticated memory system, it stored up quite a lot of problems for us later down the line. What had served us comparatively well in early games didn�t stand up very well for a game with very different gameplay styles.

If I had my time again, and was the Lead, I would force non-relocatable and relocatable resources to be separate heaps, and set my budgets accordingly (easier said than done, though).

Runtime Statistics

As might have become obvious now, our engine did a lot of allocation. Loading up a driving level and checking the number of active allocations gives a figure of over 13000! Of these allocations, 12000 are in the bantam (“small”) heap.

At the same point there are 1360 separate loaded resource files, of which 900 are theoretically relocatable. However, in a typical test these relocatable resources comprised only about 6MB of the 20+MB of the allocated heap space (this ratio varies, of course).

Some tests we did after the game was released showed that we were doing around 200-300 memory operations (including freeing memory) per frame during shooting levels.� On driving levels it was around 100-200, but with spikes of up to 700 per frame when streamed loading caused object creation (a spike has been seen with 5000 deallocations in a frame during gameplay). On previous PS2 titles this was actually worse; there was a lot of work done to recode rendering allocations which cut down this figure quite significantly. Nearly all these per-frame allocations were of small size and went in the bantam heap.

Clearly this would have an effect on real-time update for any external tool. We can get some more exhaustive statistics on this if anyone wants them.

To run reliably on driving levels, we had to maintain an overall level of at least 1.5MB free in the main heap. Any less and fragmentation was liable to cause a crash.

Debug Facilities

Per-allocation data

These are the debug fields saved per-allocation on the main different build types.

Feature	Debug	Internal disks	Master
Allocation size	Yes	Yes	Yes
Allocation time	Yes	Yes	Yes
Full name tag (supplied in code)	Yes	No	No
Allocation PC	Yes	Yes	No
Mission ID	Yes	Yes	No

Runtime overhead was 32 bytes/allocation on debug disks/masters, 48 bytes on full-debug builds.

The “mission ID” field was a late addition. It allowed us to track quite effectively leaks between missions, particularly when doing automated tests, and see how long some allocations actually persisted.

Using names tags for allocations often led to problems. Cut and paste code, and template code, often made it difficult to provide useful tags and a lot of STL-style classes ended up with all their allocations with the same name until we reworked the code. This often led to a lot of subtle leaks being ignored by programmers!

Boundary corruption checks

There was the facility to do boundary corruption checks by tagging data beyond allocations with known values. Surprisingly, this feature wasn�t that useful to us, although it took a lot of work � we found only 2 cases where it fired, and they were probably debuggable anyway, as the bugs would have corrupted internal memory pointers.

The usefulness of this feature was lessened because the checks for corruption usually fired a long time after the code responsible for the corruption, sometimes several seconds later. Checks were usually done when the block itself, or surrounding blocks changed state (usually being freed). Corrupting the memory system pointers inadvertently made the problems usually come to light earlier — usually when doing debug per-frame integrity checks.

Reports

Memory reports were normally triggered from in-game menus and provided the usual set of filtering per-pool and per-type, full dumps, sorting by tag etc. I guess this is all pretty standard.

In-game fragmentation display

The in-game display of fragmentation gave us a good quick overview of memory layout. It had a small memory footprint so we could leave it operational on all disks other than master submissions.

In-game fragmentation display. Top panel is main heap, lower left is bantam heap (dangerously full!). Different colours represent unique allocations.

When used in-game, there were keyboard controls to browse memory and check the type of any selected allocation. There were also options to highlight allocations with specific names, any allocations after a set time, or by which mission they were allocated on. This gave a quick and easy way of spotting obvious leaks, and watching allocation patterns (a sobering experience).

Unfortunately, since the control was coarse, it wasn�t that useful for in-depth analysis; we generally had to resort to written log data for that.

Memory card logs

In the final stages we saw a lot of discrepancies between even “near disk” builds — where all build options were the same as a disk build except for using the disk as data source — and the disk builds themselves. We also had a lot of cases where memory fragmentation was only being reported on testers� PS2 Debug kits.

Since we couldn�t afford the IOP memory to add network support, we logged quite a lot of statistics to memory card files. This had near-zero runtime/memory overhead.

Statistics included:

Full memory dumps at given points (in particular after level shutdowns);
Overall memory levels every 5 seconds, very useful for memory hotspots;
Full exception crash information, including memory areas around register addresses (the name of the log file matching the play-through would then appear on the exception screen for the testers to add to crash reports).

The dumps were all in normal ASCII format; this made parsing very easy. Generally the testers played all day then we copied all their memory card logs off for later analysis. Using the memory cards also meant that the testers didn�t need much external tools to be run; they could use it in a fire-and-forget manner.

We then very quickly knocked up Python scripts to parse the outputs. By also parsing the MAP file that matched the disk�s executable, we could also do lookups on all the code that allocated the memory, rather than storing debug strings at runtime.

The scripts could then give full annotated and totalised reports, and visual representation of the memory layout (similar to the in-game fragmentation display). There was no interactive use of the tools; we never really found a case where it was necessary. If you really wanted to know details of an exact allocation we just looked in the text reports.

The exception handler scripts could also generate a full binary image file that could get loaded into the PS2 debugger � this was terrific for checking out complicated crashes.

In short, this system was probably critical in hunting down most of the really nasty crash problems we had.

output from a Python script showing global heap memory layout extracted from memory card logs. This shows a snapshot dumped after an out-of-memory exception, drawn in a similar way to the in-game fragmentation display. White areas are remaining free memory. The text at the start of each allocation is the truncated name of the C++ function that did the allocation. In this view, all allocations from the same code have the same colour

the same view as above, but this time filtered to show allocations inside the current level (in pink) and those from before (green). Here it�s clearer to the trained eye that we are seeing minor fragmentation in the middle of the heap. This is because of allocations that live longer than one mission � here script allocations and the language manager

Allocation histories

The debug builds supported a rolling buffer of the last 8192 memory operations in the system. It was optionally built in using conditional compilation.

Surprisingly, we only found this feature useful on one occasion. This was a crash bug involving render DMA and cache invalidation on non-disk builds, where the fact that near-consecutive allocations partially overlapped causing unexpected data cache write-backs.

Some Conclusions

“Leaks” between levels

On PS1 our approach to memory leaks was simple; we made a simplified snapshot of the memory state at the start of, say, a level. Then on exit of the level we compared the memory state to the snapshot. Anything that didn�t match was a leak.

On PS2 this approach fell down quite badly. As well as the fact that not all systems shut down on level exit, it seems that C++ code (or maybe just our C++ code!) gave memory usage that didn�t fit this model. One example would be our implementation of dynamically sized arrays, similar to STL vectors. Although they did not leak memory, they would end up with internal buffers that would grow but would never shrink back if the number of objects within it decreased. Often these reallocated buffers would be created at extremely inopportune times e.g. when the bantam heap had overflowed into the main heap, causing consistent fragmentation.

Early on in PS2 development the decision was taken not to assert when possible leaks were flagged. (This may well have been a mistake.) The warnings were always ignored by users; often the warnings were spurious, and in many cases even if the warnings were valid, they were unhelpful e.g. on the lines of “there are more generic list objects in the world than when you started”.

Given no asserts, the main approaches we found were:

a list of “acceptable” tags for leaks. This was actually quite dangerous and led to some allocations that looked OK sneaking through later on.
lots of hard work checking every potential leak. This is what we actually ended up doing.

The automated tests and memory card logs were invaluable in gathering the data for the “hard work”.

Real-time update versus snapshots

At the start of doing the final work on memory debugging we thought that real-time update would be invaluable. As it turned out, it wasn�t nearly as important as we expected. Although the in-game display was handy for getting an overall view of what was going on, most of the crucial events were only exposed in later analysis.

The most useful debug capability to me, by far, was the ability to dump a snapshot at a known point (triggered from code), and potentially visualise them offline. This let us see very clearly obvious leaks and fragmentation islands at important points. Because we still had allocation time in the debug dumps, it was easy to trace patterns of allocation.

The other really obvious point was that having this data available in an easily-parsed format was vital. We just used simple CSV-like text files. It meant that offline processing was trivial and would allow us to do automation.

Conclusions

Here are the main things to take away from all this:

Think about your memory policy as early as possible. Get at least the ability to dump the memory layout as soon as you can — you can go far with just that
Understand the different types of memory allocator, and their advantages and disadvantages. If I had my time again, I’d stick with first-fit allocators. Be aware that fragmentation is caused at the time of de-allocation, not allocation — so try to ensure neighbouring allocations get freed at roughly the same time.
Measure — don’t guess! Every man and his dog has his opinions about why you’re running out of memory. It’s the art. It’s the maps. It’s Programmer X doing lots of little allocations. Ignore them and sit and find out how much things are taking. We only needed to look at about the top 20 memory allocating functions to account for 80-90% of the memory.
You don’t need to overboard on memory pools. I’ve seen games that split all their allocations in to different pools. For most cases these are wasting memory — you can intermingle the allocations and still track them, particularly if the allocations are relatively small. 3 or 4 pools is usually enough.
Don’t mix relocatable and non-relocatable allocations. It might buy you some time, but mainly you are just postponing the point that your problems start…
Don’t over-engineer your debugging. The time spent getting that “perfect system” can be better used elsewhere.

And finally…

Everything is a trade-off. People might think “what a rubbish system”, but if you don’t ship your game, you won’t even have an article to write…

Mugshot Gallery

The people at SCEE Cambridge (past and present). Photos courtesy of Andy Ellis.

My, what a pretty lot we are.

My Little PlayTV page

I spent nearly a year as the Lead Programmer on PlayTV, and it should finally be in the shops on September 19th.

In an interesting twist, this has probably been the most straightforward project I’ve ever worked on. So it’s nice that blogs I respect and enjoy are actually getting quite excited about the product!

My time on the project is more or less finished now, and I am back working on games concepting. However, there are lots of snippets of useful information that I’d like to share about PlayTV, but probably can’t at the moment (NDAs, legal issues). This whole “DRM” saga is particularly vexing. Maybe at some point in the future I’ll be able to release them.

And of course there is the roll of honour: special thanks to our hard-working developers: Tony Pyle, Sergio Moreira, Matt Read, Darren White, Paul Crowder, Emily Brown, Nigal Raymond, Mike Redmond, Tony Godar. It turns out we developed the product from scratch several times faster than other comparable TV products. (It helps having such a powerful machine!)

About this site

It’s not updated very often, and is mainly a dumping ground for documents created outside of my work for SCEE.

This site doesn’t have comments. Send a mail instead.

This site is now built with Blosxom. It used to use Movable Type, but Blosxom suits my less-is-more ethos quite nicely.

So Little Time…

I’ve taken the day off.

While it’s great to have a worthwhile and time-consuming project at work, it means that I’m neglecting the other, more creative areas of my brain. Sometimes you need to take a break and refresh those un-tickled corners of the mind

Things I’d like to do:

Make a game! (Watch this space).
Learn Erlang, to see how well it maps on to current console architectures (and the next ones).
Finish having a go at the ICFP competition puzzles that I started.
Investigate visual recognition systems and the maths behind them.
Finish learning how to use my Sony Alpha SLR camera properly.
~~Buy a bike~~ and use it (the bike is bought, but not delivered).
Do some writing. Maybe this is a start.

Phew! That’s too much. Where should I start?

Play TV

Now that it’s been officially announced, I guess I can say what I’m working on. I’m leading the application team for Sony’s new PVR, “Play TV”. The response seems overwhelmingly positive.

Congratulations and well done to the team: the three Tonys, the two Matts, Mark, Sergio, Manuel, Nigal, Emily and the Crowdermiser. It’s amazing what we’ve achieved in such a short space of time.

Perl is like going commando

I keep using this analogy about Perl, but forgetting to get it down on (virtual) paper. So here it is.

Perl is like not wearing underpants; it feels tremendously freeing, but sooner or later you’ll get something caught in your zipper.

Found phrases: Roshambolic

Roshambolic (adj.): the state of a badly-organised Paper-Scissors-Stone competition.

UPDATE: already coined!

We know Jack

Yes, the studio is doing the game of “24”.

GamesIndustry.biz - ‘24’ Counts Down for PlayStation 2 this Autumn

If there are mistakes in the German or Italian subtitles, it’s probably my fault. That’s about the limit of my involvement, though.

We have a huge team ploughing through it as I type.

And what am I working on?

[/me looks down nervously and mumbles] erm, something.

Ghosthunter - Array Pictures

Paul de Vrijer sent a nice mail the other day, asking why you couldn’t see the ghosts in the array after about the ship (can’t remember which level). Anyway, it seemed to waste the code and art that was in the game, so I took screenshots of all the ghosts, and here they are:

Ghosthunter Cheats

It’s about time I posted the cheats to this game.

To activate, you need to:

be in-game,
hold down a certain button combo for about 5 seconds (with no other buttons held)
tap an extra button (keeping the combo held down).

Here are the current combos:

HOLD DOWN FOR 5 SECS	TAP	CHEAT
L3, digital “right”	Triangle	Disable Laz death
L3, digital “right”	Circle	Mega damage (easier to kill enemies)
R3, Square	digital “up”	Hat Cheat. Some entities will wear silly hats/costumes.

(“digital right” refers to the button directly left of “select”, not sure what the official name is)

You will get a piece of text such as “Yes!” on-screen if each code has worked.

You need to do these in the order specified (e.g. you can’t turn on the hat cheat on its own).

Cheats will be saved until the machine is reset, but not saved on memory card.

As an example of the hat cheat: here’s the teddy monster with a handbag:

There are several others, but I don’t want to spoil the fun.