Expressive Interaction Design for Music Games and Installations→ Emergence and Improvisation? (Exploratory Research)
This exploratory research challenges the lack of non-linearity in music games and interactive sound art → I propose gameplay mechanics inspired by improvisation and emergent elements, to achieve more (social) expressivity and creativity.
1.0 Expressive Music Games?
1.1 Can Improvisation Make Music Games More Expressive and Creative?
1.2 Can Emergent Gameplay Make Musical Improvisation Accessible and Inclusive
2.0 Non-Linear Game Mechanics
2.1 Basics (e.g. Path Finding)
2.3 Open Worlds
2.2 Emergent Gameplay (e.g. Emulations of Physics, Chemistry)
2.3 Sandbox (e.g. Theatrical Behaviour)
3.0 Non-Linear Level Design
3.1 Fractal Art
3.2 Procedular Generation
4.0 Non-Linear Sound Design
4.1 Adaptive Music
4.2 Musical Creativity as a Byproduct
4.3 Procedurally Generated Music
5.0 Branches of Music Games
5.3 Rhythm Games: Call-and-Response
5.4 Rhythm Games: Dance Games
5.5 Rhythm Games: Other
5.6 Games Controlled by Musical Instruments
5.7 Sound Toys
5.8 Playful Sequencers
5.9 Augmented Music
5.10 Interactive Music Installations
6.0 Proposed Design Challenges
6.1 Hackathon Ideas
Summary of Key Points
– music games (incl. rhythm games)
– non-linear music (micro/meso/macro improvisation)
– non-linear game mechanics (interaction design)
– procedural (algorithmic) audio/visual generation
– adaptive (reactive, interactive) audio
– creative / artistic / expressive / improvisational interaction
– emergent gameplay (incl. physics simulation and a bit of AI)
1.0 Towards Non-Linear Music Games
1.1 Is Improvisation the Key to More Expressive and Creative Music Games?
→ The Limitations(?) of Current Music Games
Not many music games aim to cultivate their players’ artistic expressivity: players can’t usually develop their own musical identity through performance or composition, nor can they create interactive mechanisms, such as instruments.
In other words: most music games – however enjoyable they are – are designed for chronological, linear interaction – favouring competitive goals over (artistic) creativity. And if there is creativity, it’s often not achieved through real-time, performative, improvisation.
As a composer and designer, I’ve always been passionate about non-linearity and improvisation. But opposites attract – the (creative) universe oscillates – therefore it is my belief that we need to learn from both non-linear “complexities” and linear “limitations” in order to design innovative music games and interactive sound experiences.
Imitation and Improvisation
In 1996, Masaya Matsuura and his company NanaOn-Sha released the first modern rhythm game: PaRappa the Rapper. The game challenges players to copy – and improve upon – the rhythms of various teacher characters. This call-and-response game mechanic was visually supported by a small animated timeline giving additional cues.
In 2011, fifteen years after the release of PaRappa, Masaya Matsuura explained, in an interview for gaming website Kotaku, how music game design still faced limitations:
“Strictly speaking I do not believe that ‘Music Games’ as a genre really exists yet. We just have ‘Rhythm Games’. We can’t really grow the genre until we have some games that explore areas of music other than just rhythm. I really want us to help overcome this deficiency. I’d like to for example do something that features extremely accurate musical performance animation. As an example, if we could zoom in and see lifelike fingerwork in an animated pianist, the opportunities afforded to uncover and develop new methods of playing beyond the capabilities of humans would be a crucial step in musical evolution. The ability of ‘games’ to allow us to take part in such advances is crucial.“
Unlike many later rhythm games, PaRappa embraced improvisation. To beat the teachers/opponents, players had to improvise variations of the given rhythms (more info in Chapter 5). In 2009, Masaya Matsuura stated in the Wall Street Journal (paywalled):
“For me, the improvisation is finding a new phase. Playing the instrument, creating the game ideas, or watching the characters allows me to improvise. Those kinds of things are very important”
My favourite music game (besides Karaoke) is Dance Dance Revolution (1998, Bemani / Konami), an arcade rhythm game played by dancing (…or at least by stepping on arrows). From a macro perspective, its “linear level design” approach doesn’t allow players to create, improvise or discover alternative branches of a composition – and from a micro perspective, the players’ actions within the “levels” (songs) always revolves around a rigid rhythmic grid, keeping the players disconnected from the essence of expressivity, which is micro-improvisation (subconscious fine-motoric movement). In music performance, character and individuality arise from how one (consciously or subconsciously) defies the grid – pushing, dragging and bending the notes slightly out of their “intended”, “logical” or “calculated” positions. A performance needs a little struggle, unless it aims to be robotic (which can be a struggle/art in itself, see Kraftwerk).
Neural Rhythmic Synchronisation
Research (see link below) revealed how most rhythm games (including DDR) are not (yet) suitable to induce ‘rhythmic feel’ — with the exception of players who decide to add their own improvisations (meta gameplay, such as the DDR Freestyle scene — more on that in later paragraphs). This is a result of not instructing/stimulating players to synchronize their bodies and minds to a beat. Generally, players don’t learn to neurally, subconsciously record swings and grooves (the micro-expressions of rhyhtm) – which relates back to the concept of off-grid micro-improvisation mentioned in above paragraph. A notable exception is the game Beat Sneak Bandit (Simogo, 2012).
Referenced research paper
– Music Games: Potential Application and Considerations for Rhythmic Training
Bégel, Di Loreto, Seilles, Dalla Bella | Frontiers in Human Neuroscience, 2017
Improvise to Remix
Like PaRappa, Frequency (Harmonix, 2001) rewarded skilful players with an improvisation mode – but this time players also had the opportunity to improvise on a more macro level, as sequences of notes could be looped, arranged, fx-transformed, saved and shared. And on top of that, 4 people could use this remix mode simultaneously, allowing them reconstruct / deconstruct the (famous) songs included in the game. Remarkable about this way of composing is that it’s done in real-time, retaining a game-like sense of flow and challenge.
Sentris (Timbre Interactive, 2015) innovated this perspective by fusing it with a puzzle mechanic. “When you become better at Sentris, you become better at making music”, is what designer Samantha Kalman said in an interview with game magazine Giant Bomb.
Potentials of Emergence
But even when a game does not promote micro-improvisation (fine expressivity), meso-improvisation (e.g. note improvisation) or macro-improvisation (e.g. remixing tracks), improvisation can still emerge spontaneously…
“The mission of Dance Dance Revolution is not to make people with good dancing skills enjoy dancing more but to allow people who cannot dance to learn the fun of dancing. In order to do so, we set the game rules and converted analogue dancing to digital patterns so that anyone can enjoy dancing,” says “DDR sets certain rules, and the players try to dance well according to the rules. Therefore, improvisation is a directly-opposed idea from the concept of DDR.”
This statement was made in 2009 by DDR’s producer Yoshihiko Ota stated in the Wall Street Journal (paywal). But, in the early 2000’s, a freestyle DDR movement (Youtube) *did* emerge, based on expressive, energetic improvisation, ignoring the high-difficulty game modes. I think this is a testament to DDRs versatile (level-based) design, and hints at the concept of emergent (co-creatively evolving) gameplay (more on that later). There’s great power in our bodily intuition, if we dare to let go.
Conclusions of this section
– Both linear and non-linear elements are important to music game design.
– Micro-improvisation (realistic expression) is generally lacking, yet it has the potential to emerge through body movement, performance and intrinsic creative drive, see the ‘DDR Freestyle’ movement as a rare example.
– Remix games and loop functions (macro-improvisation methods) are relatively rare.
– Computer-assisted composing is even rarer.
– ‘SoundSelf’ is unique in offering controls through vocal micro-improvisation
– ‘Beat Snake Bandit’ is unique for relying on true rhythmic synchronisation (feeling the groove, or at least the downbeat)
Additional thoughts on interactive installation design
Similar to music games, interactive installations face expressive challenges. How could visitors/players interact artistically and leave their unique marks? How could a social artwork emerge through (group) improvisation? Can visitors be incentivised to perform in front of a public? Could player remix each others works? And so forth. I believe that innovation will arise from design exchange between entertainment arts (like video games), performance arts (music, dance, DJing, martial arts, etc.), experimental arts (like installations) and public/social space (incl. architecture), and other input.
Diagram (Mindmap) of Common Music Game Mechanics
1.2 Does Game Design Hold The Key To Accessible Improvisation?
The spiritual roots of musical non-linearity and improvisation
Some music aims to tell a story, a chronological narrative that appeals directly to a certain emotional spectrum — as is relatively common in western music. Opposed to that — and relatively common in non-western cultures — are musics not preoccupied with narrative elements like beginnings, endings, or direct reference to (dramatic) emotions. They instead employ trance-inducing loops and patterns that repeatedly evolve and break apart — with optionally symbolic (meta, abstract) references to emotions, as seen in for example Japanese Noh Theatre.
While linear music relates to emotional expression and/or storytelling, non-linear music reveals our primal connection with pre-emotional consciousness and organic chaos . All music possesses some of both perspectives, I believe. Take for example Bach. He was able to compose such intriguing music because he was a virtuous improviser in (probably) most of the styles he composed in. About Bach, piano professor John Mortensen writes in his book “The Pianist’s Guide to Historic Improvisation” (via The Guardian):
“Bach could improvise fugues not because he was unique, but because almost any properly trained keyboard player in his day could. It was built into their musical thinking from the very beginning of their training.”
So, Bach – and probably many of his contemporaries – blurred the lines between improvisation and composition, and thus between linearity and perpetuity. Bach’s compositions were so powerful because they were seamless fusions of spirituality and drama – offering the listener emotional catharsis and a sense of meditation, in symbiosis.
Now of course, we don’t need only Bachs, we also need music that exploits emotional bombast (let’s celebrate for example the cathartic power of ballads and arias). Equally so, we need music that dives deep into the patterns and silence of nature – like Korean sanjo music, which is rooted in Shamanic traditions (Wikipedia). Awareness of the cultural interactions between narrative (emotional) music and abstract (spiritual) music, enables us to inspire each other in unexpected and original ways, to preserve/revive/reinterpret underrepresented forms of music, to celebrate cultural diversity and interplay. What (new) types of music would work in less linear music games? But first we should ask, in what ways can we make improvisation-based music games accessible?
How much non-linearity can a music game handle?
If precise movements, skills and high scores are the appeal of music games, are players even interested in malleable compositions and intricate, free expressivity? What are the alternatives? Competitiveness is surely a driving factor behind many music games. It raises excitement. High speed, hyper precise action can be hypnotic. But, we can surely say it’s not the only way to achieve immersion and flow.
Zen-philosopher Allan Watts said:
“The physical universe is basically playful. There’s no necessity for it whatsoever. It isn’t going anywhere; that is to say, it doesn’t have a destination that it ought to arrive at. But it is best understood by analogy to music, because music as an art form is essentially playful. (…) When Bach writes a line of melody, he doesn’t mean anything; he doesn’t try to imitate the thundering of horses hoofs, or the sound of streams, or factory whistles and the uprising of the workers. It has no social message; it’s pure playing with sound. And for that reason, among others, it’s sublime.“
–Four Ways to the Center via “HD Philosophy Lectures” on Youtube
Music is inherently exploratory. Whether improvised or composed, it always moves within this YX-grid of Relaxed-Stressed VS Stoic-Chaotic:
I made this illustration inspired by a chart I saw my teacher Rens Machielse draw in class, around 2010. It struck me that the technical essence of music could be so simple. It doesn’t only apply to the state of a composition, it also applies to the state of any instrument within the composition, see these examples:
– Relaxed + Stable = a church choir singing a lengthy harmonic chord
– Relaxed + Chaotic = a hectic jazz drum solo played with very soft brushes
– Stressed + Stable = a demonic sounding organ chord droning on and on
– Stressed + Chaotic = a fast death metal guitar solo at its climax
Can we translate this spectrum into a playful interface?
A quote from the Aeon essay “How ambient music can help ease the trauma of the ICU” by psychologist Charles Fernyhough:
“Rhythmic unpredictability is only one of the things going on. By not commanding the listener’s attention, ambient music frees the mind to wander. “
I once had a discussion with someone who argued that Bach’s way of improvising fugues was very algorithmic (rule-based) and therefore “not free at all”. I found it a frustrating argument, because I didn’t understand why he couldn’t just appreciate it for what it was. And at the same time, I was unable to decide if I agreed with him or not. I think I was stuck in binary thinking, as I now see the answer (once again) requires spectral thinking.
In 1806, German writer Goethe wrote – very freely translated:
“Brilliance can reveal itself only within restrictions – we need (new) laws to discover (new) freedom.“
So, don’t those “mathematical” classical rules offer freedom as they allow a near-endless amount of variation? Simpler rules, like those of for example Japanese flute (shakuhachi) music, also offer a near-endless amount of variation (with intricate expressive variations in dynamics playing a large role). Improvisation can thrive on rules (simple or complex, formulated or unspoken). The only restriction these rules produces – regardless of virtuosity and creativity – is a fixed mood or atmosphere, to some degree. Don’t these (cultural) define the intent of the music? Does a higher improvisational freedom arise from being able to change the improvisation’s mood or sense of cultural/ritual identity?
Some of Bach’s Baroque-instruments, like harpsichords, offer little means of micro-expressivity. A harpsichord can only pluck strings at a single level of intensity/velocity (binary, on-off). This is like the polar opposite of jazz improvisation, where musicians explore dynamics from near-silence to screeching intensity. Can true improvisational freedom be found in these expressive finesses and extensive dynamic ranges? And does that perhaps relate to unrestrained body movements? Well, without dynamics, a Renaissance fugue composition will have some level of tempo-fluctuations, some natural grooviness that will differ from person to person. Also, amazing jazz can be played on harpsichords or non-dynamic 8-bit synthesizers. The effect of micro-expressivity on improvisational freedom is again not a binary issue.
For reasons as such, I like to think of improvisational freedom as an XY-spectrum, with one axis showing the amount of emergence (how often rules and cultural codes are broken and something highly unexpected yet musical emerges), and the other axis showing the amount of think-ahead-time the improvisational actions are consciously or subconsciously based on. Unrestrained body movement is very subconscious – while thinking ahead about what track to DJ is a conscious decision, especially in case of a live performance process of assessing context (incl. listeners) and then making a conscious decision.
And, there’s always potential for crossovers between music of various rules, various cultures, various instruments, various art forms. Not only by playing together, but also indirectly: by carrying influences (however small or large) from one music to another.
Music’s abstract, universal language is its most powerful feature. It makes music is both extremely (sub)cultural and extremely cross-cultural. This particularly applies to non-linear, spiritual music – as it’s most suitable for cross-improvisation. It’s up to us to create an open-minded (global) culture in which these direct or indirect forms of creative communication can flourish.
Zooming in on Macro & Micro Improvisation
Similar to the explorative gameplay we looked at before, group improvisation is based on mindfulness. It’s crucial for all players to be deeply aware of each other. From that state of awareness, players can respond through:
– Silence: giving space to other musicians to place the focus on their output.
– Synchronisation: complementing one or more of the currently playing elements, move the music forward in its current direction through variations or dynamic changes.
– Recontextualisation: transforming one or more of the currently playing elements by playing something around them that gives them a completely different meaning.
– Disruption: playing something that completely disturbs current atmosphere, thereby giving space for a drastic change of collective course.
These four states are in their essence not very complex to grasp, plus they are applicable to all kinds of improvisational forms, such as dance, contact improvisation, and perhaps even comedy improv (which also plays various forms of expectedness and unexpectedness).
Music’s improvisational flexibility can be roughly explained by looking at the core structural elements of music:
– dynamics (the level of intensity/pressure, resulting in fluctuations in volume and timbre)
– harmony (e.g. stacking a tone upon an existing one)
– melody (morphing and shifting motifs and patterns)
– rhythm (changing the tempo, swing, meter, or adding a polyrhythm, etc.)
But it’s not necessary for all four of these elements to be improvised. As stated before, improvisation is compatible with fixed elements, presets and rules. Jazz improvisation, for example, is traditionally based on the harmonies and rhythms from the common jazz standards repertoire. And in case a drum circle, there is of course no melody or harmony present whatsoever – it places a strong focus on synchronisation and silence, through mimicking and call-and-response – and recontextualisation is achieved through polyrhythm, the act of stacking asymmetrical rhythms (for example person A looping KA-ta-KA-ta-KA-ta-GA-ta, while person B is looping KA-ta-ga-KA-ta-ga-KA-ta-ga-KA-ta-ga). Surely we can employ beatboxing as a game mechanic?
Thanks to music technologies – such as recording media, sequencers, samplers, loopers and algorithmic music generators – improvisation can also be done on a more macro level: by manipulating larger chunks of music, sometimes ahead of time, in anticipation of their playback.
Both improvisation and algorithmically-generated (random) elements, could bring a new level of mindfulness to music games. And social creativity could open new artistic potentials. Funnily, a game as simple as Karaoke already allows players to be expressive and social, without the need for an explicitly stated score (though many new Karaoke machines do that too).
Game-Design Based Improvisation – Instinct & Intuition
So, then we arrive at the question: is it possible to be musically expressive and/or inventive without any practical or theoretical musical training? Most people can, from a young age, mentally dissect a composition into basic elements like instruments, melody, tempo and intensity – and imitate them by amateuristically singing, humming or beatboxing. Take for example the beat of Queen’s “We will rock you”: we’re able to encode the song into a meme, a simple rhythmic gesture, which we can perform by stomping our feet and clapping our hands. The receiver of this gesture, or “sound symbol”, automatically decodes the message using their memory, thereby triggering the emotions of the song.
Music is very memorable. Arguably, visual arts do not have such an efficient ‘meme’ quality. Drawing a recognisable Mickey Mouse is generally more complex than mimicking a melody, and if it’s recognisable, does it remind us of any specific Mickey Mouse story? Language-based artworks (such as jokes and short stories) can be easily reproducible in an amateur way, but they are encapsulated in an extra level of, cultural encoding, potentially causing communicative (language) barriers.
Since music doesn’t necessarily need a language, it also needs no stories (per se). Music somehow relates to our instinctive and primal forms of communication: non-verbal vocalisation, movement and dance (chimps and bonobos use gestures too). Babies can dance to music, intuitively.
“We compare a matrix of the meanings of bonobo gestures with a matrix for those of chimpanzees against 10,000 randomised iterations of matrices constrained to the original data at 4 different levels. We find that the similarity between the 2 species is much greater than would be expected by chance. Bonobos and chimpanzees share not only the physical form of the gestures but also many gesture meanings.“
– ‘Bonobo and chimpanzee gestures overlap extensively in meaning‘ (2018 | PLOS Biology | Graham , Hobaiter, Ounsley, Furuichi, Byrne)
In video games, intuition is crucial, it’s an inherent mechanism:
Bushnell’s Law: “All the best games are easy to learn and difficult to master. They should reward the first quarter and the hundredth.” –Atari founder Nolan Bushnell (Wikipedia)
Not all games have tutorials, and even tutorials are playful. So, could game design hold the key to make expressive (musician-like) improvisation accessible to anyone? And what (non-linear) game mechanics could we design for that?
Game Design Essences – Risk & Reward
One of the core philosophies of playfulness (and perhaps of life) is the so called ‘risk-reward ratio’. This dynamic isn’t always noticeable or quantifiable, it can operate on a primal psychological level – hence not all games have concretely defined risks (punishments), rewards (scores, prizes) and goals. See for example play-fighting (rough-and-tumble play, romping), which is common among humans and other animals.
Thus, we can boil ‘risk VS reward’ down to the simpler terms ‘enjoyment VS challenge’: having fun while challenging one’s mind and/or body. Then we see that the idea of play applies to a wide range of activities, like Lego-building, jamming, snorkelling, and you name it. Through challenge, play allows for a comfortable confrontation with struggle, chaos and uncertainty – it’s a window to the unknown, and thus a path to exploration, discovery, wonder and inspiration – eventually coming full circle by feeding back into the player’s motivation. Play is all about feedback loops, just like life. But what are the rules (and those loops)?
Game Design Essences – What’s Unexpected?
We’re inherently curious and social creatures. Improvisation arises through intuition and spontaneity. It’s a subconscious process. Yet, as stated before, improvisation is never devoid of rules. There is an odd duality at work, or once again: a spectrum. A) Rules can be inherently intuitive or they can be culturally learned and trained. B) Rules can be either formulated or unspoken.
Improvisers – like freestyle rappers – learn an idiom (algorithms) by heart. They train these ‘unspoken rules’ methodically. When kids play-wrestle, they instinctively understand and communicate the rules of non-violence and fair power balance. Even within rule-based, competitive sports – like chess and football – players improvise, as they constantly use their (trained) intuition to deal with unexpected situations.
Good games are “random” enough to never be fully predictable, like life. Therefore, it’s no surprise that the (evolutionary) essence of human playfulness can be traced back to how a mother ‘babbles’ with her baby through the simple rules of call-and-response communication. This is how she teaches the baby to communicate and deal with (unexpected) emotions [source: The Origins and Future of Playfulness by Gwen Gordon]. Play is thus a way of making sense of both the randomness and the patterns of life, the expected and the unexpected.
For gameplay to be artistically expressive – musically or otherwise – its experience needs to revolve around control (flow) and exploration (wonder). Just like “easy to play, hard to master”, flow is about ease of control, exploration is about keeping the mind questioning. Control arises from a clear, intuitive understanding of one action’s, no matter whether it yields successful or unsuccessful results. Exploration arises when the game transforms the player’s creation in a somewhat understandable way – or vice versa – when the player’s input/creation affects the game.
Game-Design: Emergent Gameplay is Exploration by Rule Breaking
Like improvisation, the philosophy of emergent gameplay seeks for balance between unpredictability (surprise) and predictability (clear rules / limitations / goals) – but it juxtaposes a creativity question on top of this all: how can players keep re-inventing the rules without breaking that predictable-unpredictable balance / without breaking the flow of the gameplay.
Emergent gameplay makes use of playful interpretations of complex systems close to our evolution, like physics. Within these highly dynamic virtual environments, countless variables affect each other. Yet, we have an intuitive understanding of how the the laws of nature affect the behaviour of tools, objects and characters. This also relates to (quasi-)chemistry, and simulated social interactions (which will undoubtedly improve with advancements of AI).
Gameplay ’emerges’ whenever a player manages to bend or break the rules of the game without tipping the risk-reward over to either side (which would make the game too difficult or too boring). Emergence can be the result of the player…
– designing an ingenious tactic and/or tool (eureka!)
– achieving a moment of extraordinary agility (damn!)
– discovering an accidental change of perspective (wtf?)
So to embrace emergent gameplay, game design should allow these situations to happens as often as possible. Emergence is like hacking: it aims for ingenuity and question the game designer’s authority. It explores the concept of freedom and gives players a bit of design-power. Can emergent gameplay give people the freedom of music improvisation?
The evolution of emergent (non-musical) gameplay is explored in the next chapter, in section 2.2.
Game-Design: The Potentials of Emergent Musical Gameplay
My earliest inspiration for this explorative research on ’emergent musical improvisation’ dates back 10 years. In 2010 I was part of an interdisciplinary team that designed a 10 x 10 meter, 10-player installation called Thresholder. We came up with the idea to “bind” players together in a web of floor-projected quasi-elastic lines, provoking them to exert force upon each other’s sounds (exchanging energy intricate ways). When a part of the web was stretched out too far (beyond its threshold), it would snap, with social consequences. But no musical skills were needed: every sound, including the snapping, remained coherent. Exploration and social interaction were the motivators.
The project turned out fruitful, as people remained engaged and mindful as they took minutes to explore the thresholds of the “web”. This video gives an (unfortunately slightly vague) impression of the prototype we launched at Born Digital Festival. By the way, the installation had no physical thresholds, making it really inclusive, bringing all kinds of people together.
Non-linearity, playfulness, improvisation, ingenuity and finally innovation. Two of my biggest passions revolve around this philosophy. 1) Since around 2008, I improvise electro-acoustic music with classical and traditional instrumentalists. 2) In 2008 I also started to study Composition & Sound Design for Adaptive Systems, which allowed me to explore non-linear interactivity (such as game design used for open-ended, spiritual experiences) and adaptive sound design (which we’ll explore further down). It slowly dawned on me how my fascinations for improvised music and improvised gameplay were rooted in the same spiritual origin. With this article, I hope to travel from this origin into an exciting, social, expressive future.
Around 2015, it started to dawn on me how the physics-based game mechanics of Thresholder (its web of lines inspired by elasticity, tension and fragility) showed similarities with a physics-related concept that was gaining traction in the world of popular games: emergent gameplay.
2.0 Non-Linearity in Games
2.1 Non-Linear Game Mechanics – Path-Finding
Two important elements of emergent gameplay are: tactics (thinking, re-interpreting, designing) and agility (exploring, expressing, improvising). Both have non-linear components, or are on a linear/non-linear spectrum. We can explore tactics in a game as simple as Pac-Man (1980): players are allowed to roam through the maze in directions of their choice, thereby triggering the enemy ghosts to traverse diverse paths as well. Pac-Mans game mechanics don’t revolve only around hand-eye coordination, they also incentive players to come up with more clever tactics.
Poetry is a way of non-lineair storytelling. Even when a poem expresses a chronological narrative, it aims to be abstract enough for subjectivity: readers have to zig-zag through their minds to come up with an interpretation of their own. Non-linearity is also about giving people freedom of interpretation. I think Pac-Man is quite poetic. And philosophical. Do the ghosts have a free will? Free will – or rather the illusion of it – arises within a context, which this case is a ‘game-AI’ called path finding. Ghosts Inky, Pinky, Blinky and Clyde, all have their own personalities that define their path-finding behaviour towards you. Clyde is direct but gets distracted, Inky is smart but gets confused, Blinky is angry and fast, and Pinky is tactical.
This next one made me think… what if we leave the path finding ‘paths’ visible in the game, could they become musical instruments?
Here’s a flowchart I sketched, illustrating non-linearity in creative gameplay:
– Pacman Ghosts AI explained in the video:
“Nuclear Fruit: How the Cold War Shaped Video Games” by Ahoy on Youtube
– “Why Pac-Man was light years ahead of its time” by GamesRadar+
– “Pacman Killscreen” by koolkid9997 on Youtube
2.2 Non-Linear Game Mechanics – Emergent Gameplay (Defying the Laws of Nature)
Of course, various kinds of games combine agility with tactics, e.g. sports games and first person shooters. Yet, few games revolve specifically around emergent gameplay: the concept of giving players the ability to “design” or improvise their own unique tactics, unforeseen by the game’s designers.
The essence of emergent gameplay is psychological, it’s the player’s feeling of “doing something uniquely creative or ingenious”.
Emergent gameplay can flourish when objects and environment both have highly dynamic and complex properties (variables), such as (playful) interpretations of physics and chemistry. Think of gravity, viscosity, fragility, flexibility, reactivity to water, fire, wind, and so forth. Due to advancements in AI, verbal and “physical” communication with virtual characters will also become (more) emergent.
“Plans must be devised by the player; it’s not about how clever you are [as a game designer]. (…) Games that don’t offer quote-unquote ‘real choices’ might be just fine. Some people may want a game that’s all about squeezing a virtual trigger, or moving forward like a shark, or solving a puzzle that shows more about how clever the designer is, rather than how clever they are. I’m just more interested in emergence than in scripted adventures… and I believe once players get a taste of that kind of game, it’s very hard for them to go back”
– Warren Spector interview, Gamasutra, 2013 (13 years after designing Deus Ex)
Examples of games that innovated emergent gameplay:
– The Incredible Machine [wikipedia] (puzzle game, 1993)
– Banjo Kazooie: Nuts & Bolts [wikipedia] (action-adventure, 2008)
– Scribblenauts [wikipedia] (puzzle game, 2009)
– Infinifactory [wikipedia] (puzzle game, 2015)
– Deus Ex [wikipedia] (role playing action game, 2000)
– Zelda, Breath of the Wild [wikipedia] (action-adventure, 2017)
Gameplay can also be emergent in games not primarily aimed at emergent gameplay (but the emergence is then often only momentarily). See for example ‘rocket jumping’ (Wikipedia). I remember when my friends and I were exhilarated to discover this in Unreal (1998): blasting a rocket at one’s own feet while jumping, would propel the player to unforeseen (and undesigned) locations. This required agility: one had to figure out the optimal angle for the explosion, while somehow managing to retain enough health to survive.
– “Systemic Games – A Design Philosophy” (by alexbolano82 for The Artifice)
– “Scribblenauts Director Explaining Game” (Gametrailers.com via Youtube)
– “The Rise of the Systemic Game” (Game Maker’s Toolkit on Youtube)
– “Breaking Conventions with Zelda: Breath of the Wild” (GDC on Youtube)
– “Spector: Go Emergent, Game Design Is Not All About You” (Gamasutra interview)
2.3 Non-Linear Game Mechanics – Sandbox Creativity (and Theatre)
Games like Garry’s Mod (Facepunch Studios, 2004) and Minecraft (Markus Persson, 2011) don’t even contain any specific problems that need to be solved. Instead, their sandbox design stimulates creativity and artistry by allowing players to express themselves (socially) through inventively and improvisingly playing with objects, physics, camera perspective, NPCs (non playable characters) – as a sort of on-the-fly level editing experience.
– “Gary’s Mod: Black Hole VS Destructible Town (phys_bibridgeton)” (PieNinja on Youtube)
– “101 Minecraft Build Hacks” (Grian on Youtube)
2.4 Non-Linear Game Mechanics – Open Worlds
Open world games like The Legend Of Zelda (Takashi Tezuka & Shigeru Miyamoto, 1986) and Red Dead Redemption 2 (Imran Sarwar, 2018), allow players to wander around freely and tackle quests in various (non-linear) orders.
About Zelda, Gamespot said:
“Never had a game so open-ended, nonlinear, and liberating been released for the mainstream market, and Nintendo of America was downright concerned that it would go right over the public’s head. Thus, it included a toll-free number that stumped players could call to have a genuine Nintendo employee talk them through any of the game’s many enigmas. Soon after the game’s release, Nintendo’s phone lines were deluged with calls, forcing it to establish gaming’s first major pay hint-service. The Nintendo Game Counselors were then born, and the rest is history.“
There is however no clear consensus on the definition of an open world. I believe this is because there’s a spectrum: non-linearity and linearity can be combined in various ways and to various degrees…
Non-linear storytelling: variation a game offers in terms of completing challenges in different orders, and how much effect this has on changes in the story: in how many directions can the story branch out?
Bonus play: additional variation a game offers by offering optional gameplay. For example by rewarding the players extra points for finding hidden objects or executing more difficult to perform moves.
Sandbox elements: ways in which the game allows the players to leave their artistic (incl. social, theatrical) marks.
I will address the following forms of non-linearity in the next chapters:
Adaptive audio: how the audio transforms along with the player’s behaviour and other variables in the game.
Procedural generation: the organic feeling the landscapes express through how they are generated.
NPC AI (incl. enemy AI): how do the non-playable characters interact with the players?
these techniques differ from machine learning and deep learning. Algorithms used are for example path-finding and and decision tree learning.
Here’s a diagram that roughly recaps the distinctions in non-linearity made so far:
3.0 Non-Linear Level Design
3.1 Non-Linear Level Design – Fractals
This section is under construction
Math is not me expertise (understatement!), but during this research, I was once again reminded how everything, at one point, leads back to math. So without getting too technical, let’s have a peak at the geometric art concept calle Fractal Art, as it helps us philosophise about the difference between linearity and non-linearity.
“Fractal geometry reveals that some of the most austerely formal chapters of mathematics had a hidden face: a world of beauty unsuspected until now.” – Benoit Mandelbrot (The Fractal Geometry of Nature, 1982)
The concept of fractal art is to execute a repetitive (iterative) algorithm (set of instructions) that feed some parts of its output back into its input: the instructions create a drawing and subsequently sample a part of that drawing to create the next drawing. The feedback loop within this process is recursive: it creates that typical fractal-style self-simularity (organic growth-like structuring) between elements within the structure.
Fractals grow in different want, some seem to only scale, other seem to scale and evolve. Fractals that “scale” find their strength in predictability, rhythm, linearity. Fractals that “evolve” also express a level of predictability, but they captivate us with their unpredictability, which replaces rhythm with a sense of organic shapeshifting.
3.2 Non-Linear Level Design Procedular Generativity
This section is under construction
Procedular algorithms (algorithms for controlled randomness) can be used to add a sense of variation to to creative processes (organic, realistic, or otherwise), which is aesthetically pleasing, and also reduces the need for time-costing manual labour (such as creating huge forests or underwater sea worlds). Procedural generation and can (help) shape micro, meso and macro elements of games: textures, objects, levels, behaviours, etc.
Procedural level generation was first used in the Apple II game Beneath Apple Manor (1978), which also happened to be the first commercially available role playing game (RPG). Randomisation can add an element of surprise to a game, giving it a high ‘replay value’.
Playstation 4 megahit Horizon Zero Dawn used a procedural placement system to generate rich natural environments. Level designers were able to adjust the generative system and save its output, after which they could make manual changes to the landscape, or adjust the system once again.
But which procedural algorithms do elements of nature and the cosmos use to grow and interact? A theoretical, computational research field that deals with this question is automata theory (Wikipedia), which is – for example – used to create reductionist (fundamental) models/simulations of self-replication and artificial life (Wikipedia). A famous example is John Conway’s ‘Game of Life’ (Wikipedia), a zero-player game (Wikipedia). Below is a video of Conway himself explaining what it is:
– “GPU-Based Run-Time Procedular Placement in Horizon Zero Dawn” (Lecture by Jaap van Muijden at Digital Dragons 2017 on Youtube)
– “Fractals are typically not self-similar” (3Blue1Brown on Youtube)
4.0 Non-Linear Sound Design
4.1 Non-Linear Sound Design – Adaptive Music
In many contemporary high-end games, the music seamlessly adapts itself to the player’s current situation, in terms of: health, location, activities within that location (relaxed vs threatening), room acoustics (small toilet vs big church), etc. This kind of adaptive music can work in (at least) three ways:
A. Horizontally: by chopping a composition into loops that can be re-sequenced.
B. Vertically: by adding or removing certain instruments from the aforementioned loops, also known as re-orchestration.
C. Modulation.: by adding direct flexibility to musical parameters (timbre, intensity, speed, etc.) of certain instruments.
The first game to use a combined horizontal-vertical adaptive music system, with smooth transitions, was the point-and-click adventure Monkey Island 2 (1991).
The sound design of VR-compatible Tetris Effect (Monstars & Resonair, 2018) pairs every single action (movement, rotation, etc.) with a corresponding sound effect, thereby blurring the lines between sound effects and music. GlitchHiker (2011) — a game I co-created at Global Game Jam — also used interactive music as SFX, as the various instrumental layers would respond to the gameplay by glitching.
–”Monkey Island 2 Special Edition – imuse comparison” –Youtube video by FluffyQuack
– “GlitchHiker: the game that was programmed to die” –Article by PCGamesN
4.2 Non-Linear Sound Design – Musical Improvisation as a Byproduct
Otocky (SEDIC, 1987) was a very musical game for the Nintendo Famicom (NES) game console. In this side-scrolling-shoot-em-up the player was able to shoot in 8 directions, each of which represented one of the 7 notes of a (western music) minor/major scale, plus the next octave’s root note (e.g. DO-re-mi-fa-sol-la-si-DO. This meant that you would simultaneously shoot at targets and improvise melodies on top of the background music. Your melodies would automatically be rhythmically quantised: the timing of each note would be slightly nudged to align it with the rhythmic grid of the background music.
4.3 Non-Linear Sound Design – Generative, Procedular Music and A.I.
In Portal 2 (Gabe Newell, Valve, 2011), every action (such as shooting) instantaneously generated a unique bit of music.
Recently the first tech demos of an AI-driven music system for the real-time generation of adaptive music were created by Melodrive (2018).
5.0 Branches of Music Games
5.1 Music Games – Pre-History
What about music games? A beautiful example is Mozart’s Dice Game from 1787, for which he wrote 272 measures to be arranged in any random order (non-linearly). In the video game world, most music games are neither non-linear nor improvisatory, we’ll get to the exceptions later. You can play scmb1’s version of Mozart’s Dice Game online at MIT Scratch.
In the next chapter, let’s check out the invention that might have kickstarted the whole music video game genre…
A Japanese invention from 1971. Initially it didn’t include a television screen — see Daisuke Inoue’s Juke 8 — but as karaoke spread across Asia in the 1980s, it transformed into the karaoke box, also known as KTV (karaoke television): private, sound-proof karaoke rooms with on-screen lyrics.
Although karaoke binds its players to relatively linear ruleset — singing the songs as accurately as possible — it does give players room to express themselves creatively; after all, a little improvisation or weirdness can add to the fun and social interaction.
In relatively recent Japanese karaoke systems, players can score skill-points, based on various elements of singing. A spin-off of this scoring system appeared in Karaoke Joysound (Hudson Soft, Xing, 2008) for Nintendo Wii. Playstation game Singstar (London Studio, 2004) visualised the melody as it approached, and gave its players a live-indication of their accuracy.
– “Monkey Island 2 Special Edition – imuse comparison” –Youtube video by FluffyQuack
– “GlitchHiker: the game that was programmed to die” –Article by PCGamesN
5.3 Rhythm Games – Call-and-Response
After Karaoke, Japan once again gave birth to an interactive music idea that would become hugely popular: rhythm games. Most commonly, the objective of these 1990’s games was to anticipate to rhythmic cues scrolling across the screen – thereby setting the mainstream rhythm game mechanic in stone. But in the early days, some games favoured a turn-based game mechanic, requesting the players to repeat bits of music (call-and-response).
Computer-controlled / electronic “tabletop” memory game Simon (Baer & Morrison, 1978) is probably the spiritual predecessor of call-and-response video games, as it already produced melodies. Found via ‘A History of Video Game Music‘ by Gamespot.
The first real rhythm (video) game, PaRappa the Rapper (NanaOn-Sha, 1996) for Playstation 1, did show a stream (sequence) of notes, but mainly challenged the players to repeat the teacher’s rhythms; to copy in a call-and-response way. Another thing that made PaRappa’s game mechanic quite unique, was its emphasis on improvisation: by playing extra notes in the rhythmic grid, the player could “beat” the teacher and get a chance to perform on the “cool mode” freestyle stage.
Jazz musician Bobby McFerrin played a melodic game (Youtube video) with his audience at the World Science Festival. This shows our intuitive (instinctive?) understanding of the pentatonic (ancient 5-note) musical scale. Could we translate this understanding to a video game?
5.4 Rhythm Games – Dance
In 1997, just one year after PaRappa came out, the rhythm virus spread from the Playstation to the Arcade Hall, thanks to Konami’s DJ simulator Beatmania (Yuichiro Sagawa, 1997). Players now had to bash buttons in sync with scrolling notes. 1998 marked another revolution: Dance Dance Revolution (DDR), the first rhythm game with a dance controller.
The game didn’t award its players with any extra points for improvising, yet that didn’t stop a ‘DDR freestyle’ scene from emerging. Freestylers are not focused on the game’s expert levels or extreme scores; they instead prefer to come up with new choreographies and/or improvisations. Thus DDR reached cult status… in my book at least.
Another arcade example is Pop’n Music (Bemani/Konami, 1998).
Later games, like Dance Central (Harmonix, 2010) and Dance Evolution / Dance Masters (Bemani/Konami, 2010), challenged the players to dance with their full bodies. Hand and foot gestures appear on the screen right before they have to be executed. These games use the Kinect 3D-sensor for full body tracking. DANCERUSH (Bemani/Konami, 2008) continued in that tradition.
Still a lot of games (Japanese arcade games) missing
5.5 Rhythm Games – Other
Many other rhythm games would follow: rail-shooter Rez (Tetsuya Mizuguchi, 2001), “traditional” Japanese drum game Taiko no Tatsujin (Bandai Namco, 2001), puzzle game Lumines (Tetsuya Mizuguchi, 2004), Guitar Hero (Harmonix, 2005), countless mobile games (see this Engadget article), motion controlled games like Wii Music (Kazumi Totaka, 2009) which included a Custom Jam mode for improv, “piano” game Chunithm (2015) and VR games like Beat Saber (2018).
Frequency (2001), for Playstation 2, featured a remix mode that allowed up to 4 players to improvise collaborative compositions online.
Vib-Ribbon (NanaOn-Sha, 1999) was the first rhythm game to feature an audio-based level generator; it could create new levels based on any audio CD you would insert.
5.6 Music Games controlled by Real Instruments
In recent years we’ve seen innovative games that can be controlled by improvising on real (acoustic or electronic) instruments, like Cello Fortress (Joost van Dongen, 2012): an action game in which a cellist defends a fortress from 4 gamepad-players who try to destroy it. And SoundSelf (2012): a psychedelic, therapeutic VR-compatible experience controlled by chanting (singing).
5.7 Music Education Games
Still focused on hitting the right note on the right cue, but somewhat less rhythm oriented are music-education games, as they allow you to practice on real instruments. Examples are Rocksmith (Ubisoft, 2011 – remastered 2014) for electric guitar and Synthesia (2006) for MIDI keyboard. Robert Nichols hacked Rocksmith to make it work with violin as well.
5.7 Sound Toys
Sound toys are arguably similar to sandbox games (having no fixed goals). Let’s look at various examples.
– SimTunes (Toshio Iwai, 1996)
– Electroplankton (indieszero & Nintendo, 2005)
– Physics-based Musyc (Fingerlab, 2013)
– And the more graphical remix/jam app Playground [App Store] (HLO S.A, 2017)
– Table-top inspired ReacTable (2005), which can be scaled large enough for multi-player improvisation.
– SimTunes composition ‘Abblahh’ by Dogman15 on Youtube
– Performance of ‘Electroplankton Opus 1-3 (Nintendo DS x 5)’ by AABattery on Youtube
– Video review of Musyc by Jarrid Pennypacker on Youtube
– björk’s official promo video of the Biophilia app suite (Youtube)
– Playground app promo video by Herrmutt Lobby on Youtube
– ‘The Making of Ninja Jamm’ video on the Ninja Tunes Youtube channel
– ‘ReacTable Trance live performance #2’ video by ReacTj’s Youtube channel
5.8 Playful Sequencers
Quite similar to the sound toys category, but a little less based on direct control, and a little more based on patterns and sequences….
– Sentris (Samantha Kalman / Timbre Interactive, 2014) is possibly the first real / advanced hybrid of a puzzle game and a sequencer (software to write and produce music with):
Non-game influenced sequencers have also become more playful and non-linear
– Stochastic sequencer SECTOR (Kymatica, 2014):
Multi-playhead sequencer Fugue Machine (Alexander Naut, 2015).
Cellular automata inspired sequencer JR Hexatone (Ilya Plavunov, 2009):
Playful yet feature-packed Max for Live device New Path (Studio The Zooo / Jerke van den Braak, 2017):
iPad app Gestrument (2018), a customisable interactive music engine/instrument that supports eye tracking, gestural movements, audio tracking, and more:
Neural Beatbox (Qosmo, Ltd., web-version 2020) uses AI to generate beats from sounds the user records.
– SECTOR tutorial on Youtube channel The Sound Test Room
– Sentris extended gameplay plus interview on Giant Bomb
– Jordan Rudess introducing JR Hexatone on his Youtube channel
– FUGUE Machine tutorial on Youtube channel The Sound Test Room
– Gestrument Pro demo on Youtube channel The Sound Test Room
– New Path 1.1 promo video on The Zooo’s Youtube channel
5.9 Augmented Music
Reality Jockey Ltd. pioneered augmented music in 2008 with their smartphone app RJDJ. It picks up microphone and sensor input, to generate music that reacts and adapts to the listener’s environment. RJDJ’s interactive compositions (scenes) are programmed in the visual programming language Pure Data. RJDJ is not on the market anymore, but its successor ‘Hear‘ is available on the App Store.
Check out this video of the amazing RJDJ scene “WorldQuantizer” by Roman Haefeli. It samples sounds and creates rhythmic loops in real-time. It’s likely that we will see the market for artistic augmented reality apps grow when mainstream AR products like Apple Glasses will hit the mainstream.
5.10 Interactive Musical Installations
When it comes to large (public) interactive installations, there are many examples feature interactive music or sound, like the simple but engaging Piano Stairs (VW & DDB, 2009), the body tracking avant-garde noises of Device Unknown (Sem Shimla, 2018), the immersive, brainwave (EEG) controlled lasers and sounds of On Your Wavelength (Marcus Lyall, 2015).
As an example of visual improvisation: in the floor-projected installation Healing Pool (2008), people can collectively manipulate the (regenerative) visual world.
But it’s rare to find an installation that combines collaborative interaction (social game design) + music/sound improvisation. Something that comes close is The Music Box Village (2018): a collection of architectural sound toys that can be used by artists and audience to jam with otherworldly (textural) sounds.
“The Music Box Village is created and built by New Orleans Airlift, an artist-driven non-profit organization, as one of a range of large-scale community art projects. (……) a place where play, imagination, experimentation, collaboration, community and hard work come together as artist-made, interactive “musical houses.”” –musicboxvillage.com
6.0 Proposed Challenges
From the information gathered by this blog post, I propose the following challenges for designing (non-linear) music games:
– Melody/harmony games inspired by rhythm games. Moving away from the on-screen rhythmic cues, towards actively listening (perhaps to randomly generated sequences), challenging players to respond with suitable notes and timings.
– Rhythm games with a stronger focus on synchronising with grooves, aiming to instil neural rhythmic synchronisation (rhythmic “feel”). Research has shown that almost none of the current rhythm games achieve this. Perhaps it can be achieved through continuous engagement with repetitive patterns.
– Music games that are equally as suitable for beginners, hardcore (high-score) players, and a third group: freestylers (DDR is a successful example of this).
– Edutainment games that teach players how musical elements like rhythm, harmony, melody actually work; so they can use this to become more musical and learn how to improvise.
– Masaya Matsuura’s idea: exploring post-human virtuosity, for example by giving supernatural influence over a virtual pianist. Perhaps related to inventing ways to conduct expressively, in stead of performing every single note. A form of cyborgian micro-conducting is present in Wii Music Jam (in the shape of hyper fast tremolo picking, for example).
– Music games that stimulate the players to improvise more, for example through an intelligent system judges how well players improvise (this is already implemented in Parappa 2).
– Music games in which computers improvise along with the player, complementing the player. Together, or turn-based (with a rhythmic clock).
Apple’s ‘Drummer’ in GarageBand for iPhone
– Music games with endless variations of music by implementing procedural (generative/algorithmic) music systems.
– Music games revolving around emergent gameplay, by for example letting players build their own instruments by using physics-based playgrounds. Or by navigating through unrelated genres in crazy but coherent ways.
– Music games to improve social contact (incl. games for the elderly, like the Tovertafel).
– Music games crossed-over with sports games. And rhythm games connected to fitness, for example by using weights as controllers.
– Music games embracing unrestrained body movement, about losing control in movement, chaos theories. Primal playful behaviour. See also: contact improvisation.
– Music game elements in serious electronic instruments.
– Music games integrated in architecture / public space / social environments.
– Music games suitable for large groups of people, in which individual expressivity and social awareness are both continuously respected.
– Open source music games allowing for co-design, or other co-design methods based allowing co-creators to build upon certain design pillars.