ATMI 2007 Keynote
Extending the Musical Experience--From the Physical to
the
Digital… and Back
Dr. Gil Weinberg
Georgia Institute of Technology
It is widely perceived that the computer has enriched and advanced the art form of music. Digital technology brought new palettes of sounds, composition techniques, and production methods; Innovations in digital compression and distribution changed music consumption and listening practices; for performers, novel musical instruments and controllers have been developed based on a variety of sensing, interaction, and mapping approaches. But after more than two decades of research in computer music, a fundamental question must be asked – has digital technology truly innovated and enriched the expressive, emotional, and creative core of the musical experience? It is not clear that the answer to this question is as positive as we, music technologists, would like to think.
During the last ten years, inspired and motivated by the prospect of innovating the core of the musical experience, I have explored a number of research directions in which digital technology bears the promise of revolutionizing the medium. The research directions identified – gestural expression, collaborative networks, and constructionist learning – may lead to musical experiences that cannot be facilitated by traditional means. The first direction builds on the notion that through novel sensing and mapping techniques, new expressive musical gestures can be discovered that are not supported by current acoustic instruments. Such gestures, unconstrained by the physical limitation of acoustic sound production, can provide infinite possibilities for expressive and creative musical experiences for novice as well as trained musicians. The second research direction utilizes the digital network in an effort to create new collaborative experiences, allowing players to take an active role in determining and influencing not only their own musical output but also that of their co-performers. By using the network to interdependently share and control musical materials in a group, musicians can combine their musical ideas into a constantly evolving collaborative musical activity that is novel and inspiring. The third research direction utilizes constructionist learning, which bears the promise of revolutionizing music education by providing hands-on access to programmable music making. Through interaction with physical computational objects, learners can construct personally meaningful musical artifacts that enhance and deepen their learning.
While facilitating novel musical experiences that cannot be achieved by traditional means, the digital nature of these research directions often led to flat and inanimate speaker-generated sound, hampering the physical richness and visual expression of acoustic music. In my current work, therefore, I attempt to combine the benefits of digital computation and acoustic richness, by exploring the concept of “robotic musicianship.” I define this concept as a combination of musical, perceptual, and social skills with the capacity to produce rich acoustic responses in a physical and visual manner. The robotic musicianship project aims to combine human creativity, emotion, and aesthetic judgment with algorithmic computational capabilities, allowing human and robotic players to cooperate and build off one another’s ideas. A perceptual and improvisatory robot can best facilitate such interactions by bringing the computer into the physical world both acoustically and visually.
I will present six projects that represent this journey that were initiated by extending acoustic music with digital technology and ended by enhancing digital music through acoustical means. The first project in the journey, titled Musical Playpen, was developed in an effort to explore new musical gestures that can provide young children with meaningful musical experiences. The instrument sensed children's motion inside a playpen filled with plastic balls and mapped their gestures to musical output. This allowed “players” to use familiar play activities while exploring the musical consequences of their actions. In an effort to further investigate constructionist learning techniques, I developed the Musical Fireflies – palm-sized digital musical instruments that were designed to physically introduce mathematical concepts in music without requiring prior knowledge of music theory or instruction. Through simple controllers, the “fireflies” allowed players to input rhythmic patterns, embellish them in real-time, synchronize patterns, and trade instrument sounds. Since interaction with other players increased the richness and complexity of the experience, the Musical Fireflies also motivated collaboration and social play. This led to a further exploration of networked musical interactions with the Squeezables, a collaborative musical instrument that allowed a group of players to perform and improvise musical compositions using a set of squeezing and pulling gestures. The instrument, comprised of six squeezable and retractable gel balls mounted on a small podium, was designed to provide an alternative to asynchronous and sequential interactions with discrete musical controllers by allowing multiple channels of collaborative simultaneous input. The continuous nature of the interaction with the Squeezables led to the development of the Beatbugs, which featured more discrete and sequential group collaboration. These hand-held percussive instruments facilitated the creation, manipulation, and sharing of rhythmic motifs through a simple percussive interface. When multiple Beatbugs were connected in a network, players could form large-scale collaborative compositions by interdependently sharing and developing one another’s musical ideas. The Beatbugs were also featured in a following project, iltur, which was designed to explore a novel method of interaction between expert and novice musicians. Here, Beatbug players recorded live musical input from trained musicians and responded by transforming the recorded material in real-time, creating motif-and-variation call-and-response routines on the fly. The most recent stop in this journey is a perceptual robotic percussionist named Haile, which can listen to live players, analyze their music in real-time, and use the product of this analysis to play back in an improvisational manner. It is designed to combine the benefits of computational power and algorithmic music with the richness, visual interactivity, and expression of acoustic playing. When collaborating with live players, Haile is designed to inspire players to interact with it in novel manners that combine human expression and robotic algorithms. Similar to the projects described previously, the robotic percussionist project was designed to address the same open question I established more than ten years ago. Like the other projects, it generated a set of new open questions that may assist in defining the future of music.
Biographical Note
Gil Weinberg is the Director of Music Technology at Georgia Institute of Technology, where he founded the Master of Science in Music Technology program and is currently in the process of establishing a new research center in this field. He holds professorship positions both in the Music Department and the College of Computation. Dr. Weinberg received his M.S. and Ph.D. degrees in Media Arts and Sciences from Massachusetts Institute of Technology, after co-founding and holding a number of positions in music and media software industry in his home country of Israel.
In his academic work Weinberg attempts to expand musical expression, creativity, and learning through meaningful applications of technology. His research interests include new instruments for musical expression, musical networks, machine and robotic musicianship, sonification, and music education. Weinberg’s music has been featured in festivals and concerts such as Ars Electronica, SIGGRAPH, ICMC, and NIME, and with orchestras such as Deutsches Symphonie-Orchester Berlin, the National Irish Symphony Orchestra, and the Scottish BBC Symphony. He has published more than 30 peer-reviewed papers in publications such as Computer Music Journal (MIT Press), Leonardo Music Journal (MIT Press), Organized Sound (Cambridge University Press), and Personal Technologies (Springer Verlag), among others. His interactive musical installations, notably the Beatbugs and the Musical Playpen, have been presented in museums such as the Smithsonian Museum, Cooper-Hewitt Museum, and Boston Children's Museum. With his perceptual robotic percussionist, Haile, he has traveled around the world, featuring more then ten concerts in Asia, Europe, and North America. As a result of this project, Weinberg has recently been recommended by National Science Foundation to receive a $450,000 grant to continue to explore the concepts of machine and robotic musicianship. Based on his most recent project – a set of musical applications for cell phones – he is currently in the process of establishing a new company that will attempt to bring innovative research in music technology to the general public.