JOE PALCA, host:
Next up, we're talking about a show opening next week. It features a drama-filled story, lots of music and ambitious technology designed for the stage. No, not "Spider-Man." It's "Death and the Powers," a new opera by composer and inventor Tod Machover.
The opera tells the story of a wealthy businessman who downloads his memories and his personality into technology and leaves the world behind. For the show, Machover and his team at MIT's Media Lab designed a high-tech musical stage set. The show even has singing, dancing robots, or opera-bots.
Machover doesn't just create opera, though. The technology for the videogame "Guitar Hero" came out of his lab. And right now, he's also working on ways to use music to detect Alzheimer's disease.
So if you want to talk about music and the future of music and some crazy, interesting ways to make music, give us a call: 800-989-8255. That's 800-989-TALK. Or tweet us @scifri.
Tod Machover is a professor of music and media at MIT.
And thanks for joining us.
Dr. TOD MACHOVER (Professor of Music and Media, MIT; Composer and Creative Director, "Death and the Powers"): Great to be here.
PALCA: And when - what's this about? Why do you need to include robots and artificial type of music into an opera?
Dr. MACHOVER: Well, I guess, it goes back a ways. I have always loved music, grew up as a cellist. And I grew up in a house where my mom is a pianist and a very creative music teacher. And my dad is an engineer, and one of the people who started the field of computer graphics.
So I guess one of the reasons I combined these things is I sort of came by it naturally. But, you know, I grew up playing classical music and then rock music in high school, and started imagining blending of all kinds of different sounds that weren't easy to make in the real world. And then when I was a young teenager and "Sergeant Pepper's" came out, The Beatles album, in 1967...
PALCA: Dr. Machover - Tod Machover...
Dr. MACHOVER: Yeah.
PALCA: I started with this long question, and I had a short amount of time, so we're going to have to take a brief - a short break. But you'll stay with us, right?
Dr. MACHOVER: I sure will.
PALCA: Okay. Stay with us. We're going to talk more about robotic operas in just a moment. We'll be right back.
(Soundbite of music)
PALCA: This is SCIENCE FRIDAY, from NPR.
(Soundbite of music)
PALCA: This is SCIENCE FRIDAY. I'm Joe Palca. We're talking this hour about a new, high-tech opera called "Death and the Powers." If you'd like to join the conversation, call us at 800-989-8255. My guest is Tod Machover. He's a professor of music and media at MIT. He's the director of the Hyperinstruments and Opera of the Future Groups at the MIT Media Lab.
And you were just starting to say about your own interest in music, and I want to tie it, if I can, to this new production that you've got. And where do those come together? You obviously had a background in music, so it was important to you. But how do you get to this point?
Dr. MACHOVER: Yeah. So I was always interested in combining the live expressive, spontaneous performance that musicians are so good at. And if you think about something like "Sergeant Pepper's," which came out when I was a kid, which was the first album The Beatles made that could only be done in the studio, I thought, ever since that time, how can you take that beautiful, complex, rich sound and make it possible for people to play that live? Because that was an album that was designed - you could never play it live.
PALCA: Mm-hmm.
Dr. MACHOVER: So over the years, I've made what we call Hyperinstruments, which are instruments that actually know the way they're being played. They know the expression. They know what you're emphasizing. And then when you play the instrument, you can add all kinds of different layers.
And with this opera, which we premiered about six months ago in Monaco and is coming to Boston in - a week from today, our idea was how to make the entire stage come alive. Because when you think about stages and theater and technology, first of all, the experience we usually have in live theater these days is very, very far from what we experience in a movie theater or with a videogame. It feels quite old-fashioned.
And, actually, if you think of what technology does in, let's say, arena rock shows - you know, these days if you're a touring rock band, you kind of have to go to a very big venue with 10,000 or 20,000 people for it to be economically viable.
And then you've got these huge loudspeakers that push sound out at the audience. It's not very pleasant. And you usually have giant screens that blow up the performers or maybe have some kind of music video playing while the performers are playing.
And my view is that the more technology in live performance goes in that direction, the more the live performers shrink. I don't feel that they get bigger, and I don't feel you're closer to them. I feel that they kind of look like ants running around.
So in this show, we wanted to give tools to the performers so that the entire stage would feel like an extension of a human being. The stage could come alive. The technology could feel as human as possible, and the audience could feel as close to these human beings as possible.
And that's - and we invented a story around it, about a man who wants to live forever and downloads himself into his environment. And, in fact, the stage, as much as we can, does come alive and is really the main character of this opera.
PALCA: Wow. Well, let's hear it - let's hear a little segment from it. Maybe after we hear it, you can tell us what's going on.
Dr. MACHOVER: Good.
(Soundbite of opera, "Death and the Powers")
Mr. JAMES MADDALENA (Actor): (as Simon Powers) See you later.
(Soundbite of music)
PALCA: So are those the opera-bots that we're hearing?
Dr. MACHOVER: Yes. That's the combination of things. First, we heard the main character, whose name is Simon Powers. He's rich, successful, powerful, late 60s. And he wants to keep everything about himself in the world. He's actually kind of tired of the world. He wants to check out, but he figures out a way to build something called the system. And he downloads himself into his environment. It's the exact moment where he turns the system on, says see you later, and then the system starts to take off.
There are 12 robots on stage. They're called opera-bots. And these robots are kind of part of his experiment to extend his life, to extend human beings, but they're an intermediary. The system is actually all the walls, all the furniture, everything else.
And the robots, at this moment in the opera, start to scurry around the stage. They all look kind of like chess pieces. They're pretty big. They're about four feet high, and they can grow. They extend to about seven feet high. They're -they have a triangular base and a triangular head. They kind of look like something Apple might make. They're sort of white Plexiglas on the top and on the bottom, with translucent Plexiglas rods connecting the bottom to the top. And they're really quite good dancers.
(Soundbite of laughter)
Dr. MACHOVER: They have beautiful wheels. They kind of glide around stage. They light up. The funny thing is that they don't look at all like human beings, but they convey a huge amount of emotion. They have eyes and ears, so they react to the characters on stage. And they do make a variety of sound.
Actually, at the beginning of the opera and the very end, they sing. But for the rest of the opera, they just send out sound that makes - it's electronic sound. It blends with what the orchestra's playing. And what we just heard was - it was purely electronic sound coming out of the opera-bots.
PALCA: So are they pre-programmed, or are they responding? I mean, if somebody walks on and makes a mistake, will they respond to that appropriately?
(Soundbite of laughter)
Dr. MACHOVER: The bigger question is what happens if the robot makes a mistake?
(Soundbite of laughter)
PALCA: That's impossible.
Dr. MACHOVER: That could be...
PALCA: Right.
Dr. MACHOVER: They're very, very fast. It's very pretty dangerous.
PALCA: Mm-hmm. I see.
Dr. MACHOVER: They act in a few different ways. So, at certain points, they're own their own. So you tell them to go from point A to point B on stage with a certain kind of swagger, and they'll move on their own and make sure they don't hit something. At other times, you can tell them to go from point A to point B, but you might plug in something that someone else is doing, maybe what a singer is doing or maybe what one of the instruments in the orchestra is doing, so you could imagine the robot gliding along. And then, depending on the tremolo of a string instrument, the head might bob in a certain way or it might twist from side to side.
And then, at other times, we have people behind stage, basically with video game controllers, grabbing each of the robots. This happens when the choreography has to be very, very precise, so they have to line up in a geometry. Or if they're going astray, you want somebody is there to pull it back.
PALCA: Well, there's a lot of people who want to ask questions. And so I think we better invite some of our listeners in to the conversation. Let's go first to Ryan(ph) in Pensacola. Ryan, welcome to SCIENCE FRIDAY. You're on the air.
RYAN (Caller): Hey, thank you very much.
PALCA: Sure.
RYAN: I appreciate it. I have to say I'm a first time caller but avid video gamer. So I very appreciate the work you're doing out there.
Prof. MACHOVER: Oh, great.
RYAN: I appreciate that a lot.
PALCA: Mm-hmm.
RYAN: Anyhoo, I had a question about - are you familiar with something called acoustic levitation?
Prof. MACHOVER: Oh. Actually, I do not know what that is.
RYAN: OK. I honestly just saw it demoed on the History Channel not too long ago.
Prof. MACHOVER: Yeah.
RYAN: Very interesting. In which the sound can actually be used at very high frequencies to make objects levitate, essentially creating an antigravity machine. Now I only saw it demonstrated with like, you know, a 10-gram ping pong ball. But is something like that feasible in the audible spectrum of music?
PALCA: Ryan, good question. Thanks for the call.
Prof. MACHOVER: It's a great question, and I honestly do not know the answer. I do know that one thing that we do in this "Death and the Powers" opera is we do a lot of acoustical tricks and illusions. And one of the reasons we do that is because we blend a normal orchestra with all kinds of new sounds. And to blend them - not only do we have to put them through very sophisticated loudspeakers, but unlike the rock concert I talked about before, where you're blasting sound at people in a very unpleasant way, we have about 150 loudspeakers around each of the very big theaters that we play in.
And we use a technique that we developed at the MIT Media Lab. It's a new version of ambisonics, which actually can calculate how to take any sound, how to split it up into space, how to make it come off the stage and feel like it's anywhere in the room, and to make it soft but also very present.
And one actual trick - it doesn't levitate anything, but this is actually pretty cool. One of my students, a wonderful undergraduate at MIT named Ben Bloomberg, has been working on a technique called Wave Front Synthesis. And what that does is - it's actually the world's - I think it's the world's longest, thinnest speaker, which is on the very front of the stage. It extends the entire length of the stage. So it's about 40 feet. And it's very thin, so it's only about six inches high. And this long loudspeaker generates one continuous waveform.
And the trick with this waveform is that it allows a sound to give the illusion that it's coming from an incredibly precise place anywhere on the stage.
You know, if you wanted like, let's say I have a prop, like a chair...
PALCA: Mm-hmm.
Prof. MACHOVER: ...and it's on the stage, and I want it to sound like the chair is creaking or making a sound, if I put a loudspeaker on that chair, the loudspeaker resonates in a way that it's imprecise as to where it is. You could never give the illusion that it's coming from there. Whereas, if I have somebody talk on the stage, you'd know it came from them.
With this Wave Front Synthesis, I can calculate the wave so that it makes the sound appear to come from anywhere on stage. So for instance, a robot which is racing around the stage, the sound absolutely travels with it.
The same is true with my singers. You know, if you go to a Broadway show these days, every body is amplified. And they usually have a pretty good sound system, but it's just a normal stereo sound system. So you amplify the singer, the sound comes out of a bunch of loudspeakers, which aren't anywhere near the singer. And, you know, you can hear the singer or the speaker louder, but it doesn't really sound like it's coming from their body. That's one reason why it's sort of detached.
PALCA: Mm-hmm.
Prof. MACHOVER: But with this Wave Front Synthesis, I have a sensor on the singer. I know where they are. I amplify them. And then with the Wave Front, the sound that comes back absolutely sounds like it's coming, not only from their body, but from their mouth.
So it - that's - I'm not levitating them as far as I know. But I am definitely - we're definitely able to place the sound, like a miracle, anywhere we want to on the stage. That's kind of a nice thing.
PALCA: It really is. I mean, it's - as you say, it's transformational because you get such a different sense and, I imagine, so much more a sense of being present when you hear the sound that way.
Prof. MACHOVER: I think you do. And one of the nice things about a very sophisticated audio setup is that it doesn't have to be overpowering. It can, kind of, sneak up on you or, you know, the other way of thinking about it is it can kind of invite you in. It can make it feel like you're being invited to step up on stage with the performers, with these robots.
You know, we have these giant walls that are part of Simon Powers' house, which also come alive and move around. A gigantic chandelier, which looks like kind of - well, it looks kind of an odd chandelier but it turns out it's a big robotic music instrument. And the audience can feel like they're really inside the chandelier or inside the main character's voice. And that's, I think that's kind of a new thing.
PALCA: Cool. Let's take another caller now and go to Mike(ph) in Akron, New York. Is that Akron, New York, Mike?
MIKE (Caller): Yes, it is.
PALCA: OK. Welcome to the program.
MIKE: I hope this isn't too much off topic. In the introduction, you talked about that your host had a way to test for Alzheimer's with music. Did I get that correct?
PALCA: No, no. You did get that right. And I was going to ask that as well. Todd Machover, What's that about?
Prof. MACHOVER: Yeah. So I guess the major motivation I've had, you know, since I was a kid was to make music as much a central part of people's lives as possible because I love music. Many people love music. Music is entertaining. But music gets under our skin and affects us very, very deeply. It's a very powerful and transformational experience as well as being enjoyable.
So it turns out that this isn't just an impression. Research, both at MIT, at Harvard, at many places these days, is showing that music actually has a very, very powerful physical and mental effect on people and that we can begin to study what that is, not only to understand it intellectually, but to make musical experiences that can actually change somebody's state of being, hopefully, in a positive way.
So for instance, we know now that most people with Alzheimer's disease - at the later stages of Alzheimer's disease, music is usually the very, very last thing that people will still respond to. You can find somebody who can't recognize themselves in a mirror anymore, but you could almost always find a shard of music that that person will recognize and immediately come to life around, maybe stand up and start singing that music with the words even though much of the memory is gone. And so little by little, we can bring back people's memories around that music.
What we've started to do at the media lab is the opposite. One of my students who just finished a doctorate and is now a post-doc is named Adam Boulanger, and he's one of the experts in what we call music, mind and health. And what he has been studying is whether there are a set of music activities which can actually give us an idea about the state of someone's memory just by playing with these activities.
So there are two kinds of things we're developing, one are a set of composing tools. We actually have software called Hyperscore, which allows you to draw music with lines and color, so anybody can do it, but you can make quite sophisticated music.
And then we have another set of activities, which are kind of like playing with music. In fact, one of them that seems to be most effective is like that old game Concentration with cards, where you turn over a card and you have to match that card with another one. You have to remember in space where these cards are.
Well, it turns out that music, both because people love it and want to spend lots of time around music, people will go back and - over and over to a good music activity. Also, because music seems to use more different parts of our brain than any other activity, it seems to connect different parts of our brain, it's a very good way of assessing the state of somebody's mental abilities.
PALCA: Todd, let's - I'm sorry, I have to interrupt just for a second to remind people that this is - I'm Joe Palca and this is SCIENCE FRIDAY from NPR.
Sorry, go ahead and finish the thought.
Prof. MACHOVER: Absolutely, Joe. Yes. So just to say that Adam has developed these techniques, where you can play a set of musical games, you can make your own music, you can play this kind of concentration game, where bits of music are hidden, not under playing cards but on the computer, and you can match which sound goes with which. And it turns out that if you do that, over and over again, it's quite fun.
But we also - in doing it, we can develop data about the state of people's memory. And it looks like - yeah, this is in clinical trials right now at McLean Hospital outside of Boston.
PALCA: OK. I'm sorry - it's a fantastically interesting topic, and I know we could go on, but that's - I'm afraid we've run out of time.
Prof. MACHOVER: OK.
PALCA: So people are just going to have to go to your website and find out and stay tuned, and we'll hear more from you, I'm sure. Thanks very much.
Prof. MACHOVER: Thank you.
PALCA: Todd Machover is a professor of music and media at MIT. Transcript provided by NPR, Copyright NPR.