Rehearsing interactive computer music for classical performers

I've been on hiatus from the blog, as I've been organizing home parties, inviting close friends for dinners, house-guests, the usual holiday fair plus the kids out of school, and the NYC blizzard which doesn't help :)   I'm also working on a MaxMSP program for cellist Joel Krosnick this week, for him to perform Ralph Shapey's Solo, Duo, Trio.  Joel approached me earlier in the year, since he wanted to perform this work interactively.  The piece is written for solo cello, over-dubbing twice.  He has been performing it with the recording of himself playing the 1st and 2nd parts, but Shapey explicitly said that he wanted this to be done live.

Since the piece is entirely written, it probably is a good candidate to "score follow" to make this piece interactive.  But score following, as I described in my earlier post, to me is rather unmusical performance wise.  For the computer to "follow" all the notes and beats in order to accompany a human player seems quite unnecessary and too complicated since the human performer can perfectly "follow" the sound itself than following the score, which is just the representation of what is actually happening with the "flow" of the music.  So I'm doing something a lot simpler, and it is not too difficult to do technically.  But rather, I'm spending a lot of time on rehearsal schemes.

Since Joel will be alone on stage, and there will be no human assistance on or off the stage, AND he has no hands so to speak--he is playing the cello--he needs complete autonomy.   That goes without saying for the performance patch, but what I'm trying to do is for him not to have to use both hands to touch the computer even during the rehearsing of the piece.  He has his instrument on his left hand as I do my violin, so the only possibility to touch the computer is the right hand, or even one finger of the right hand while he is still holding the bow.   I'm trying to make the system that's versatile as possible for him that only thing he will have to do is just to type a few keys to rehearse, not even scroll the mouse, drag and click the mouse or anything like that.  While I was at IRCAM this summer, the first thing I asked the team to do was to consider me hand-less; many computer programs require the operator to use two hands or type, and that's not good enough for a string player interacting with a computer, not to mention on stage, but also during the rehearsal. I think.

Since I primarily program for myself up until now, I don't have to be so caring about the performer's needs and I can put up with my own un-elegant programming :)  But this time around, and for my new piece for the Cassatt Quartet, I am learning to take more care of the interface that musicians, who are not used to operating interactive systems, will be able to use it with ease.   The system I'm working on, Joel could rehearse from where ever in the piece he wants with a stroke of a key or two, without having to put down his bow.   I'm trying to write the Max patch so that there will be no scene where he needs two hands to touch the computer.  Well we will see if it works!  :)

Visualizing Music vs. Physical Competence

As I mentioned, I just received a 2010 Fromm Commission to write for a quartet and interactive computer.  I have been thinking for quite a while, how to go about it.   When I compose, I think I tend to visualize the performance even before the piece is written.  That is to say I actually see the quartet on stage, with a computer on the side, and start listening (in my imagination) what and how they are playing.  It might seem strange, since the piece is not yet written down on paper.   But for me, when I have the clear visualization on how they are sounding, the rest, putting down notes etc. is very fast and easy.  On the other hand, if I start from writing down notes themselves before I am able to visualize the performance, which I also have done in the past, the compositional process takes a lot longer and not necessarily with the best results.  I think I'm trying to listen to the musical logic or the "flow" of music which I described before, or see how the "Magic Carpet" is flying, even before I put down the notes.

My proposal stated that I would write the work so that the quartet is completely autonomous on stage, without the need for the 3rd person or assistant to be "performing" the computer.  At the moment, I'm "visualizing" how this is going to be done.  I am somewhat fatigued by the conventional "score following", which happens a lot in computer music.  For me, making the computer follow a score is not very interesting; why would I want to do that, instead of a having a real person to do the same?  Instead I would be more interested in creating a behavior itself, and having computer interpret musical expression, creating more symbiotic relationship with the human players.

When I'm doing this visualization of music, so to speak, I often am rather physically decapacitated.  I would be loading a dishwasher, (which seems to be a good time for visualization or imagining my musical schemes) my hands are stopped in mid-air holding dishes, water running in the sink, and I'm as if the time has stopped, standing there frozen.  It must be very funny to witness this, as my daughter who found me in this state and laughed, "Mommy, you stopped!"   I would be cutting something like a broccoli, a knife in my hand and I'm stopped mid-way through the vegetable.   If I do this "visualization" work on the subway, I miss my stops, take a wrong train, get out from a wrong exit; all kinds of real-life small catastrophes happen.

I have been curious why this happens to me, and it seems that I'm "slowing down" time in my head, listening to what I'm creating, almost like a slow-motion.  That seems to be the explanation to why I seem to lose track of time and focus in real-life.  There is a time-stretching going on inside my head which stops my real-life kinetic movements!   I'm also curious, when we day-dream, is this what we are doing, stretching time in our heads?

"Hopping" onto the next

Well, not exactly because today is my birthday, but I just had kind of a visual thought of "hopping on to the next"  :)

Yesterday at Juilliard I video-taped my students trying IRCAM's improvisation system called Omax, on a special stand-alone version created for me this summer.  That is to say, Omax is usually operated by an off-stage operator, or "computer performer" (see this online presentation on IRCAM page).  However, I wanted everything to be operated from my violin without assistance, using my various data alone without the 3rd person; I want a compete autonomy on stage.  So Benjamin Lévy of the Musical Representation team at IRCAM created this special stand-alone version for me this summer.

Omax is not "listening" to the behavior of the performer, but rather it analyzes incoming sounds,  then chooses and segments your sound on its own and plays back.  So, in a way, it is "listening" but the musical decisions are not truly made based on the continuous input; that is the role of the "computer performer".  The computer operator is the "ear", who can control many parameters in realtime and can "perform" with the player.   In my "stand-alone" version I used, to create my new piece called Viomax this summer which I premiered in NYC this October, I control some parameters on my own via pitch, loudness and bowing detections.

Even with this limitations, i.e., there isn't anyone "home" so to speak, a performer could create quite an interesting performance.   To attest to this, both my students and myself, can go on for quite a while without losing musical interest, ending up playing with Omax and create interesting results.  It could create quite an engaging musical performance scenarios on the spot, provided that the "live" person is quite accommodating to the Omax's behavior.   Since Omax plays back what you played, but perhaps not in the strict order you did and keeping the history of your improvisation in the memory, the "playback" could be compositionally interesting.  But by chance.    That means that it still could be NOT interesting.  The reason why us performers can go on despite of this, is that when Omax does something that is rather 'out of context' or musically not expected, a skilled performer/improvisor could adjust the trajectory of the music, correct and carve the path of past-present-future, or just to "Hop On" to the next on the fly, ending up making sense of the music at hand.  It still boils down to the human performer's musical will.   I look forward to continuing working with IRCAM on this, and to develop this paradigm in the future.

Both sides of the fence

This week, I'm finding myself in two opposite sides: I'm evaluating projects as a music juror and I'm writing a proposal for myself for a grant application.  This is not uncommon for musicians, but it makes you think.

When you apply for grants, I find it easier for those who read it, that the proposal is concrete and specific as possible.  As a juror, it is not easy to go through paragraphs of work description written in philosophical or metaphoric manner, or too vague, covering too wide a range of possibilities, or overly elaborate in manner.  In fact, when you see that kind of a proposal, the juror's task seems to become more like deciphering what is really being proposed there.

To save both sides the trouble, the best way is to read the application guidelines as carefully as possible, and try to find out first, if the grant or the venue is truly the match for your project and proposal.   Of course there are other reasons for people to apply: money, prestige or both.  But without the concrete purpose and the matching of the grantee and grantor, the chance of success isn't very good.  At least that is what I am seeing.

Now, I'm also on the other side, having to apply for a grant.  Sometimes, a particular grant is indeed, asking for something grander, vast in scale and long term.  So far my compositional projects have been quite limited to my own (violin), and I have had just a few big-scale proposals that went through, namely a work for Youth Symphony which received New York State Council on the Arts grant, and a Violin Concerto, my first orchestral work I wrote for myself using Subharmonics, which received Jerome Foundation grant.   These are projects that are bigger than one violin, but nevertheless, limited in scale, in terms of timing and scope.   For the first time I am having to think in longer term with different aspects, and it is a little overwhelming.

Some grantors maybe actively seeking for something grander in scale, rather than a composition of "usual fair".  I might have to go elaborate, philosophical, and metaphoric this time.

Birthing a Project, and Choosing the Tools

This week, I am doing some paperwork.  I am on a music jury reviewing some proposals which is due soon.  I am also finally getting to finish the progress report from my residency at IRCAM this summer, which is also due.  I am also at the moment, beginning stage of several compositions, including two commissions: a duo with cellist Joel Krosnick, and for the Cassatt String Quartet.   And, I am also working with several students at Juilliard on building their projects.   In another words, this is a week of "birthing" the projects.

I think composers approach a new project in different ways.  Sometimes, especially when you are starting out, you might feel overwhelmed, and the overwhelming feeling could get multiplied when it involves technology, as the possibilities are just too vast, as I mentioned in my earlier entry "Freedom and Limitations".   In one of these cases, I have suggested to some students who claims to be "stuck", in fact, to start writing the program notes first! :) Composers compose because they have desire to express, before they know which tools, motifs, the building blocks they will use.  Sometimes, it is useful to spell it out trying to explain to others what you are trying to express, which could help in fact, in choosing the tools you need.  It's like watching my children play; sometimes the tools are already in front of them, for example, a play-dough, and the tools themselves inspire them to create.  Sometimes the play-dough gets to be combined with something that happen to be lying around, such as wooden blocks or LEGO (yikes!), and become multi-media productions.

Sometimes they have something specific they want to make, and they seek the tools they need to make that happen.  And when the tools turn out to be unavailable, (such as scotch tapes this mom fiercely guard against being exploited!) they go to the end of the earth to find the substitute for their projects.  Or sometimes, when they only have sticks, leaves and stones in the field, they still manage to create a theater if they want to.   The will and desire to create, and functions that enable them, are most important; the tools are always of secondary importance.

Obituary: Yumiko Kano (鹿野祐美子）

My mother emailed me yesterday that she received a "Deferred New Year's card" from the parents of my best friend from Toho School named Yumiko Kano, that their daughter has passed away back in March.  In Japan we still have the tradition of sending New Year's card, which is to precisely arrive on January 1st, called "Nenga-jo" (年賀状).  However, when you have a death in the family, we send in the greetings in advance.  I was quite shocked that I didn't learn of her passing until now.

Yumiko was a composer/pianist.  As a young teenager she was a class ahead of me at Toho School, and was at the top of the ear training class.  Toho School's Music School for Children (like Juilliard' Pre-college) was where you get the most rigorous western musical education in Japan.  Yumiko was at the highest level which means that she could pretty much sight-read orchestra scores on the piano perfectly, transpose, improvise, solfage, analyze and compose in any style on the fly.  We became friends, and I have performed her compositions, when not many violin students were playing composition students' works. I was introduced to courses in harmony, orchestration, analysis that "normal" violin students wouldn't take at Toho School.

Part of the reason which put us together, was a rare group of composers who formed a "mountain climbing club" at Toho School!  I wanted to join the club, and they made it a condition that I perform their pieces :)   Imagine, these rather geeky group of composers had an odd member, the only violinist :)  I fondly remember our trips in high mountains, chased by approaching thunderstorms, cooking on mountain tops, sleeping in cabins together.  Yumiko taught me how to climb for a long time without much effort, the method I still use to climb up stairs.  You basically "roll" from one to the next step, without pausing, carrying your weight from one to the other.   It curiously has a lot in common with music and rhythm.  We spent hours discussing the difference between how Asians (Japanese) and westerners walk, and the kind of steps we take, influence the rhythmical interpretation.  We talked about shoes worn by westerners; the tip of their shoes are curved upward, where Japanese shoes are not, since Japanese don't walk the same way as the westerners.  Japanese were used to wearing sandals for a long time, and our clothing: kimono, didn't allow us to walk the same way for generations.  And we discussed how this traditional Japanese character might be influencing when Japanese perform western music.  I make use of our discussion today for my own performance and phrasing in music.

Yumiko and I performed a lot together; she was a formidable pianist as well.  She went on to study at the Conservatoire in Nice, France after Toho School.   She leaves behind 2 CDs published in Japan: "The Seasons of High Attitude Plants", and "Quatre Tableaux Féeriques"(4 pictures of Féeriques) for four-hand piano, which won her the 1st prize at the 2nd International Composition Competition for Piano Duo in 1992 in Japan.    These works are published from Ongaku-no Tomo ("Friends of Music"), the prime music publishing house in Japan.   The program include:  Quartre Tableaux Féeriques-Chansons de France pour les Petits Français, Fantasie sur les Thèmes des Alpes, Sept Vielles Chansons: Veilles Chansons et Rondes, accompanied by her advice notes on performance.  (Ongaku-no Tomo, No. 438610)

Yumiko taught at Toho school, and performed and taught widely in Japan.  This Amazon link is in Japanese unfortunately and it says it is out of stock, but says will be filled shortly.  Above is the CD of this work,  available on Amazon.  It is published from a Japanese company called Nami Records, which lists a contact info. The CD number is WWCC-7264.  The aforementioned "The Seasons of High Attitude Plants"is WWCC-7361.

Yumiko also published piano textbooks, which are quite amazing, including pieces like "Let's have fun with *mixed time signature*", "combined rhythm, 2 against 3, 3 vs. 4, etc", illustrating uniquely and clearly through her compositions, quite advanced musical concepts for children.  The scores are in two volumes, "Expressive Piano Lessons" book 1 and 2. (Also Ongaku-no Tomo, but I couldn't find it in the website. I have them)

Yumiko leaves behind her parents, after a long battle with breast cancer.  She fought very hard for several years, after it metastasized to her spine.  She researched all the radio-active hot springs and knew all the data, measurement that she needed to slow down her cancer.  The last time we spoke, she rather joyfully explained to me in length the locations of hot springs, which extended to Taiwan I think, and the radiation levels, how often, how long she should go into the hot spring etc.  It was very analytical and typical of Yumiko, that she took her condition quite scientifically and dealt with it as the best she could.

The news of her passing slowly is marking me as the end of an era in my life, although she passed away last March.   I owe her our countless hours of lively questioning and discussions, thinking and searching of music.   I miss her terribly, and I celebrate the life and contributions of Yumiko Kano.

Subconscious vs. Conscious Performance and Composition

I have always noticed when I have a performance, if I am completely focused, or "with it" so to speak in my head during the performance, the performance itself to the audience, isn't necessarily the best (!).

On the other hand, when I am slightly distracted because of the nervousness or other reasons, and not necessarily "with it" every single minute of the performance, the outcome to the audience seems better.

In the first scenario, I am personally very satisfied that I was "with it" perfectly and did everything I was supposed to do.  I assume the audience felt the same and were happy.  In the second scenario, often I'm a little disappointed with my own lack of concentration and not sure of the outcome, and assume that audience was able to tell my unfocussed-ness.

It isn't so.   I have found time and again, audience's responses are opposite to my own perception, at least to a certain degree.  It seems that when I am too "with it" there is some kind of area, a subconscious area of performance that isn't there.   I'm too close to the performance, and my focus and concentration, in fact limit subconscious freedom to take wing. On the other hand, even if I'm a bit space-out or unfocused or lose concentration (*if* I am technically well prepared) it seems to give that subconscious level of freedom in performance that audience appreciate.  It is very difficult to explain but I don't know how else to describe it.

Now, I'm wondering.  Is there a parallel to this in composition?   Improvisation, yes I believe.  But a written composition, planned and laid out, is there a room for composer to be working on the subconscious level, and if so, how would the listener respond?

"Flow Following" not "Score Following"

I just put up a YouTube video from one of the demonstrations I did in Berlin recently.  It's basically an "automatic accompanist" thing, which, musically doesn't interest me at all; why would you not want to use a human pianist?   I made this demo in the summer at IRCAM during my residency in Paris, just to illustrate that there maybe an alternative to the tradition of "score following" in computer music, where people have been developing sophisticated systems to detect pitch, beat, rhythm etc. in order for computer to "follow" the human player.  Although "score following" doesn't interest me as a musical tool in interactive music, I wanted to show that it is the "musical flow" which is more crucial to performance than notes and beats themselves.

When I thought about it, I just imagined how do we, in fact, performers "follow" the other player during performance.  We do listen to the pitch, beat etc., but we don't really try to match note by note without the "flow" of the music.  We are following the "flow" of music that notes and beats and the rest of it "fit" inside.

So my approach was, to create the "flow" using the bowing motion sensor, in a very "violin" way, by simply tracking a "sustaining" motion, which carves the "flow" of the phrases.  The result, with this very simple "one trick pony" approach, was surprisingly accurate.   Today people still ask "How are you tracking this?" even though I repeatedly say there is no "tracking" or "score following" involved.


In the MaxMSP program I have used some of the "common sense" of performance, such as setting the the minimum and maximum tempo.  No sane performer would slow down or speed up beyond the acceptable musical limits, for example :)   So no matter how I try to speed up or slow down using the bowing "sustaining" motion, there is a "limit".  In this video therefore, I am simply controlling the tempo within those limits.

The mathematics behind the system was created by Nicolas Rasamimanana of the Realtime Musical Interactions Team at IRCAM.  We worked together to come up with the correct calibration which was the most time consuming part.  Nicolas is also a violinist, so it was so easy for him to understand what I wanted.  When I first approached him with this idea, what he said was, "But we already have that function, although we never thought of using it this way!"  There is however, a hand-drawn calibration table I made inside the program where I listened and tried so many times to get the scaling just right.  And that really is user-specific.

For those who are in computer music, in this demo I am running MaxMSP inside Ableton LIVE's Max for LIVE, using API to simply control the tempo of the quantized MIDI piano sequence.    By the way, the piece I'm playing for this demo is "Eu Te Amo" by Tom Jobim and Chico Buarque.  It is a very soothing music perfect for me trying over and over again calibrating :)

Categorizing and Labeling

As a musician, a performer/composer who comes from a classical violinist background who is also composing, I face this almost everyday: labeling and categorizing of my identity as a musician.

I am not frustrated, but rather puzzled.  People have the need to put you into a "line" or a "box" where it can be "shopped" in stores.  A few weeks ago I posted about "Head Space" saying that I have known performer/composers who stopped performing other people's music.  I was thinking in particular about one person whom I looked to as kind of a model, although he isn't a violinist.   Yesterday I was told that this person HAD to stop performing other people's music in the 70s and 80s, since that was the only way to be taken seriously as a composer!!!!  WHAT. THE....

I am not sure if this is entirely true, but 3 decades later, if I am puzzled being asked, "So, you are a performer.  AND a composer, right?"  as if that is something extra-terrestrial (never mind Vivaldi, Corelli, Paganini and the gang) I am just wondering, WHEN in the world, this "categorizing" has become a norm, and being both performer and composer in classical music world has become a novelty; just in the 1900s?

If I were in charge of conservatory education, I think I would like to see that all performers are REQUIRED to compose and improvise, both.  You learn so much about performance by understanding composition, or the process of composition.   I am not saying that all performers need to be Ligeti or Boulez :)  But wouldn't it be so much more fun to liberate ourselves creatively?   On the other hand, "all composers must interpret and improvise" might not work, but still they are, as I hope, required to play at least one instrument.  I do think though, if a composer could be an instrumental performer at the level who can give a public performance, that might actually help their compositions.

Would it be possible to have a structure where performance and composition world can have more fluid relationship?

Performer / Composer

Today, I received another surprising and humbling news that I was awarded the Fromm Commission 2010.  I submitted the proposal specifically for composing for the Cassatt String Quartet, which I also posted about last month on their kids-friendly concert.  (I said "another" news, since in this astonishing one year, I not only received the IRCAM residency, but also the Guggenheim Fellowship)

I was revisiting my proposal to Fromm, which stated that I would compose an interactive piece for the quartet, which doesn't require a computer operator or assistant on stage or off-stage, but a stand-alone computer system that would work when they take it on tour.   I also stated that, as a violinist who has been composing for myself mainly and have been working with interactive systems, I would have intimate knowledge of the strings as well as interactive performance.

This time, I am embarking on a strictly "composer" role.  I have composed limited amount of pieces where I did not perform, but mostly, even I wrote for other instruments and orchestra, I was part of the ensemble or I was participating as a violinist.   This new work for the Cassatt, I am totally retired as a performer, letting others play my music without me participating, which makes me feel a little bit in a new territory.

I feel comforted that I have a huge advantage this time, since the first violinist of the Cassatt Quartet is Muneko Otani.  She studied with the same teacher with me, Toshiya Eto at Toho School in Japan.  (Eto was a student of Efrem Zimberlist at Curtis, and is the first Japanese violinist to give his Carnegie Hall recital.)  In terms of our violin playing, we are somehow of the "same breed" so to speak.  I know how she would change her bowing strokes, sound quality etc. because she was taught the same way I was.   I am formating my compositions this time not from motifs or materials, but from their sounds, her playing and her sound.  I guess that's a typical performer/composer's approach  :)

Mario Davidovsky, the only composition teacher I ever had, said, "You have to write for performers. If they like your piece they will kill themselves to play it, but if they don't, forget it!"