A Duet of Music and Computer Programming Brings a Concerto, and a Dream, to Life

Dr. Matt DeJongh laps up languages: computer, verbal, musical. So when it came time to begin his 2022–2023 sabbatical research project, there were many directions he could’ve taken. There was, however, one especially compelling aspiration DeJongh believed a computer could make a reality if it were programmed just right.

“My project stems from a very specific dream that I have, which is to be able to play a Mozart piano concerto,” the professor of computer science says. “In piano concertos you have a pianist playing and an orchestra accompanies them. It’s very beautiful, it’s very challenging, but in order to do this, you need to have an orchestra — and those are hard to come by if you’re just an amateur pianist.”

But DeJongh knows computers, and thought one could stand in for the orchestra, given the right instructions. The technical process is called music informatics.

“A computer is a tool for creating your own universe in some ways,” he says. “As long as you learn to speak its language, you can tell it what you want it to do, and it’ll do it — within its capabilities. That was the goal of my research: to play a Mozart piano concerto and have my computer be the orchestra that accompanies me.”

“In piano concertos you have a pianist playing and an orchestra accompanies them. It’s very beautiful, it’s very challenging, but in order to do this, you need to have an orchestra — and those are hard to come by if you’re just an amateur pianist.”

With a recording of the orchestral score of Mozart’s Piano Concerto No. 12, the skills and software of likeminded collaborator Dr. Christopher Raphael at Indiana University, a computer-and-microphone setup, and his knowledge of programming and mathematics in hand, DeJongh got to work. He trained his computer to listen to piano pieces and identify the many simultaneous notes the instrument can produce. He dug into the mathematics and programming and began instructing the computer to play a prerecorded orchestral part and match its tempo with the piano’s as he played — without warping the pitch, as recordings are wont to do when the tempo is tweaked.

“It is able to maintain the pitch at the same level using some fancy mathematics,” DeJongh explains.

In a piano concerto, the piano’s the star. The orchestra follows, taking its cues from the pianist. Concertos include sections designed to give the pianist solos called cadenzas whose length is up to the musician. Training a computer to recognize what a conductor would signal to a live orchestra at those moments means taking a different approach than a programmer would to produce a karaoke track of classical music that would give the pianist an unyielding length of time for each cadenza.

“That’s sort of like saying, ‘Drive from Grand Rapids to Detroit and get there in exactly 137 minutes,” DeJongh says. “Well, you’re just not going to hit it right on the nose. The cool thing is that the computer does hit it right on the nose.” He trained it to recognize what a conductor would: the pianist’s flourishing trill signaling an imminent end to the cadenza.

DeJongh hopes to develop the technology into a software application that piano students could use to experience playing a concerto and to practice without an orchestra.

“There’s a really strong parallel between music performance and computer programming,” DeJongh says. From the outside peering in, both look remarkably like cryptic scribbles on a page (or screen). It all comes down to how the computer or performer interprets and enacts those instructions. “The goal of the computer program isn’t to look at those words and say, ‘Oh, what a nice computer program.’ It’s to produce a live experience. It’s the same with a music performance. The goal isn’t to look at those blobs of ink on a page; it’s for somebody to take that and interpret it and turn it into a live performance,” DeJongh says.

DeJongh enjoys melding computer science with disparate fields. Taking a philosophical approach, he’s also considering how computer behavior, and particularly algorithms, affects human activities — now and into the future.

“Computers are, despite what most people think, not really capable of random behavior,” he says. “But when the algorithms get complicated enough, they look random to the outside observer. And this is going to be, I think, the dilemma of the next century: We’re entrusting our lives to algorithms and we don’t understand how they make decisions.”

In the senior seminar he teaches, The Future of Being Human, DeJongh combines computer science with philosophy and theology. The course delves into his fascination with “how artificial intelligence and computer technology change our conception of what it means to be human, and of the soul. It’s about how digital technology is transforming society — transforming the ways that we communicate and our conceptions of ourselves,” he says. In every class, “there’s a whole range of views, and it makes for interesting conversation.”

“[A]ll of a sudden you realize: ‘Wait a minute — I can.’ To me, that is really what education and research are all about. It’s making discoveries and creating opportunities to do something really good that you didn’t even know you’d be able to do. It’s just an incredible experience.”

Contemplating the thousands of hours that go into a research project, alongside the myriad more invested in the education and collaboration that make it possible, DeJongh is awed.

“When I played that piano concerto for the first time, I started crying,” he says. He suspects that’s common when a heretofore unrealized dream reaches the cusp of reality. “And then all of a sudden you realize: ‘Wait a minute — I can.’ To me, that is really what education and research are all about. It’s making discoveries and creating opportunities to do something really good that you didn’t even know you’d be able to do. It’s just an incredible experience.”