In what is touted to be the first human-machine collaboration in a live musical performance, the Hong Kong Baptist University (HKBU) Symphony Orchestra recently held a public concert that demonstrated how artificial intelligence (AI) can be a creative force in music, cross-media art and dance.
The orchestra and an AI virtual choir staged a newly arranged choral-orchestral version of the song Pearl of the Orient, with an accompanying cross-media visual narrative based on the lyrics and music created by an AI media artist.
The performance was one of the highlights of the annual gala concert by the HKBU Symphony Orchestra held at the Hong Kong City Hall last month. Dubbed a “Lovers’ Reunion”, the concert was conducted by the orchestra’s music director Professor Johnny Poon.
“By leveraging our established strengths in both the arts and sciences, HKBU is taking a trans-disciplinary approach that will advance the development of the arts, culture and creative media in Hong Kong. Our art-tech research also enables musicians and artists to go beyond the traditional forms and interact with the audience in brand new ways,” said Poon.
Homegrown AI technologies
Research scientists at the HKBU developed the AI technologies used during the concert as part of the university project called "Building Platform Technologies for Symbiotic Creativity in Hong Kong". Led by Professor Guo Yike, vice president for research and development at HKBU, the research project is funded by a HK$52.84 million (US$6.77 million) grant from the theme-based Research Scheme under the Research Grants Council.
Professor Poon, who is also the associate vice-president for interdisciplinary research) and the founding dean of HKBU's School of Creative Arts, serves as the project’s deputy coordinator.
The project team is composed of computer scientists from HKBU's Augmented Creativity Lab. The unique mix of scientists and artists in the project team enables them to merge their cross-disciplinary knowledge and produce fresh ideas on how to unleash the endless possibilities of art creation with the infusion of cutting-edge technologies.
AI takes centre stage
During the concert, the project team presented a performance where an AI choir – voiced by 320 virtual performers – had combined with an AI-generated visual storyteller to perform interactively with a conductor and an orchestra.
"AI is revolutionising numerous fields of knowledge, and it is playing a more and more significant role in many creative processes like music, visual arts and architecture,” said Guo.
The AI virtual choir was "trained" by HKBU researchers using the generative models of singing that they built by extracting and disentangling key contributing features of vocal singing from a collection of songs recorded by professional singers.
An AI media artist learnt from the lyrics of the Pearl of the Orient, and it created a stunning cross-media visual narrative of the song according to its interpretation of the underlying meaning of the lyrics.
Unlike the conventional AI machines that import images as a reference for the algorithms to mimic, it used textual lyrics as the sole input so that it can associate the underlying meaning of the lyrics with an appreciation of the beauty of Hong Kong.
“The innovative concert showcases how the university is using technology to push the envelope of human imagination in the arts and cultural sphere,” said Poon
Another highlight of the concert was a ballet performance featuring AI virtual dancers in Ravel's Daphnis et Chloé, accompanied live by the HKBU Symphony Orchestra.
With the help of professional dancers from the Hong Kong Dance Company, the AI virtual dancers interpreted the underlying emotional and aesthetic connections between the music and the dance. The dance movements, inspired by a newly discovered species of box jellyfish in Hong Kong, were then choreographed.
