A "marriage of art and technology" is how Professor Horace Ip describes Body-Brush, a new tool that transforms body movements into real-time three-dimensional (3D) paintings. Professor Ip, Chair Professor of Computer Science and Head of CityU's Department of Computer Science, developed the innovative computer-based motion analysis system in collaboration with CityU's School of Creative Media (SCM) and Hong Kong artist Mr Hay Young.  

Professor Ip's research background is computer graphics and image analysis-that is, given an image, he develops techniques to analyse and understand what the image contains. "I've done a lot of work on medical image analysis and graphical simulation of the human face as part of a surgical planning system," he explained. But Body-Brush is quite different from anything he's done in the past; his previous work involved applying image processing or computer graphics techniques to more technically-oriented applications. So when Professor Ip sat down with Mr Young and exchanged ideas about how they could work together, he was intrigued by the prospect of working on a project that involved merging art and applied technology. "I like to work with people, and I've developed computer graphics techniques for simulating Chinese calligraphy, so when I discussed the concept of Body-Brush with Mr Young, I thought it was do-able, that the technology was there to realize this concept."

Transforming movement into art

All human movement is three-dimensional because we operate in a three dimensional world. But painting is traditionally done on a two-dimensional canvas, and for some time the art world has tried to break free from that dimension. To respond to body motion and gesture, based on the artist's motion in 3-D space, is the concept behind Body-Brush-the resulting painting is no longer two-dimensional and the user can view it as a 3-D object by wearing 3-D goggles. And instead of using a brush and a palette, the artist's body becomes the brush, transforming movement into art.

Technically, capturing body motion and gesture to allow the computer to "see" them involved developing a camera system to capture the image and input it to the computer. "And that took a bit of research," Professor Ip said, "because you want a lighting system that doesn't interfere with human motion." There are devices that capture motion but they require special clothing or sensors, which would interfere with the artist's movements. "We wanted something completely free, where the type of clothing didn't matter, so we came up with an infrared lighting system and infrared sensitive cameras."

One camera is not sufficient to give the 3-D position, so at least two are used-on the ceiling and from the side. With the array of infrared lamps, the images have a good contrast and are easy for the computer to analyse and to compute the 3-D location of the person in 3-D space.
Not only does the computer have to see what the artist is doing, however, it also has to generate a painting and project it back to the space where the artist is moving around. If a normal lighting system is used, the projection won't be good because it requires a relatively dark area to be effective. This is where the infrared imaging system has the advantage, Professor Ip said: "The human eye does not respond to infrared, so you can light up the person in a space dark enough to generate a 3-D painting in the same space. It's an innovative application of an infra-red imaging system."

Another technical challenge for Professor Ip and his research team was to devise a means of allowing the computer to pinpoint where the artist is in 3-D space, analyse what his body posture is, and transform that numerical information into painting attributes: that is, colour, width of stroke, and where the stroke is to be drawn in 3-D space. "And this all has to be done in real-time because you get an immediate response. We had to do a lot of image processing and graphical rendering at the same time as the video images were arriving."

This was the point at which the artist's input was required, and the team collaborated with Mr Hay Young on mapping the body posture and gesture and numerical data into aesthetic and drawing attributes. Eighteen months later, Body-Brush was unveiled.

The many applications of Body-Brush

While painting is the most immediately obvious use of Body-Brush, another application particularly interests Professor Ip-using the technique to allow children to express themselves. "Chinese school children are pretty shy and when they arrive at the laboratory they're reluctant to try Body-Brush. But eventually some brave kids will start moving their arms a bit and once they see what happens, they all forget about being timid and start jumping around. It's a great way for them to express themselves."

Professor Ip believes Body-Brush could also be of use as a treatment for psychiatric patients, by helping to show their inner worlds. Not only does the Body-Brush generate a painting but the computer captures every movement in 3-D space. "You could do a kind of off-line analysis of the movement and use it to work backward to determine a patient's emotional state," he said. Professor Ip would also like to explore the conjecture that different cultures move differently. Chinese people move in a way that may be characteristic of Chinese culture, as do Spanish people or Italians. "It would be interesting to get people from different cultures to try out Body-Brush. You'd get a 3-D painting out of it-a series of paintings of characteristics of different people."

Meanwhile, the Body-Brush project team is working on the next stage: the Body-Baton. With a grant from the Centre for Media Technology, Professor Ip and his team will attempt to translate body motion into music. Although the image capturing aspect is already developed, the difficulty lies in the musical side. "Other laboratories have done something similar but the music is always pre-recorded or random. We want the music to have some musical sense, to be based on musical theory," he explained. The research team is currently working with a colleague in the SCM who has musical training and they hope to get further input from local musicians.

Body-Brush is attracting a lot of attention overseas. After it was featured on CNN, Professor Ip received several enquiries from the US about other applications, such as a new medium for modern dance. And in September, Body-Brush won a Prix Ars-an Honorary Mention in the Interactive Art section of the prestigious festival Ars Electronica in Linz, Austria. More than 400 visitors tried out Body-Brush on the first day of the festival and the team was interviewed by television stations from all over the world, including Japan, Italy and Austria, Professor Ip said. And with Body-Brush invited to show in other art and graphic conferences in Korea, Japan and Switzerland, Professor Ip is looking forward to receiving suggestions for many other applications of the new creative medium.