in Technologie


Man vs Robot #2: Man vs Nao

Teaching Gaia, a NAO humanoid robot, boxing.

In this blog I will write about my experiences in the process of teaching Gaia, a NAO humanoid robot, boxing.

The “Robot in residence” project

Gaia staying with me is part of the project “Generation A = Algorithm” of the Goethe Institute, which wants to encourage people to think about the consequences, challenges and dangers of Artificial Intelligence. Part of the program is “Robot in residence”, in which the robot Gaia travels through Europe. At each stop, the robot stays in residence for a month to work and live in a different culture and learns something new at each place.

Gaia stays the whole month of October 2020 with me, I’m going to teach Gaia boxing.

NAO first time on

About MAN vs NAO

The project is called MAN vs NAO and it is part of the bigger MAN vs ROBOT project

By teaching boxing to a robot, I want people to experience the dilemmas involved in machines behaving agressive in a (relatively) safe way on a human level. How do people react to an aggressive robot? Can you get the public to sympathize with the robot? What happens if the robot cheats? I hope that MAN vs ROBOT will give people more insight into these questions.

The first part of Man vs Robot was a short theatre performance in which a human fought a boxing match against a self-built, real life robot. This first robot was not interactive and just followed a predefined sequence that was made realistic with theater. In part #2 I’m trying to give the robot interactive behavior.

In the original project plan, each participant would bring the robot in person to the next location. Because of Covid-19, unfortunately that is not possible, therefore the preface of my month with Gaia is waiting for the DHL parcel delivery. And that takes some time. Being send on October 1st, first Gaia seems to arrive on October 2nd, then October 3rd and finally she arrives on Monday 5th of October at the end of the day, that’s 5 days less to spend with her. While people can travel very limited, Gaia has made an odyssey from Bremen to Leipzig, that by plane to Amsterdam and then to Groningen. And Bremen is only two hours from Groningen, meeting at the border would have been a better solution.

However, she was there! After unwrapping the two packages and opening a very professional looking suitcase Gaia was sitting right there in my own house. Time to turn her on, exciting! Some flashing multicolored leds, some sounds, some flashings leds again, but then nothing… Maybe if I push the NAO button on the chest something will happen? Yes, she’s telling me her IP address in Italian. I have an IP-address, so I exist. But then nothing happens again, or wait a minute, she’s following me with her head, she’s looking at me!

That’s a very good start, in boxing, always keep looking at your opponent.

dhl delivery
NAO PR picture

What’s the next step? Publicity pictures off course. A test-match is planned in Forum Groningen on October 31st and they want to make a press statement as soon as possible. With the right boxing gear Gaia already looks like a professional boxer. Now that’s my Million-dollar baby. Now there’s a clear deadline, get ready to start training. 

(The test-match is here:

To teach Gaia more skills I have to connect her to a laptop that’s sent together with her. With a special program called Choreographe you can design and upload all kind off different behaviors to her. There are also some routines present made in the other places Gaia stayed. In Rome it learned to dance and to walk by the hand and in Bremen it learned a quiz about Bremen and how to kick a football. The rest of this first week we spent on discovering all the different possibilities Gaia has and do small tests with it. Among others, we created animation sequences, made her speak (unfortunately the Dutch language is not installed), tried face following and some simple interactions. The possibilities are really amazing and I wonder if I will be able to let her go by the end of the month.

At the end of this week we’ve made two short sequences in which Gaia shows she can talk aggressively like a true boxer and hit a punch bag, with gloves and also with bare fists.

Next week we will work on the footwork and different interactive punching techniques. Float like a butterfly, sting like a bee, as Mohammed Ali said

October 19th the second week, No pain no gain

This week I am planning to teach Gaia footwork and also some interactive behavior, like fending off punches. Let’s see if she’s strong, stable and fast enough do be a nice sparring partner. On Tuesday, I take her to my workplace at Broedplaats de Campagne to start working on that.

Broedplaats de Campagne (Creative Breeding Spot the Campagne) is a dream in progress of me and 9 othercreative entrepreneurs. We create our own workplace and workshops on the former Suger factory site in Groningen, the Netherlands. On 2400m2 of land we build a circular and largely self-sufficient and circular workplace with space for technology, nature, innovation and creativity.

It’s a nice place to work where Gaia can also meet some other Dutch people.

The first thing we’re trying to do is let Gaia walk towards a punchbag, punch and walk back. I tried to let her recognize the back, but that didn’t work, so she just walks some steps, punches and walks some steps back. Unfortunately, this doesn’t work out too well, somehow, she keeps on falling, even with the standard walking functionality she has baked in. We tried different surfaces on the ground to see if that helps, but not really. She keeps falling while walking, especially when the arms are up to defense, that seems to disturb the balance. It feels a bit mean to see her falling so many times, especially because she’s shouting “Auch” loudly every time she falls. The best solution for walking seems to be a very flat, almost slippery surface and leaving the arms on in the walking algorithm, which means no defense while walking. There surely is some trouble in balancing in the right way.

The second test was letting Gaia react to touching the head by defending with the arms then walk towards a person she sees and punch. This can be done by connecting the different arm movement sequences of the first week to the touch sensors on the head and the person tracking and walking abilities Gaia already has. This seems to work pretty well. The only problem is that the face tracking doesn’t work anymore if Gaia is too close to a person. It would be nice to use the sonar sensors in the last part of approaching a person, but somehow, they don’t seem to work. For now, she just does some extra steps and punches. I will contact the technical support that is available from the Robotics lab of the University of Wildau for the project about this.

We didn’t work on boxing all the time. On the day off we had on Wednesday, I took Gaia home. She played with my family and also tried helping to clean the house. It really looks like she has always lived here when she sits there between the Duplo and other toys on the floor of our cozy but messy household. Before starting to work on Thursday, we went for a ride in a real Dutch way, in a bicycle car. I showed her some of the environment around here, which she really liked. After that we got back to work.

In the third test this week was a training routine to see if Gaia can react to voice commands of a trainer and test the strength in her arms. This really works well, the voice commands have to be repeated sometimes, but she nods when understood and she has enough power in the arms to do push-ups. She can lie down on command and do push-ups if asked to do so in a real bootcamp style while counting the push-ups. Only sometimes she complained that she can’t move a part of her arm or about motors that are getting too hot. It helped to give her some rest than to get her going again.

The last part of the week I used to get the sonar sensors working. After some mailing back and forth with Wildau, we decided it was best to create a new sonar module with the Python programming language. After spending some hours on that, I created a digital Nao building block that can see if something is approaching the chest of the robot from the left or the right. When connecting that to defending arm routines, Gaia should be able to defend herself against approaching hands of opponents. We tested that off course. In the test it turned out that the reaction was not very fast and the approaching of a hand was not always seen. So the reaction was not fast enough to defend against a real punch but for slow-motion boxing or a simple interactive not-touching game it should do.

At the end of a week in which we did a lot we can draw some conclusions about the abilities for boxing Gaia has:

  1. Simple arm movements for punches and defense are possible, but are not fast and strong enough for real boxing
  2. Walking and pointing towards a person is possible but the balancing is not very good
  3. Interaction by using different abilities like person tracking, touch sensors, voice recognition and sonar is possible but has various limitations in speed, range and reliability
  4. Gaia also likes to have fun

One final note I would like to make in the context of my bigger autonomous weapons project is that it already is possible to give Gaia a weapon she can trigger with her hand, recognize a certain person, point an arm to the person and pull the trigger. It is even very easy to build this routine with standard building blocks. That is a bit of a scary thought…

Having said that, let’s move on to the next week, which is already the second last she is here. After testing all the abilities Gaia has in the first two weeks, in next week I’m going to try to combine these abilities to a meaningful interaction routine. And I’m also going to try to influence the perception people have of Gaia. Is she nice, mean, neutral or something else. What do people think a robot thinks?

October 23rd, the third week, Picky and Evil

It is already the second last week Gaia will stay with me. I’m surely going to miss her and with her. Gaia being here really generates a constant stream of creative ideas on what can be done with a robot. If she would have been here for a year, I could have filled that period too with interesting experiments and videos, I think.

This week I want to focus on changing the perception people have of Gaia. She is designed to look, sound and act cute and friendly, a bit like a child. Her moving and behavior are all a bit clumsy, which also seems to make her more friendly. So how can you change that?

The first experiment is trying to let her look different. By putting different patterns on her face with tape and pen the look of her face can be changed. Dark eyebrows, a moustache or even a monster look works a bit. However, even then the small childlike body still makes her look kind of cute.

The next few experiments are trying to give her behavior that makes her more unfriendly.

First, I made a video where it looks like Gaia hits a phone from somebodies’ hand. Works fine on video, unstoppable fight machine! But in real live it’s obvious it is faked.


The second attempt is behavior inspired by the evilest doll that I know, Chuckie, from the Childs play horror movie. What if Gaia walks around with a big knife and red flickering eyes? I also programmed her to drop the knife and act innocent when she sees a face (that assumes she is discovered while wanting to stab your back). An evil hypocrite robot with a big knife should be perceived as unfriendly and dangerous. I’ve made a video of it and put it on Youtube to see how people react to it. But I’m afraid people will still think it’s funny.

While discussing the experiments with Chantalla Pleiter, another artist that works at the Campagne, she had the idea that giving the robot antisocial behavior could also work. The face tracking algorithm in Gaia is now following a person by looking at, turning to or walking to that face, which make her look interested and aware of you. What if Gaia does the opposite, turning her back when she sees a face? That might make her unfriendly. It turns out that this works a bit, but her small body and the clumsy movements make her look more like an unhappy child, which is still more cute than unfriendly.

Finally, I let Gaia tell the gender, age and mood of someone standing in front of her. Guessing someone’s age is experienced as very unpolite and uncomfortable by a lot of people, especially when the guessing it is much younger or older than the real age. When testing it with myself for several times, Gaia told me I was 27, 28, 32, 33, 38 and 40 years old (I’m 43). So, it is not very accurate. However, when testing it with a visitor in are workspace she felt pretty uncomfortable and didn’t really like what Gaia was saying.

The conclusion of the experiments of this week is that it is not so easy to make a robot that is designed to be cute, look un-cute. The main reason for this is that the robot will always stay small, a bit clumsy and looking like a child. We humans are programmed to like that. However, this week gave a small insight on the possibilities there are to accomplish the goal of making a robot look un-sympathetic.

Next week is already the last week Gaia stays with me. At the end of the last week there was an audience confrontation planned in Forum Groningen, a place with a lot of audience. Unfortunately, that confrontation was canceled because of new stricter Covid19 regulations. Instead of that I’m trying to arrange a small series of one to one robot encounters in my workspace to test how different people react to all of Gaia’s new behavior. Videographer Pieter van Dijken is going to make a video/mini documentary about it to share the reactions. The last week I’m going to focus on combining all the experiments and findings to a final routine to see how people react to a cute robot trying to be aggressive. I’m really looking forward to it!