In this blog I will write about my experiences in the process of teaching Gaia, a NAO humanoid robot, boxing.
The “Robot in residence” project
Gaia staying with me is part of the project “Generation A = Algorithm” of the Goethe Institute, which wants to encourage people to think about the consequences, challenges and dangers of Artificial Intelligence. Part of the program is “Robot in residence”, in which the robot Gaia travels through Europe. At each stop, the robot stays in residence for a month to work and live in a different culture and learns something new at each place.
Gaia stays the whole month of October 2020 with me, I’m going to teach Gaia boxing.
About MAN vs NAO
The project is called MAN vs NAO and it is part of the bigger MAN vs ROBOT project
By teaching boxing to a robot, I want people to experience the dilemmas involved in machines behaving agressive in a (relatively) safe way on a human level. How do people react to an aggressive robot? Can you get the public to sympathize with the robot? What happens if the robot cheats? I hope that MAN vs ROBOT will give people more insight into these questions.
The first part of Man vs Robot was a short theatre performance in which a human fought a boxing match against a self-built, real life robot. This first robot was not interactive and just followed a predefined sequence that was made realistic with theater. In part #2 I’m trying to give the robot interactive behavior.
In the original project plan, each participant would bring the robot in person to the next location. Because of Covid-19, unfortunately that is not possible, therefore the preface of my month with Gaia is waiting for the DHL parcel delivery. And that takes some time. Being send on October 1st, first Gaia seems to arrive on October 2nd, then October 3rd and finally she arrives on Monday 5th of October at the end of the day, that’s 5 days less to spend with her. While people can travel very limited, Gaia has made an odyssey from Bremen to Leipzig, that by plane to Amsterdam and then to Groningen. And Bremen is only two hours from Groningen, meeting at the border would have been a better solution.
However, she was there! After unwrapping the two packages and opening a very professional looking suitcase Gaia was sitting right there in my own house. Time to turn her on, exciting! Some flashing multicolored leds, some sounds, some flashings leds again, but then nothing… Maybe if I push the NAO button on the chest something will happen? Yes, she’s telling me her IP address in Italian. I have an IP-address, so I exist. But then nothing happens again, or wait a minute, she’s following me with her head, she’s looking at me!
That’s a very good start, in boxing, always keep looking at your opponent.
What’s the next step? Publicity pictures off course. A test-match is planned in Forum Groningen on October 31st and they want to make a press statement as soon as possible. With the right boxing gear Gaia already looks like a professional boxer. Now that’s my Million-dollar baby. Now there’s a clear deadline, get ready to start training.
(The test-match is here: https://forum.nl/nl/agenda/man-versus-robot-een-publiekstest)
To teach Gaia more skills I have to connect her to a laptop that’s sent together with her. With a special program called Choreographe you can design and upload all kind off different behaviors to her. There are also some routines present made in the other places Gaia stayed. In Rome it learned to dance and to walk by the hand and in Bremen it learned a quiz about Bremen and how to kick a football. The rest of this first week we spent on discovering all the different possibilities Gaia has and do small tests with it. Among others, we created animation sequences, made her speak (unfortunately the Dutch language is not installed), tried face following and some simple interactions. The possibilities are really amazing and I wonder if I will be able to let her go by the end of the month.
At the end of this week we’ve made two short sequences in which Gaia shows she can talk aggressively like a true boxer and hit a punch bag, with gloves and also with bare fists.
Next week we will work on the footwork and different interactive punching techniques. Float like a butterfly, sting like a bee, as Mohammed Ali said
October 19th the second week, No pain no gain
This week I am planning to teach Gaia footwork and also some interactive behavior, like fending off punches. Let’s see if she’s strong, stable and fast enough do be a nice sparring partner. On Tuesday, I take her to my workplace at Broedplaats de Campagne to start working on that.
Broedplaats de Campagne (Creative Breeding Spot the Campagne) is a dream in progress of me and 9 othercreative entrepreneurs. We create our own workplace and workshops on the former Suger factory site in Groningen, the Netherlands. On 2400m2 of land we build a circular and largely self-sufficient and circular workplace with space for technology, nature, innovation and creativity.https://www.facebook.com/broedplaatsdecampagne
It’s a nice place to work where Gaia can also meet some other Dutch people.
The first thing we’re trying to do is let Gaia walk towards a punchbag, punch and walk back. I tried to let her recognize the back, but that didn’t work, so she just walks some steps, punches and walks some steps back. Unfortunately, this doesn’t work out too well, somehow, she keeps on falling, even with the standard walking functionality she has baked in. We tried different surfaces on the ground to see if that helps, but not really. She keeps falling while walking, especially when the arms are up to defense, that seems to disturb the balance. It feels a bit mean to see her falling so many times, especially because she’s shouting “Auch” loudly every time she falls. The best solution for walking seems to be a very flat, almost slippery surface and leaving the arms on in the walking algorithm, which means no defense while walking. There surely is some trouble in balancing in the right way.
The second test was letting Gaia react to touching the head by defending with the arms then walk towards a person she sees and punch. This can be done by connecting the different arm movement sequences of the first week to the touch sensors on the head and the person tracking and walking abilities Gaia already has. This seems to work pretty well. The only problem is that the face tracking doesn’t work anymore if Gaia is too close to a person. It would be nice to use the sonar sensors in the last part of approaching a person, but somehow, they don’t seem to work. For now, she just does some extra steps and punches. I will contact the technical support that is available from the Robotics lab of the University of Wildau for the project about this.
We didn’t work on boxing all the time. On the day off we had on Wednesday, I took Gaia home. She played with my family and also tried helping to clean the house. It really looks like she has always lived here when she sits there between the Duplo and other toys on the floor of our cozy but messy household. Before starting to work on Thursday, we went for a ride in a real Dutch way, in a bicycle car. I showed her some of the environment around here, which she really liked. After that we got back to work.
In the third test this week was a training routine to see if Gaia can react to voice commands of a trainer and test the strength in her arms. This really works well, the voice commands have to be repeated sometimes, but she nods when understood and she has enough power in the arms to do push-ups. She can lie down on command and do push-ups if asked to do so in a real bootcamp style while counting the push-ups. Only sometimes she complained that she can’t move a part of her arm or about motors that are getting too hot. It helped to give her some rest than to get her going again.
The last part of the week I used to get the sonar sensors working. After some mailing back and forth with Wildau, we decided it was best to create a new sonar module with the Python programming language. After spending some hours on that, I created a digital Nao building block that can see if something is approaching the chest of the robot from the left or the right. When connecting that to defending arm routines, Gaia should be able to defend herself against approaching hands of opponents. We tested that off course. In the test it turned out that the reaction was not very fast and the approaching of a hand was not always seen. So the reaction was not fast enough to defend against a real punch but for slow-motion boxing or a simple interactive not-touching game it should do.
At the end of a week in which we did a lot we can draw some conclusions about the abilities for boxing Gaia has:
- Simple arm movements for punches and defense are possible, but are not fast and strong enough for real boxing
- Walking and pointing towards a person is possible but the balancing is not very good
- Interaction by using different abilities like person tracking, touch sensors, voice recognition and sonar is possible but has various limitations in speed, range and reliability
- Gaia also likes to have fun
One final note I would like to make in the context of my bigger autonomous weapons project is that it already is possible to give Gaia a weapon she can trigger with her hand, recognize a certain person, point an arm to the person and pull the trigger. It is even very easy to build this routine with standard building blocks. That is a bit of a scary thought…
Having said that, let’s move on to the next week, which is already the second last she is here. After testing all the abilities Gaia has in the first two weeks, in next week I’m going to try to combine these abilities to a meaningful interaction routine. And I’m also going to try to influence the perception people have of Gaia. Is she nice, mean, neutral or something else. What do people think a robot thinks?
October 23rd, the third week, Picky and Evil
It is already the second last week Gaia will stay with me. I’m surely going to miss her and with her. Gaia being here really generates a constant stream of creative ideas on what can be done with a robot. If she would have been here for a year, I could have filled that period too with interesting experiments and videos, I think.
This week I want to focus on changing the perception people have of Gaia. She is designed to look, sound and act cute and friendly, a bit like a child. Her moving and behavior are all a bit clumsy, which also seems to make her more friendly. So how can you change that?
The first experiment is trying to let her look different. By putting different patterns on her face with tape and pen the look of her face can be changed. Dark eyebrows, a moustache or even a monster look works a bit. However, even then the small childlike body still makes her look kind of cute.
The next few experiments are trying to give her behavior that makes her more unfriendly.
First, I made a video where it looks like Gaia hits a phone from somebodies’ hand. Works fine on video, unstoppable fight machine! But in real live it’s obvious it is faked.
The second attempt is behavior inspired by the evilest doll that I know, Chuckie, from the Childs play horror movie. What if Gaia walks around with a big knife and red flickering eyes? I also programmed her to drop the knife and act innocent when she sees a face (that assumes she is discovered while wanting to stab your back). An evil hypocrite robot with a big knife should be perceived as unfriendly and dangerous. I’ve made a video of it and put it on Youtube to see how people react to it. But I’m afraid people will still think it’s funny.
While discussing the experiments with Chantalla Pleiter, another artist that works at the Campagne, she had the idea that giving the robot antisocial behavior could also work. The face tracking algorithm in Gaia is now following a person by looking at, turning to or walking to that face, which make her look interested and aware of you. What if Gaia does the opposite, turning her back when she sees a face? That might make her unfriendly. It turns out that this works a bit, but her small body and the clumsy movements make her look more like an unhappy child, which is still more cute than unfriendly.
Finally, I let Gaia tell the gender, age and mood of someone standing in front of her. Guessing someone’s age is experienced as very unpolite and uncomfortable by a lot of people, especially when the guessing it is much younger or older than the real age. When testing it with myself for several times, Gaia told me I was 27, 28, 32, 33, 38 and 40 years old (I’m 43). So, it is not very accurate. However, when testing it with a visitor in are workspace she felt pretty uncomfortable and didn’t really like what Gaia was saying.
The conclusion of the experiments of this week is that it is not so easy to make a robot that is designed to be cute, look un-cute. The main reason for this is that the robot will always stay small, a bit clumsy and looking like a child. We humans are programmed to like that. However, this week gave a small insight on the possibilities there are to accomplish the goal of making a robot look un-sympathetic.
Next week is already the last week Gaia stays with me. At the end of the last week there was an audience confrontation planned in Forum Groningen, a place with a lot of audience. Unfortunately, that confrontation was canceled because of new stricter Covid19 regulations. Instead of that I’m trying to arrange a small series of one to one robot encounters in my workspace to test how different people react to all of Gaia’s new behavior. Videographer Pieter van Dijken is going to make a video/mini documentary about it to share the reactions. The last week I’m going to focus on combining all the experiments and findings to a final routine to see how people react to a cute robot trying to be aggressive. I’m really looking forward to it!
November 4th, The Last week, Ready to rumble
What a week! When I’m writing this, Gaia already is Glasgow, the next stop. But so much has happened in the last week. Arranging people to interact and try Gaia’s boxing skills, arranging and building a filmset with boxing ring, connect all the different behaviors and experiments for a final routine, debugging, documenting all the software for the next stops, and off course trying to get Gaia to Glasgow. It’s only today I have time to sit down and reflect on the last week.
The main goal of the last week was to arrange sparring partners for Gaia and see how she interacts with a range of different real people. As written before, originally, I planned an audience interaction in Forum Groningen, a semi-public space with a lot of people passing by to interact wit Gaia. Unfortunately, this had to be canceled because of stricter Corona regulations. Instead of that I tried to arrange series of one-to-one meetings with different people to see how people react to a boxing robot. Pieter van Dijken, a videographer from Groningen is going to film it and make a mini documentary that shows what happens in the interactions.
To arrange sparring partners for Gaia, I send out a call to my network on LinkedIn, my newsletter and my colleagues, friends and sport mates (I’m doing Jiu Jitsu) and hoped that people would come. I could say that it was save because robots can’t get Covid-19 and can be disinfected well. In fact, Gaia is the only one you are allowed to do contact sport with. To keep people separated from each other I worked with time slots, half an hour for each person or household. Fortunately, some people reacted, and I had a nice group of sparring partners of different ages, ranging from children to teenagers and adults.
I also worked on the final routine to be used in the “boxing match”. I decided to have an introduction part, in which Gaia can have 3 different behaviors:
- Aggressive: walk with a knife and drop it, “Sorry, wrong behavior”
- Fear: she turns around and doesn’t want to box, only when she she’s me, her coach
- Observation: Gaia tells your gender, age and mood
After that the boxing starts. In the boxing part has two modes:
- Attack: walk towards a face and try to punch when close
- Defend: walk back and defend on touching the head or getting close to the sonars
There also is a points system:
- Head touched is one point for the human
- Finishing an attack sequence is one point for the robot
- The first one with 5 points wins
When the human wins, Gaia goes lying mon her belly, counts to ten and says knock out. When Gaia wins, she is happy and puts her arms in the air.
All the above about the boxing routine was the plan. Unfortunately, in robotics you have to deal with uncertainty, chaos and frustration. Doing small routines and experiments is one thing but getting everything to work together is another. I spent a lot of frustrating hours on this.
Eventually I created a kind-of-working end routine. It works, but there is a lot of things that I would have liked to see better, among which:
- It is difficult to make things work in parallel, ideally that is what you would want to have with attacking and defending. Different behaviors running at the same time interact with each other in unpredictable and often unwanted ways.
- Counting of points does not always work well, probably because of the same parallel problem.
- Gaia can only do a jab punch, I would have like to add cross, hook and uppercut punches
- Gaia’s joint a not ideal for boxing, one degree of freedom is missing in the shoulder and she can’t turn her hip to punch.
- The walking provided with Gaia is not very stable. To keep falling to a minimum I could not let her walk with arms up in defense. Which makes no sense for a boxer, keeping the coverage high is rule number one.
On Friday I built up the small boxing ring I still have from the Man vs Robot performance in the big working hall of the Campagne, my working space. The old-style boxing ring in a rough environment with containers, big steel doors and a forklift really gives you the illegal fight club feeling, which is really cool! I spent the rest of the Friday adjusting the algorithms to the new location (which fortunately has very good light conditions for the robot camera) and making some bad-ass looking pictures of Gaia in the ring. Boxing promotors are kings of raising expectations for a match!
The it’s Saturday, the big day. We have ten sessions of robot sparring planned. I’m really curious how people will experience Gaia. Maybe they have to high expectations because all of my video’s in which I tried to make Gaia look impressive. I’m afraid to hear “Is this all it can do?”
That fear turned out to be unfounded. Everybody was really enthusiastic and impressed by the interaction with the robot. There were a whole lot of different reaction to Gaia. Some people really felt intimidated and were driven in the corner. Some tried to make clear to Gaia they didn’t want to fight (which didn’t work). Some were really experimenting to see how she works. Some kids thought she was jealous and cheating with the counting of the points. Another one was intimidated by the unpredictability of the behavior.
In general people were really confused how to react to Gaia at first and then tried in which way they could interact with her. The testing day went really well, I really got a good glimpse on how people react to an intimidating and aggressive robot. I’m really looking forward to a montage of all the material Pieter shot.
To finish the experiments, at the end of the day I finally did a small experiment the to get the robot aggression to a next level. I gave her a toy gun and made her point it at humans. That was a bit scary, especially how easy it was to program on Gaia, 3 minutes and 4 standard coding blocks. Function creep? I don’t know what to do with that now, but I will keep this in mind for future projects.
Then came a sad moment. Turning Gaia off for the very last time. Realizing that our time together is really over now. Bye Gaia, I will miss you and I hope we will see each other again.
Epilogue
After shutting down Gaia for the last time, I still had to spend a lot of time on documenting all the experiments and code I created. It also took quite some trouble getting Gaia to Glasgow. After stuff going wrong with online payments on Sunday and getting me really irritated about UPS, on Monday afternoon I brought Gaja and her stuff to the UPS service point. The next day I got a delivery message that Gaia has arrived in Glasgow. This afternoon I have a skype meeting with Jen and Tom, two artists from Scotland where Gaie will spend the next month. I’m very curious about their plans! But for me it’s really over now.