Also – can robots beat humans at soccer?

AI researchers from Sony have created a system capable of beating human players at the racing video game “Gran Turismo 7.”

Dubbed GT Sophy, the AI was built using reinforcement learning, meaning it was given ‘rewards’ when performing correct actions, effectively encouraging it to learn what decisions are good to make in order to improve.

The deep reinforcement learning approach eventually made the system capable of competing with human players — making near-perfect turns and braking at the right times.

AI Business recently sat down with Peter Stone, executive director of Sony AI and a professor of Computer Science at the University of Texas at Austin. Stone talked about how Sony built GT Sophy and the likelihood of it being featured in retail copies of “Gran Turismo 7.”

Stone is the immediate past president of the RoboCup Federation. RoboCup is a yearly event where AI developers, roboticists and researchers meet to take part in the challenge of trying to get robots to play soccer. Started in 1997, RoboCup celebrated its 25^th international competition earlier this year, and has the goal of beating a professional human soccer team by the year 2050.

Stone talked about the realistic chance of when that win will occur and what roboticists hope to achieve by teaching robots how to play soccer.

The following is an edited transcript of that conversation. You can listen to the full chat in the latest episode of the AI Business Podcast below, or wherever you get your podcasts.

AI Business: Why did you make GT Sophy?

Peter Stone: We set out to see whether an AI agent could achieve the levels of control, precision and tactics that are needed in a head-to-head high-speed racing competition.

In contrast to RoboCup, we set out to do this with an end-to-end approach: using reinforcement learning to learn from scratch, how to drive against people. And when I say learn from scratch, I mean it practices starting from random behavior. In the beginning, GT Sophy just drives off the road, runs into walls and has very random behavior. But through a sequence of getting rewards for driving quickly and punishments for running into things or driving off the track, it gets better and better.

AI Business: What made you choose Gran Turismo over other video games? Was it purely because it’s a Sony-owned title? What made it stand out?

Stone: It is one of the most popular racing games and has a unique set of challenges that we thought were the type of technology that we were bringing to bear, which is deep reinforcement learning. There were some tough scientific questions that we needed to answer like, ‘can you use reinforcement learning to control a car where a decision may cause you to lose an advantage in a race? What’s the sweet spot?’

And then, I don't think we chose it because of this, but one of the very interesting aspects that we came across as we were developing, was that Gran Turismo, and racing games, in particular, have this complex code of etiquette of sportsmanship, where you want to drive as fast as you can, but you're not allowed to, for instance, cut another car off or force them off the track; you have to make sure they still have a driving line.

Those rules are not well-defined rules about what's considered fair and what's not. There's a human steward who makes judgments as to whenever there's a collision or a near collision, whether a car should be penalized and forced to slow down.

This turned out to be a really interesting and difficult challenge for an automated learning system. How do you teach it to respect the unwritten rules of etiquette, while also being sufficiently aggressive to win a race?

This became one of the big technical challenges for us in devising a reward function or reinforcement signal for this automated learning agent that could balance this aggressiveness and ability to be considerate in the proper way.

AI Business: How important is this AI for racing video games going forward?

Stone: We are actively working with the game developers to try to incorporate a version of GT Sophy into “Gran Turismo 7” that will allow the general public to interact with it.

We recently demonstrated it in Salzburg (Austria) at the human-racing GT championships showing that it can be incorporated into games and that it can run on the PlayStation, rather than on external computers. The real goal is to make the games more entertaining.

AI Business: What can we expect from the Sony AI team over the next 12 months?

Stone: We're going to continue to try to work towards getting GT Sophy deployed in GT seven − that will be a big milestone for us. We don't know when that will happen, there are still some technical challenges, but we hope that'll be something you'll be able to see.

Talking near term, we're working on academic publications − there are more detailed scientific studies that we can do to examine, for example, of the different algorithms that went into GT Sophy, to what extent they can be more broadly applicable.

RoboCup: Robots playing soccer

AI Business: Conscious of your 2050 deadline for robots to beat humans at soccer, how close do you think you are to achieving that?

Stone: It's difficult to say, and it's important to recognize that these targets are aspirational and meant for motivation. They're never correct, even if it happens decades after 2050, it would still be a really impressive result.

With regard to how close (to reaching that goal), we're definitely making a lot of progress. The community as a whole is making strides in terms of hardware needs and more agile robots that can move more quickly. But it's still far from the agility that professional soccer players can exhibit.

The RoboCup community is ready to pull it all together to make some of those important advances. If it's possible, there's no better collection of people to do it than the people who've been dedicated to this over the years.

AI Business: Do you think football fans are ready to embrace robotic players? How are you trying to get in touch with the wider sporting community in this regard?

Stone: RoboCup is fundamentally and primarily a research initiative. It's not an effort to change the sporting world. Even if and when robots can beat people at soccer, it won't spell the end of human competition by any stretch.

People still play chess, there are still human chess champions. If and when we get to that point, it would be thrilling, but I don't think it would be the end of the human element of soccer.