SDS PODCAST EPISODE 110 ALPHAGO ZERO Show Notes: http://www.superdatascience.com/110 1
Kirill: This is episode number 110, AlphaGo Zero. Welcome back ladies and gentlemen to the SuperDataSceince podcast. Today I would like to talk about AlphaGo. It's going to be an interesting session, I'm going to try to cramp up quite a lot of things that happened over the past couple of years in the space of artificial intelligence, playing the game of Go. It s going to be very fun if you're not up to date especially in that case, if you're not up to date with these advancements. If you are, then there might be a few things that you may have missed out on especially of the recent kind. Alright, so let's get started. The game of Go, I'm sure everybody listening to this podcast or hopefully almost everybody, is familiar with the game of chess and the game of chess has been around for ages of course, and is said to originate in India. Computers for the first time won the game of chess against humans in 1997 when the computer program called Deep Blue for the first time beat the world champion at the time, Garry Kasparov who was a Russian player. In 1996, the computer was not able to do that, Garry Kasparov won, but then the team building the software went back and a year later they turned around and beat Garry Kasparov, and now pretty much any computer that you have or any software that you have that plays chess even on your phone, or the basic ones that you get online, if you put them on the hardest difficulty, there's no way you can beat them, full stop. Because that's the way the algorithm is designed that they will not make a mistake and that's pretty much what the game of chess is about, you wait until your opponent makes a mistake, and humans usually do make mistakes, and computers are now able not to make mistakes. Show Notes: http://www.superdatascience.com/110 2
Basically, that shows that computers have come a long way since then. Back in 1997 it was like World News that a computer won chess, now we all know that computers just play better chess. And now what has that got to do with the game of Go? Well, the game of Go has actually been around for even longer than chess, I believe, but it's been around for thousands of years. It's also a board game, it s played on a board. In chess, you have eight by eight squares, so you have 64 squares in total and you have all these different pieces. In the game of Go, you have a board with 19 horizontal, 19 vertical lines, and instead of pieces you have stones, you have white and black stones, and one player puts white stones on the board, the other player puts black stones on the board. In the game of Go, the rules are as follows: You can put a stone on the intersection of horizontal and vertical lines; Wherever you put a stone, if you surround an opponent's stone with four of your stones and then you capture that stone, it's taken off the board, I believe that's what happens; At the end of the game, it's all about the territories. Your stones, the stones that you have, it's calculated how much area they cordon off within them, like what kind of border they make and that area within them is counted towards your points, and the opponent s points are counted in a similar way, and whoever has more points wins the game. Show Notes: http://www.superdatascience.com/110 3
That's what the game of Go is about. As you can see, it s even much more basic than chess because it doesn't have those pieces, those different types of moves, it s just putting stones on the board. Well it sounds more basic than chess. However, the game of Go was not expected to be won by an artificial intelligence against a human until approximately the year 2025, just based on computational advancements and how quickly computers could process things. The reason for that is there s a couple of reasons. First of all, the board for the game of Go is much greater, it's nineteen by nineteen versus eight by eight, and therefore the number of combinations is astronomically larger than the number of combinations, number of possible games in the game of chess. That s number one. Number two, in chess you have a value function, you can derive a value function. By looking at a board, you can tell who is winning, black or white, simply because all pieces have approximately their values. Like, a knight is two points, bishop is arguably two or three points, pawn is one point, and so on, so pieces have their values and you look at the board, you can tell who is winning, so you can program a value function into a computer and it can keep track of that value function. In the game of Go, all pieces are the same, you cannot have a value function, and therefore it s much harder. The next one is in the game of chess, you work from a full board, full of pieces to the number of pieces is reduced. They capture each other and they re taken off the board. In the game of Go it's the opposite. You are building the board, you're adding pieces to the board all the time, so that's another thing that rather than reducing the amount of pieces and therefore the complexity of the game over time, you re increasing the amount of pieces and there's lots of different ways that this can go. So, multiple reasons, therefore, that Show Notes: http://www.superdatascience.com/110 4
lead to the fact that the game of Go is much harder for a computer to understand. Kind of the main one is that because there's so many combinations and because these stones are all the same, in the game of Go, a huge component is intuition. Human players have always been playing with intuition and just basically looking at the game of Go. In chess, it's logic, it's calculation, it's forecasting, it's looking ahead. In Go, it's a lot of the game, a lot of the time if you ask one of the champions or grand masters of Go or World Masters, why they made a certain move, a lot of the time they will answer, it felt right. And just through their experience, they have this intuition about what feels right, what doesn't, and in the end, it usually works out for them, like several moves or towards the end of the game, several moves down the track or towards the end of the game, that move that they made based on intuition pays off and helps them win the game. And so, computers would have to develop intuition to play the game of Go. That's a quick overview of what the situation was or is. And then there is this company, DeepMind, which is a subsidiary of Google, which was acquired by Google not so long ago, and they took on the challenge of building an artificial intelligence which would play the game of Go, and they developed AlphaGo, this is the program that eventually challenged the world grandmaster in the game of Go. In March 2016, so last year March, AlphaGo played against the world grandmaster. In fact, it was the 18-time world champion, Lee Sedol. I think Lee Sedol is a Korean player if I'm not mistaking. Yes, Lee Sedol is a Korean player, 18-time world champion of Go. AlphaGo, the computer program, played against him and it beat Lee Sedol four games out of five. That was huge, it happened last year when it was expected to happen only ten Show Notes: http://www.superdatascience.com/110 5
years later. It happened ten years earlier then it was expected to happen, a huge advance. Meaning if you judge the progression of artificial intelligence by just that one feat, you can see that we are already ahead of where we should have been, where we were expected to be. The way that AlphaGo learned, it's a deep learning algorithm, so it's artificial intelligent deep learning, the way it learned, is it analysed lots and lots, millions of games played by humans, which are available online and you can get them through the Go servers. It analysed those games and then it played against itself millions and millions of times, and that's how it learned. Eventually through that, it was able to beat Lee Sedol in the previous year. That's all good and well and you ve probably even heard of that, you might have heard of that, but what happened next is the most interesting and exciting part. This is really crazy. In one of the Five Minute Fridays a couple weeks ago, we talked about exponential technologies, exponential trends, and this is a very apt demonstration of what we are dealing with in the modern day and age. In 2016, a year and a half ago, in March 2016, AlphaGo beat Lee Sedol four games out of five so you can say it was a close game, it wasn't three versus two but they were playing, people watching, it was kind of close. They were playing on a very similar level. Then what happened is the guys from Deep Blue thought, okay, why did AlphaGo lose that one game? Something doesn't sit right with us, they shouldn t have lost that one game, and so they were like, okay, we'll go back and redesign the algorithm. After that, they came up with AlphaGo Master, and AlphaGo Master is the next version of AlphaGo and this happened in, I think it was January or February this Show Notes: http://www.superdatascience.com/110 6
year, so about six months ago, or nearly a year ago. AlphaGo Master was the advanced version of AlphaGo, and what AlphaGo Master was able to do is it played against the top 60 players in the world of Go, I think it was 60, like a number around 60, the number that's in my head right now is 60. It played against the top 60 players in the world, and it beat them all 60-0, it won every single one of those games, it didn t lose even once. That was AlphaGo Master, next level. Then they didn t even stop there. What happened is they started working on something brand new and just recently, in the middle of October, so about a month ago, just over a month ago, they released AlphaGo Zero. This is truly mind blowing. This time, instead of training the software or the algorithm on human games and watching how humans played and learning from that and then playing against itself, they trained the whole algorithm, they designed it in a way that they could train it from the very scratch, just playing against itself. Zero human intervention, zero human experience and the algorithm is not even told anything about the game, not told the rules it just can place its stones, it has to experiment and just through reinforcement learning, trial and error. It tries and tries, it fails, tries and fails. The secret trip or cool hack that they did was they made it play, it played a game against itself from the very start and so it always had an opponent of the same level as itself. Instead of having like a very smart opponent or humans who have experience and knowledge in this game to learn from, it was learning from an opponent which is itself. Therefore, they have the exactly the same level of experience in the game, exactly the same level of understanding of the game, and so at the start they both don t even know how to play. Then some time passes, at least they know how play the game but they both played terribly bad, Show Notes: http://www.superdatascience.com/110 7
terribly wrong. Sometime passes, and by time we mean iterations. This is happening really, really, fast on the Google servers, these iterations are churning out. Then they play better and better and better, and in the end, they came up with AlphaGo zero which beat the original version of AlphaGo or AlphaGo Master, so After just three days of self-play training, this is a quote from the DeepMind blog, AlphaGo Zero emphatically defeated the previously published version of AlphaGo, which had itself defeated 18-time world champion Lee Sedol - by 100 games to 0. 100 games to 0, it beat the version of AlphaGo that beat the world champion one and a half years ago. After 40 days of self-training, AlphaGo Zero became even stronger, outperforming the version of AlphaGo known as Master, which has defeated the world's best players and world number one, Ke Jie. There you go, I find that really crazy that just first of all through self-training it was able to do that and defeat everything that was before trained on human examples and was playing against humans at the start. Also, what I find the most mind-blowing is it s only been one and a half years. It s only been one and a half years since they developed the first version of AlphaGo that people were thinking, can it beat Lee Sedol, can it beat the world champion or not, it s 10 years early, maybe it s not going to happen. Literally the whole of Asia stopped at the time the games were played, they were broadcasted across all news channels, all channels in South Korea, in China, in Japan, that s the three countries where this game is big, everybody was playing. The sales for the game of Go in online stores skyrocketed to the point where the stock for the game was gone. That s how big the event was one and a half years ago. Everybody was watching, the whole world was watching, the whole world of the East was Show Notes: http://www.superdatascience.com/110 8
watching. And now, they just casually created an algorithm that outperformed not just that version, outperformed the next version and that is all through self-training. This happened in one and a half years. That s what blows my mind, how big of an accomplishment it was one and a half years ago and now we ve surpassed that by a million times or like many, many, times past that and it s only been one and a half years. That s the rate of technological change, that s the rate of how quickly these algorithms are developing, how quickly AI is advancing, and how quickly data science is growing. Because this is all part of data science, that whole team of DeepMind also develops algorithm for business purposes. We ll talk about that in a separate episode, but that s the world we live in now. A lot of these things happen kind of behind the curtains, luckily this was published because it s about a game and this was their whole point to popularize artificial intelligence and show the world what can be done. I highly encourage you to check out the blog, so we ll link this in the show notes, you can find them at superdatascience.com/110, or you can simply just google DeepMind AlphaGo Zero and you ll find this blog post. Yeah, that s the world we live in. Something that we don t often stop to think about. And how can these things affect your career, how can you take them into account in your career in data science and how can you consider them when planning for the future or thinking of the projects you will be working on, thinking where you ll be, we re not even talking about 10 years from now or 5 years from now. You can see this huge progression happened in one and a half years, so where will you be in one and a half years from now? Show Notes: http://www.superdatascience.com/110 9
I think that s a wrap-up, I hope you enjoyed today s episode and I look forward to seeing you here next time. Until then, happy analysing. Show Notes: http://www.superdatascience.com/110 10