Automating Swords & Souls training — part 1
As mentioned in the previous post, the topic of the first few articles in the AiF series will be automating the minigames in the popular Swords & Souls flash game. In this article, we will be focusing on the block training part, where our character needs to block the incoming projectiles (apples).
Problem description
The problem itself is rather simple, actually. This is exactly why we start this series with the block training. During the training process, our character is “glued” to the center of the screen, unable to move. He can only rotate himself and his shield to defend against the deadly incoming projectiles, the ultimate nemesis of doctors — the apples. These red abominations keep spawning in different positions all over the edge of the game window and then start rolling towards our character. If our character can successfully defend himself, then he gains experience and his combo goes up by one. On the other hand, should he be hit by the incoming projectile, the combo will break instantly, dropping to zero. To solve this problem, we will need to make the character recognize the apples and defend against the closest one. The following section will list some of the techniques that we could use to tackle this problem.
Possible solutions
We could use at least three different methods to solve this problem. These are the convolutional neural networks, template search and simple pixel search. Frankly speaking, while each one of them is frequently used for problems of this kind, there are still subtle differences between them as well as things to look out for when using them, so we should start by listing their advantages and disadvantages and choose the most suitable one.
CNNs are by far the best performers for this kind of problem due to the sophisticated architectures recently developed. However, they usually need a large training set to learn from and gathering all this data can be really time-consuming. Not to mention that relying purely on a CNN can produce some issues later on if the network is not validated correctly. Since we want to keep things really simple (and this data gathering and validation generates quite an overhead), we will look for another solution.
Template search (or template matching) on the other hand is really easy to implement (for example with OpenCV) but its performance leaves much to be desired in some cases. Another problem is that in this mini-game the r̶e̶d̶ ̶a̶b̶o̶m̶i̶n̶a̶t̶i̶o̶n̶s̶ apples keep rotating until they are either blocked or they hit the character. This means that simple template search (which is used for static images) will not work for this problem. We could fix this by using SIFT, SURF or KAZE, which are all more advanced template search algorithms and they theoretically all work even if the objects are rotated but sadly even though I have tried all of them, I did not manage to make any of them work for this problem.
Locating the apples by their RGB color is the simplest solution by far but as graphical objects are rarely made up of a single color (e.g. there are smooth gradients inside the object that result in different shades of the given color) we need to manually look for the correct thresholds for each channel (R, G and B). Despite this, we will use this method due to its simplicity and reliability (if the algorithm works for one apple then it will work for all the apples in the game, regardless of their positions and/or angles of rotation).
The solution — step by step
First, we need to exclude any extra information that could influence the decision making of our agent, like the UI of the browser and the background. For this, we will simply crop the screenshot so that only the game window remains. To achieve this, we will ask the user to manually select the desired window at the start of our script.
Now we can finally look for the apple pixels by their RGB color inside this cropped window.
Now that we know where the apples are, we can start looking for the closest one. This means that we will loop over all the red pixels and select the one that is the closest to our character (using euclidean distance).
After this, if there are any red pixels, we will move our mouse cursor so that it points at the location of the closest one and we are practically done.
The following GIF demonstrates the thinking of our agent. The green pixels indicate the character, while the red pixels show the apples’ positions and the closest apple pixel is indicated by the blue pixels.
Results
Now let’s have a look at the results! Keep in mind that we did not implement any “catch the stars” mechanism. The reason for this is that there is practically no need to do so as the agent will catch the stars even without being explicitly told so due to the speed of the incoming projectiles. Just look at how we went from mission #7 to mission #65!
Now those are some epic results! More than 13K points into the defensive stat and a lot of level-ups. Quite insane if I may say so. At this point, our lovely character is basically somewhere around the level of a T-34 tank. Or more precisely a T-34 without a turret and barrel as his damage is sadly quite low right now. But fear not as we will tackle all the mini-games in the upcoming posts!
For those that would like to try out this script or simply experiment with automating the block training in Swords & Souls, the whole code can be found on GitHub (https://github.com/automatingisfun/SnSBlock).
You can also check out the YouTube video that corresponds to this post if you are more of a visual type at https://youtu.be/puAV_16aULU.
See you in the next post!
Disclaimer
Automating is Fun (AiF) does not approve or encourage hacking in any way. The series is made purely for educational purposes. That is exactly why we will only automate games or mini-games that cannot cause the owner of the product or other players any direct or indirect damage.
