The code for this project can be found in its repository on my GitHub page.
Metin2 was one of the first MMORPGs (Massive Multiplayer Online Role-Playing Games) on the market when it was launched in 2005. I started playing the game at age twelve in 2008 and it continues to be one of my all-time favorite games to this day.
The story of the game, as described on its website, is rather simple:
The appearance of the Metin stones has torn deep rifts through the once flourishing world of the Dragon God. War has erupted between the kingdoms – wild animals have transformed into fearsome beasts and the dead have arisen to bloodthirsty, demonic life. Defend your kingdom and hurl yourself into incredible battles full of action and magic.
How exciting this description might seem at first, improving the equipment of one’s game character and leveling up through the ranks often comes down to repetitive hunting of animals and evil creatures to collect their loot. This task quickly becomes tedious and repetitive, so as a teenager I wrote scripts that automated these parts of the game. However, because scripting languages back then weren’t as powerful as they are today (Python, I’m looking at you!), and because it was only the beginning of my software engineering journey, my scripts weren’t that sophisticated.
This year, however, I decided to dedicate some of my free time during the Christmas holidays to revisit my former passion of writing Metin2 bots – but this time with advanced knowledge in scripting, automation and machine learning. I downloaded the game client of a private server of the game and began working. For the Christmas season, the private server replaced the models of the Metin stones with snowmen – what a festive change.
My goal was to write a program that would hit snowmen and collect their loot automatically, moving to a new one whenever the old one is destroyed. This idea involved multiple subparts that had to be solved individually:
- Detecting the snowmen visually
- An intelligent agent that decides on actions based on inputs
- Interacting with the GUI of the game client
Snowman Object Detection
To be able to click on snowmen automatically, the program has to first recognize and locate them. To do that, I used OpenCV’s Python library to take screenshots of the game client, apply an HSV color filter, and then pass it through a machine learning model to actually detect the location of the objects.
OpenCV provides the implementation of Haar feature-based Cascade classifiers. These machine learning models learn through training which features are most important to detect the desired object. Features can be one of three types of compositions of black and white pixels (see below), with variable pixel sizes of each feature.
However, a plethora of features is possible. The authors of the original paper calculated that for an even 24 x 24 pixels small image, 160,000 different features are possible. The model finds clever ways to learn the most important ones from training data. More on the workings of the model can be found on the OpenCV documentation.
I went ahead and labeled snowmen in my training data for the Cascade classifier. Negative samples (so-called ‚background images‘) were created by saving images of the game client without any snowman, and by programmatically cutting out matches from positive images. After two hours of labelling effort and more than two hundred positive samples, the model was trained and I found the performance to be excellent!
Given that Convolutional Neural Nets (CNNs) are state-of-the-art for computer vision and object detection nowadays, I was surprised that even a rather simple CV machine learning model like Cascade classifiers, which were mostly designed for resource-constrained real-time face detection, yielded such positive results. I contribute the strong model performance to good HSV filtering in the preprocessing step, as the filter pretty much singled out snowmen.
Intelligent agent
The actual bot was implemented using a state machine, which runs in a separate thread parallel to the image capture and object detection thread. Farming Metins, or in this case snowmen, was divided into the individual tasks of searching for matches, validating matches, moving to a snowman, hitting the snowman, finally collecting its loot and then starting over. Furthermore, when automating graphical user interfaces, one has to occasionally rely on sleep timers to allow for the UI to update in time. I implemented certain fail-safe measures like rotating the view when no match could be found, recalibrating the camera in case it changed, and teleporting back to the original position if no snowmen can be found repeatedly.
Interacting with the game UI
Obviously but unfortunately, the game does not offer an API to send commands to game characters. Making matters worse, elaborate hack shield have been introduced in an effort to prevent hacks and bots. Therefore, GUI automation is the last frontier that can be used to imitate player input, which is generally proof against any software-based protection measures.
The screen was captured using Python’s Windows API, clicks were executed using Python’s PyAutoGUI library and key strokes were sent using Windows‘ on screen keyboard, as the hack shield prevented programmatic key strokes to be sent to the game client. This pipeline perfectly replicates human perception and interaction with the game.
Final result
The video above shows a close to final version of the bot. Snowmen were detected largely without any issues by bot. After some bugs were fixed, sleep timers were reduced to a minimum and additional fail-safe and reset mechanisms were implemented, the bot ran magnificently for hours and hours. To be able to monitor the bot remotely, e.g. while watching Christmas movies in the living room, I set up a Telegram messenger bot to notify me whenever a snowman was defeated. I quickly realized that the bot can run for very long time without me, so I tested the extremes.
I let it run for as long as I could afford not to use my computer, and after more than 14 hours and roughly 3500 defeated snowmen, I decided to end the experiment successfully. Overall, I had loads of fun building the project, learned a lot about OpenCV and object detection in general, and gained valuable experience in scripting with Python. Most of all however, I enjoyed the nostalgia of delving into one of my childhood games and enjoyed sweet victory of my technical skills over obstacles from the past.
Merry Christmas and a happy new year 2021!