Reverse-Engineering and Fuzzing
Custom Network Protocol

OVERVIEW

NOTE: This training is a 2-day (8 hours/day) course from September 5 to 6.

The communication protocol defines the format and semantics of message exchange between applications. In modern times there are a myriad of proprietary application protocols like Skype Protocol, Dropbox Protocol, etc. which applications use to achieve various goals like bandwidth efficiency, custom encryption/compression, etc. These protocols could have security vulnerabilities. Protocol Reverse Engineering (PRE) is not only useful for offensive purposes but also used by modern Intrusion Detection Systems(IDS), they use the knowledge of protocol specification to do Deep Packet Inspection(DPI) which can enhance its capabilities, where it earlier relied just based on pattern matching which may produce lots of false positives. Custom protocols are not only used by legitimate applications but also by malware and botnets like Zeus, Emotet, etc. By reversing malware protocol you can connect to malware servers and track their campaigns.

Protocol Reverse Engineering(PRE) is an art and science of recovering the protocol specification of the obscure/proprietary protocol whose documentation is unavailable or poorly documented. There are efforts to develop automated PRE tools but they are largely academic and are not mature enough to be usable, and can’t give the accuracy a human analyst can offer. Automated tools face the challenges of heterogeneous protocol data which is often a mixture of text and binary, and it has different data types and variable-length fields and this is the reason I have created this training, it is to help you understand these challenges and learn to recover protocol specification.

This training is divided into two parts, in the first part we will learn about Protocol Reverse Engineering principles. We will look at some of the common data formats and other protocol structures and with that understanding we will write a protocol dissector using Scapy framework for a target Desktop game Minetest (open source implementation of Minecraft). Minetest is online multiplayer game in which different players can connect to the server and play with other players, there are also many public servers which you can connect and play. Once we have written the decoder we will sniff the connection and look at the communication flow between the client and the server which we will capture and re-analyze the traffic to improve the dissector further, using this newly improved dissector we will implement a custom game client/bot which will connect to the server and play as a Bot player.

In the second part, with a decent understanding of the Minetest Protocol we will move on to the offensive side of the training and try to fuzz the game server to find some security vulnerabilities, we will start with basic Fuzzer and try to do incremental improvement such that we have good code coverage. Leveraging their reverse-engineered understanding of the protocol, participants will employ Generational Fuzzing by defining the protocol specification in the Boofuzz fuzzing framework and subsequently fuzzing the application. The training will also explore Mutation Fuzzing as an alternative approach to identify potential crashes or vulnerabilities.

This hands-on game hacking training is a takes project-based learning approach, ensuring a comprehensive and practical understanding of Protocol Reverse Engineering. In summary, this training aims to equip you with the knowledge and skills to reverse engineer and understand obscure protocols, enhance IDS capabilities, and explore offensive techniques such as protocol fuzzing to uncover potential security weaknesses. The ultimate goal is to empower participants to create their own tools in the realm of protocol security.