Helping Johnny to Analyze Malware: A Usability-Optimized Decompiler and Malware Analysis User Study

Abstract

Analysis of malicious software is an essential task in computer security, it provides the necessary understanding to devise effective countermeasures and mitigation strategies. The level of sophistication and complexity of current malware continues to evolve significantly, as the recently discovered "Regin" malware family strikingly illustrates. This complexity makes the already tedious and time-consuming task of manual malware reverse engineering even more difficult and challenging. Decompilation can accelerate this process by enabling analysts to reason about a high-level, more abstract from of binary code. While significant advances have been made, state-of-the-art decompilers still produce very complex and unreadable code and malware analysts still frequently go back to analyzing the assembly code. In this paper, we present several semantics-preserving code transformations to make the decompiled code more readable, thus helping malware analysts understand and combat malware. We have implemented our optimizations as extensions to the academic decompiler DREAM. To evaluate our approach, we conducted the first user study to measure the quality of decompilers for malware analysis. Our study includes 6 analysis tasks based on real malware samples we obtained from independent malware experts. We evaluate three decompilers: the leading industry decompiler Hex-Rays, the state-of-the-art academic decompiler DREAM, and our usability-optimized decompiler DREAM++. The results show that our readability improvements had a significant effect on how well our participants could analyze the malware samples. DREAM++ outperforms both Hex-Rays and DREAM significantly. Using DREAM++ participants solved 3x more tasks than when using Hex-Rays and 2x more tasks than when using DREAM.