A hybrid technique to detect botnets, based on P2P traffic similarity

The botnet has been one of the most common threats to the network security since it exploits multiple malicious codes like worm, Trojans, Rootkit, etc. These botnets are used to perform the attacks, send phishing links, and/or provide malicious services. It is difficult to detect Peer-to-peer (P2P) botnets as compare to IRC (Internet Relay Chat), HTTP (HyperText Transfer Protocol) and other types of botnets because of having typical features of the centralization and distribution. To solve these problems, we propose an effective two-stage traffic classification method to detect P2P botnet traffic based on both non-P2P traffic filtering mechanism and machine learning techniques on conversation features. At the first stage, we filter non-P2P packages to reduce the amount of network traffic through well-known ports, DNS query, and flow counting. At the second stage, we extract conversation features based on data flow features and flow similarity. We detected P2P botnets successfully, by using Machine Learning Classifiers. Experimental evaluations show that our two-stage detection method has a higher accuracy than traditional P2P botnet detection methods.