In the near future, objects have to connect with each other which can result in gathering private sensitive data and cause various security threats and cyber crimes. To prevent cyber crimes, novel cyber security techniques are required that can identify malicious Internet Protocol (IP) addresses before communication. One of the best techniques is the IP reputation system used for profiling the behavior of security threats to the cyber–physical system. Existing reputation systems do not perform well due to their high management cost, false-positive rate, consumption time, and considering very few data sources for claiming IP address reputation. To overcome the aforementioned issues, we have proposed a novel hybrid approach based on Dynamic Malware Analysis, Cyber Threat Intelligence, Machine Learning (ML), and Data Forensics. Using the concept of big data forensics, IP reputation is predicted in its pre-acceptance stage and its associated zero-day attacks are categorized via behavioral analysis by applying the Decision Tree (DT) technique. The proposed approach highlights the big data forensic issues and computes severity, risk score along with assessing the confidence and lifespan simultaneously. The proposed system is evaluated in two ways; first, we compare the ML techniques to attain the best F-measure, precision and recall scores, and then we compare the entire reputation system with the existing reputation systems. Our proposed framework is not only cross checked with external sources but also able to reduce the security issues which were neglected by existing outdated reputation engines.