Nowadays, one of the most critical cyber security problems is the fight against malicious software, precisely, the problem of detecting it. Every year, new modern computer viruses are created that are capable of mutation and changing while running. But unfortunately, the developers of antivirus software do not have time to quickly add all types of malicious programs to the signature databases. In this regard, it is sensible to use heuristic detection methods based on algorithms of machine learning. The purpose of this paper is to present several classification methods based on machine learning techniques for detecting zero-day attacks. In particular, the following algorithms were tested: random forest classifier, support vector classifier, greed search in svc, and k-nearest neighbors. The dataset was taken from the Kaggle website. It consists of 19611 executable files of the PE format, 14599 of which are malicious, and 5012 files are benign. This article presents recommended classification and detection methods with advanced analysis of important metrics that allow you to assess and compare machine learning algorithms’ effectiveness and performance for detecting malware.