As connected sensors and services become part of operational infrastructure, unusual network activity can be difficult to distinguish from legitimate changes in device behavior. IoT anomaly detection offers one way to classify that activity from data rather than relying only on known attack signatures. A comparison of five machine learning approaches found strong results on a synthetic IoT dataset, with Random Forest performing comparatively well across several attack categories. The findings are useful for understanding model selection, but they are not a substitute for validating detection performance in a live industrial environment.
Why IoT anomaly detection matters
IoT devices exchange data across networks and services, creating opportunities for disruption, unauthorized control, information exposure, or misconfiguration. The attack and anomaly categories examined in the study included Denial of Service (DoS), Data Type Probing, Malicious Control, Malicious Operation, Scan, Spying, and Wrong Setup. Normal activity was treated as an additional class, making the task a multiclass classification problem rather than a simple normal-versus-attack decision.
These categories represent different operational concerns. A DoS attack can overwhelm a target with unwanted traffic, while Data Type Probing involves writing a data type that does not match the intended type. Malicious Control and Spying can involve unauthorized access or information gathering; Malicious Operation describes activity that interferes with intended device behavior. Scan and Wrong Setup capture other forms of suspicious or disrupted activity represented in the dataset.
Signature-based detection compares activity with previously recorded patterns. That approach may be less effective against threats not already represented in its database. Data-driven classification instead learns patterns from labeled examples. It can help identify activity by its observed features, although its usefulness depends on the data and evaluation conditions.
Comparing machine learning models for IoT anomaly detection
The researchers compared Logistic Regression (LR), Support Vector Machine (SVM), Decision Tree (DT), Random Forest (RF), and Artificial Neural Network (ANN). The models were evaluated using accuracy, precision, recall, F1 score, and Receiver Operating Characteristic (ROC) analysis. These measures describe different aspects of classification: accuracy summarizes overall correctness, while precision and recall help show how predictions for a particular class compare with false alarms and missed instances. F1 combines precision and recall.
The dataset contained 357,952 records and 13 features, including source and destination information, device or service types, operation, value, and a normality label. It was based on traffic traces from a virtual IoT environment in which microservices communicated using MQTT. The data included 347,935 normal records and 10,017 anomalous records, distributed across seven anomaly types.
Before training, missing and unexpected values were handled, the timestamp feature was removed, and nominal categorical features were converted using label encoding. The data was divided into training and testing sets in an 80:20 ratio, and the models were also examined using five-fold cross-validation.
What the results showâand what they do not
Decision Tree, Random Forest, and ANN each achieved reported test accuracy of 99.4%. Logistic Regression reached 98.3%, while SVM reached 98.2%. Looking at accuracy alone, the top three appear equivalent. The class-level results, however, point to differences in how models handled individual categories. The study identified Random Forest as the strongest overall approach for this dataset, particularly for classifying several anomaly types.
Confusion-matrix results also reveal an important limitation: the top-performing models confused some DoS examples with normal activity. In the reported test results, Random Forest misclassified 403 of 1,178 DoS samples as normal, while it misclassified 18 of 69,571 normal samples as DoS. This illustrates why a high aggregate accuracy does not by itself establish that a detector will reliably catch every operationally important event. A model can perform well overall while still missing a meaningful share of a particular class.
The dataset was also heavily weighted toward normal records, and the study used synthetic data from a virtual environment. Its reported accuracy should therefore be read as a result for this dataset and test setupânot as a general performance guarantee for plant networks, other IoT architectures, or previously unseen attacks. The article reported ROC analysis, but its summary metrics do not provide numerical area-under-the-curve values for each model.
Practical considerations for automation teams
For engineers, integrators, and operations leaders evaluating machine learning intrusion detection, the comparison suggests several practical considerations:
- Evaluate per-class performance. Review confusion matrices, precision, and recall for specific threats, not just overall accuracy.
- Account for class imbalance. A large normal class can dominate aggregate results, so examine whether less frequent attack types are being detected.
- Validate against relevant data. Test with traffic and operating conditions representative of the intended environment; synthetic results alone do not establish real-time performance.
- Plan for changing behavior. IoT services may behave differently over time, and normal variation can resemble an anomaly. Detection systems need evaluation against those changes.
The comparison also shows that a complex neural network is not automatically necessary to achieve strong results on a given dataset: the Decision Tree and Random Forest performed at the same reported accuracy as ANN. That observation is specific to this experiment; model choice should still reflect the data, detection objectives, and operational requirements.
Conclusion
Machine learning provides a structured way to classify multiple types of IoT attacks and anomalies, and Random Forest performed comparatively best in this study. For industrial automation teams, the central lesson is to look beyond a headline accuracy figure. Class-level misses, dataset composition, and the gap between synthetic traffic and real operating conditions all affect how useful a detector may be. Treat these results as a basis for further evaluation, then validate candidate models with data and scenarios that reflect the system they are intended to protect.
Access Official Technical PDF