Introduction

Modern computer networks are suffering from an unprecedented escalation in both the volume and complexity of cyberattacks. Zero-day exploits, which leverage previously unknown vulnerabilities, are a significant challenge since they naturally evade traditional IDS models reliant on labeled datasets or predefined attack signatures. As networks evolve, these systems are unable to generalize to new traffic patterns and unseen attack vectors.

Deep learning has emerged as a powerful tool for modeling complicated and nonlinear data distributions. Within the domain of deep learning architectures, AEs are a prominent approach due to their unsupervised feature-learning ability. Training on normal network behaviour, an AE learns to reconstruct benign traffic accurately. Therefore, any deviation from learned patterns develops into a higher reconstruction error that indicates an anomaly.

This paper reviews recent work that leverages deep autoencoders for unsupervised anomaly detection, putting a special emphasis on their zero-day detection capability. The contribution of this review is threefold:

1. It systematizes recent works, between years 2020 and 2025, in a taxonomy of AE architectures utilized in IDS research.

2. It makes a critical comparison among the detection performance, datasets, and evaluation methodologies.

3. It highlights the main gaps in the prevailing research and further suggests directions for exploration.

Background

Anomaly Detection in Network Traffic

Network anomaly detection involves identifying traffic behavior that is somehow different from a model of "normal" activity. Broadly speaking, approaches fall into one of the following categories:

• Signature-Based Detection: This form of detection relies on known attack patterns.

• Supervised learning: trains classifiers on labeled benign and malicious samples.

• Unsupervised learning: It builds a model of normal traffic distribution and flags deviations from it as anomalous.

In particular, unsupervised methods have the further appeal of being immediately suited to zero-day attacks insofar as they do not rely on attack data that is labelled.

Autoencoders for Unsupervised Learning

It contains an encoding part that compresses the input, $x$, into a latent representation, $z$, and a decoder that reconstructs, $x^{\prime }$, from $z$. The model minimizes reconstruction loss, $L(x,x^{\prime })$, typically mean squared error. When this is trained exclusively on normal traffic, the reconstruction errors for anomalous traffic increase drastically.

Other variants, such as CAE, VAE, and RAE, enhance feature extraction by exploiting spatial or temporal dependencies.

Evaluation Metrics

The evaluation of unsupervised intrusion detection systems needs intensive consideration when choosing evaluation metrics, especially when dealing with imbalanced classes or zero-day attacks. Anomaly detection based on reconstruction, contrasting supervised classification, uses thresholding on anomaly scores instead of predictions.

Accuracy, though frequently cited, might be misleading because the prevalence of normal traffic strongly dominates the results. Therefore, metrics related to detection reliability are preferred.

"Precision": the proportion of detected anomalies that are malicious, directly reflecting the quality of the alert and the analysis work.

“Recall” (also Detection Rate) is a measure of the system’s capability to detect true attacks and has become a key aspect for zero-day threat containment.

F1-score, being the harmonic mean of precision and recall, is more robust against class imbalance than precision or recall.

A high False Positive Rate (FPR) is very important for operational IDS systems, where too many false alerts can cause the system to be considered ineligible. A few recent works have reported the results of FPR at certain points of recall, such as FPR@TPR = 0.9.

Area Under the ROC Curve (AUC) measures ranking performance irrespective of the threshold values, and it has been widely used in the test of general ability. AUC does not measure the ability of the model deployment.

In reconstruction models, reconstruction error distributions can directly serve as anomaly scores; threshold strategies, whether static, percentile, or adaptive, have a large impact on results reported. Validation of zero days requires more advanced forms of evaluation, including attack family holdout evaluation, time splitting, and cross dataset validation, which avoid information leakage and enable realistic assessment of generalization, as illustrated in [31, 42].

Taxonomy of Autoencoder Architectures

Comparison of Recent Research

In the last five years, research on unsupervised DAE for zero-day anomaly detection in network traffic has accelerated significantly. Several scholars have researched a wide spectrum of architectural enhancements, hybrid learning strategies, and evaluation protocols to improve the generalization capability of IDS models.

The following section critically summarizes and compares the findings of fifteen key contributions from the period between 2020 and 2025. These papers have been selected based on their relevance, methodology, quality of datasets considered, and unsupervised/semi-supervised anomaly detection concepts involving an autoencoder-based model.

Overview of Important Related Research

Narrative Summary for Remaining Papers:

A few more studies, which have not been included in above table, support such views. Some of these studies include [8, 14, 16, 17, 20, 24-27, 30, 33-38, 40, 41, 43] which deal with architectural optimization, threshold adaptation, drift issues, adversarial abilities, and online learning, respectively. Even though they achieve very good detection accuracy, most of these studies employ randomly chosen training and testing sets, which tend to be biased and render zero-day performance poor and unreliable in actuality. Recent book reviews, such as [12] and [45], establish this view and establish a taxonomy along these lines.

Comparative Insights

A close comparison of the aforementioned forty-five reviewed studies highlights some consistent and statistically significant trends across autoencoder-based intrusion detection research.

1. Hybrid architectures have always provided better robustness and enhanced precision. It is shown in a large fraction of recent works that the integration of autoencoder-based representation learning with secondary decision mechanisms—be it clustering algorithms, support vector machines, or ensemble voting—significantly enhances anomaly discrimination and reduces the false-positive rates. Hybrid AE–SVM, AE–clustering, and ensemble-based frameworks obtain far more stable decision boundaries in the latent space compared to standalone reconstruction-based autoencoders under heterogeneous and class-imbalanced traffic conditions.

2. Temporal modeling yields better detection for streaming and sequential traffic. Explicitly modeling temporal dependencies by using LSTM-based auto-encoders, CNN-RNN hybrids, and transformer-based architecture consistently outperforms static dense auto-encoders in detecting low-frequency and stealthy attacks. Such methodologies explicitly modeling temporal dependencies are highly effective for real-time monitoring of attack behaviors, which evolve over time rather than as single point events.

3. Correlation-aware and structured latent learning improves generalization. Meanwhile, several works emphasize the significance of maintaining relationships between features in latent-space learning. Autoencoders that utilize kernels, mutual information constraints, graph models, and contrastive learning improvements give better results compared to traditional mean squared error-based autoencoders due to their ability to model correlated and relational patterns in traffic, which often signal either coordinated or multi-stage attacks.

4. Zero-day evaluation practices remain inconsistent across the literature. On the contrary, despite the high reported accuracy in detection, only a few of these reviewed works use some strict zero-day evaluation strategies such as attack-family holdout, cross-dataset testing, or time-based splits. Most of these reviewed studies rely on random data partitioning. This means that during training, there is unintended leakage that exposes their models to the general attack characteristics, thereby inflating their performance metrics. This inconsistency underlines the requirement of standardized benchmarking protocols that can actually evaluate the true zero-day generalization.

5. Attention and transformer-based autoencoders represent a growing research frontier. Some recent works have increasingly integrated attention mechanisms and transformer encoders into autoencoder frameworks to enhance both detection accuracy and interpretability, including [7], [15], [32] [39], and [44]. These models dynamically emphasize salient traffic features with limited explanatory insights into anomaly decisions, thus becoming particularly appropriate for IoT, cloud, and 5G environments with high-dimensional and non-stationary traffic.

6. Architectural choices are based on practical deployment considerations. While complex architectures achieve high detection performance, several studies emphasize trade-offs between accuracy, computational overhead, and real-time feasibility. Lightweight, quantized, and pruned autoencoders ([22], [29], [34]) are favored for edge and IoT deployments, whereas deeper temporal and transformer-based models are more suitable for centralized or cloud-based IDS infrastructures.

Critical Evaluation

While most recent autoencoder-based intrusion detection research reports detection accuracies of over 98%, a closer look at their experimental approach indicates that such metrics often present an inflated view of real-world performance. In many cases, this involves random train–test splits. Such a method inadvertently exposes the model to attack traffic characteristics during training. Only a few studies employ proper zero-day validation strategies, such as excluding attack families, time-aware splitting, or cross-dataset testing ([1], [6], [15], [31], [42]). Cross-dataset evaluation, such as training on CICIDS2017 and testing on UNSW-NB15 or TON-IoT, remains rare despite offering the most realistic assessment of generalization capability.

One of the persistent and widely reported limitations in the reviewed literature is the trade-off between detection sensitivity and false-positive rates. The deep autoencoders, while being trained on exclusively benign traffic, tend to overfit normal data distributions and may reconstruct subtle or slowly evolving attacks with low reconstruction error. This behavior leads to increased false negatives or forces practitioners to adopt conservative thresholds, thereby increasing false-positive rates and reducing operational usability. Hybrid frameworks that embed clustering, SVMs, ensemble voting, or adaptive thresholding have partially mitigated this issue but at the cost of increased complexity and further tuning requirements.

Interpretability remains another significant weakness of current autoencoder-based intrusion detection. The majority of the reviewed studies consider the autoencoder as a black-box model that provides very limited insight into which traffic features or latent representations contribute to decisions about anomalies. Only a few recent works ([7], [32], [44]) try to fill this gap by using attention mechanisms, feature attribution techniques, or disentangled latent spaces. In the absence of explainability, analyst trust is significantly degraded, as well as forensic investigation and regulatory compliance, in critical sectors such as finance, health, and industrial control systems.

From the deployment viewpoint, most of the top-performing architectures - including deep LSTM autoencoders and transformer-based variational autoencoders - bear high computation and memory overheads as per [15, 39, 41]. In this regard, such models will be unsuitable for real-time detection at the edge of the network or in resource-constrained IoT settings. On the contrary, lightweight and quantized autoencoders, such as those proposed in [22, 29, 34], tend to be efficient with degraded detection accuracy and adaptability.

In an overall sense, this critical review of benchmark accuracy demonstrates that mere high values are not enough to achieve real-world zero-day intrusion detection. From now on, any future autoencoder-based IDS research should stress the issues related to zero-day evaluations, sensitivity and specificity trade-offs, explainability, and computational efficiency. Overcoming these challenges is crucial for the future development of adaptive and trustworthy intrusion detection systems that work reliably in large-scaled dynamic network environments.

Discussion and Research Trends

Evolution of Unsupervised Deep Autoencoders in IDS

From 2020 to 2025, there has been a notable shift in focus from simple feed-forward autoencoders to hybrid and attention-enhanced architectures. Early works ([12], [14]) developed dense AEs for dimensionality reduction and unsupervised anomaly detection. By 2022–2025, hierarchical, variational, and transformer-based architectures ([9], [15]) that could model nonlinear and context-dependent traffic features started to attract attention.

The primary motivation for this evolution is the need for zero-day robustness—ensuring IDS systems can detect anomalies unseen during training through probabilistic modelling in VAE, temporal context in LSTM-AE, and feature correlation in kernelized AEs.

Architectural Innovation Trends

1. Variational and Probabilistic AEs: VAEs model the distributions of the latent space, enabling probabilistic thresholds of anomaly scores. The most recent work in this paper by Khalaf et al. [15] extended VAEs with transformer encoders, achieving better results in IoT environments with non-stationary data streams.

2. Hybrid AE Frameworks: Sharper decision boundaries in the latent space are obtained when autoencoders are integrated with discriminative classifiers, such as LS-SVMs [3] or clustering modules [8]. Such models are computationally heavier but deliver better separation between normal and abnormal samples.

3. Temporal and Attention-Based Models: Sequential dependencies in traffic data are efficiently captured by LSTM-AEs [4] and CNN-RNN hybrids [9]. Attention mechanisms further enhance these models by assigning adaptive weights to features most relevant for anomaly identification [7].

4. Latent Learning with Correlation Awareness: Roy & McNeely [5] enforced mutual-information constraints within the loss function, which granted the guarantee that the encoder would preserve the inter-feature correlations that are usually ignored when using a reconstruction-only objective.

Dataset and Evaluation Practices

This is a recurring problem with most of the studies: their lack of uniformity in evaluation protocols.

• Most of them have used popular datasets such as CICIDS2017, NSL-KDD, and UNSW-NB15, which are already acknowledged to possess redundant or outdated attack types.

• Only a few papers include newer IoT and 5G datasets such as TON-IoT (2023) or Edge-IIoT (2024), which can better represent modern network infrastructures.

• True zero-day validation (that is, excluding whole categories of attack from training) is done inconsistently.

This will be future work; cross-dataset testing should be performed along with time-based splits to ensure temporal generalization. The recent benchmarking proposals combine datasets, such as training on CICIDS2017 and testing on UNSW-NB15, to mimic real-world deployment scenarios.

Industrial and Real-World Applicability

While real-time detection remains one of the big challenges,

• Temporal models - the LSTM/Transformer AEs offer accuracy but at higher computational costs.

• Lightweight AEs can run on either edge or IoT devices and have poor precision.

It is important to balance latency with computational cost and the reliability of detection in deploying AE-based IDSs in production systems.

In domains such as finance, healthcare, and IoT, explainability is also crucial. Organisations require explanations for why a packet or flow was classified as anomalous. Their application to AEs represents an area of emerging research where visualization tools and feature-attribution methods (e.g., SHAP or gradient-based saliency) are adapted to AEs in order to bridge this gap.

Directions for Future Research

Several exciting directions have emerged recently:

• Adversarially Robust Autoencoders: Integrate adversarial training to make them resistant to evasion attacks by leveraging the vulnerabilities of the models themselves.

• Self-Supervised Learning: Using pretext tasks to enhance feature representations without labeled data.

• Federated Learning for IDS: The training of distributed AE models in different organizations while preserving the privacy.

• Explainable Autoencoders (XAE): The proposal of interpretable latent factors to make the explanation of anomalies transparent to the analysts.

Taken together, these approaches portend a direction toward explainable, trustworthy, and decentralized intrusion detection systems.

Open Challenges and Future Directions

Despite the remarkable achievements made so far, a number of challenges have not been resolved in autoencoder-based zero-day detection systems. Careful understanding of these challenges will usher in the next phase of research and practical deployment.

Insufficient True Zero-Day Testing

Most of the published works make use of random data splits, mixing the known attack traces inadvertently into both training and test sets, inflating accuracy and failing to reflect conditions that are realistic for zero-day.

Future work: standardize attack-family holdout protocols and conduct more cross-dataset testing, such as training on CICIDS 2017 and testing on UNSW-NB15/TON-IoT, to estimate true generalization.

Overfitting and Poor Generalization

Large deep autoencoders can memorize normal traffic patterns, allowing the reconstruction of malicious flows with low error.

Future direction: Introduce regularization that includes dropout, sparsity, or variational noise, introducing domain adaptation techniques so that the models stay robust when any change in network topology or protocol mix arrives.

Explainability and Interpretability

Most of the AE-based IDSs are black boxes in their behavior. Analysts want interpretable results to understand the alerts.

Future direction: Develop XAEs, which attribute the latent variables to specific network features, and provide visualizations of reconstruction discrepancies using SHAP, Grad-CAM, or latent attention heatmaps.

Adversarial Robustness

Attackers can craft adversarial packets that bypass anomaly detectors by slightly perturbing features.

Future direction: Use adversarial training and robust loss formulations-e.g., min-max objectives-to resist such evasion strategies.

Computational Efficiency and Deployment

Complex architectures such as LSTM-AE and Transformer-VAE yield high accuracy but are inappropriate for edge or real-time environments.

Future direction: design lightweight, quantized, or pruning-based AEs deployable on routers, IoT gateways, or SDN controllers without dependence on any GPUs.

Data privacy and federated learning

Centralized IDS training violates data-sharing regulations quite often.

Future direction: utilize Federated AEs in which multiple organizations collaborate in training of the local models and share only the gradients, ensuring confidentiality with generalization enhancement.

Standardization of Assessments

Disparate metrics and inconsistent baselines impede comparison.

A proposed future direction: use unified benchmarks-report AUC, FPR@TPR=0.9, and latency-release reproducible code and models, ensure transparency.

Summary Highlights

• The prominent architectures: include CNN-AE, VAE, LSTM-AE, and Hybrid AE-SVM models that dominate the literature of 2020–2025. An investigation on transformer-based and graph-structured autoencoders should be pursued where there is a complex topology.

• Datasets: CICIDS 2017 and UNSW-NB15 remain the norm. TON-IoT and Edge-IIoT are two new state-of-the-art datasets for IoT/5G networks.

• Performance: Most models report 98% under relaxed conditions. Enforce genuine zero-day validation and cross-dataset testing.

• Explainability: Emerging attention-based AEs improve transparency slightly. Need interpretable latent features and visualization tools.

• Deployment: High-accuracy models are often too heavy for real-time usage. Light-weight and energy-efficient AEs shall be developed targeting edge environments.

• Future Research: Integration of AEs with federated, self-supervised, and adversarially-robust training frameworks. Multimodal AEs fusing network, host, and behavioral data.

These insights align directly with the Ph.D. research focus of Kajal Patel: extending unsupervised deep autoencoder frameworks that generalize across datasets and detect truly unseen zero-day threats.

Conclusion

Unsupervised deep autoencoders have indeed emerged as one of the most promising paradigms towards zero-day anomaly detection. Theoretically, they are capable of modeling normal network behavior without labeled data and revealing unknown attacks. Significant architectural advances have been made over the last five years: from vanilla dense AEs to hybrid, probabilistic, and attention-driven models. However, there are still a number of gaps that remain to be bridged in realistic zero-day evaluation, interpretability, adversarial defense, and deployment efficiency. Cross-domain benchmarking, explainable latent representations, and privacy-preserving federated learning are crucial directions in future IDS research. Addressing these challenges will enable trustworthy, adaptive, and scalable intrusion detection systems fit for next-generation network infrastructures.