An Empirical Comparison of ML, DL, and LLM in Phishing URL Detection Across Datasets

What is it about?

Phishing attacks often rely on fake web addresses (URLs) that look almost identical to legitimate ones. Security tools use artificial intelligence (AI) to identify these malicious URLs, but many studies train and test their models on the same dataset. This makes the results look better than they actually are. In this study, we first identify 32 useful features that can be extracted directly from a URL without visiting the website. This approach makes phishing detection faster and safer because it does not rely on webpage content or external services. We then compare a wide range of AI techniques, including traditional machine learning (ML), deep learning (DL), and large language models (LLM), on two large phishing URL datasets. Instead of limiting our evaluation to a single dataset, we also test each model on a different dataset that it has never seen before. This better reflects real-world conditions, where phishing attacks constantly change. Our results show that many models achieve very high accuracy on familiar data but lose much of that performance on unseen data. Among the methods we evaluated, some DL models remained more reliable than others, while general-purpose LLMs did not consistently outperform simpler approaches.

Featured Image

Photo by Markus Winkler on Unsplash

Photo by Markus Winkler on Unsplash

Why is it important?

Many published phishing detection systems report excellent accuracy, but those numbers often come from evaluations that do not reflect real-world use. In practice, attackers constantly create new URLs with different patterns. A detection model must work on these unseen examples, not only on data that resemble its training set. Our study emphasizes this gap by evaluating models across independent datasets rather than only within the same dataset. The results show that strong performance on one dataset does not guarantee reliable performance elsewhere. This finding can help researchers design more realistic evaluations and encourage the development of phishing detection systems that remain effective as attacks evolve. The study also provides a carefully selected set of URL-based features that others can use to build phishing detection systems without relying on webpage content or external information sources. This makes the approach more practical for real-time deployment and easier for future researchers to reproduce.

Perspectives