Developing models for detecting prompt injection attacks in language models

Abstract

Prompt injection attacks represent one of the most critical security challenges in modern systems based on large language models, as they enable malicious users to bypass constraints and manipulate model behavior. This paper investigates the applicability of pretrained BERT-based language models for detecting prompt injection attacks after fine-tuning.

The experimental study includes fine-tuning and evaluation of several transformer-based models on three datasets: a publicly available Kaggle dataset used for training, an independent test dataset obtained from the Hugging Face platform, and a manually constructed dataset of Serbian-language prompts. Model performance was assessed using standard binary classification metrics, with particular emphasis on recall and F1 score due to class imbalance and the security-sensitive nature of the task.

The results indicate limited generalization performance on the independent Hugging Face dataset, suggesting sensitivity to dataset distribution and prompt formulation differences. In contrast, significantly better results were achieved on the Serbian dataset, demonstrating the models’ ability to adapt effectively to language-specific characteristics after fine-tuning. Among the evaluated models, MiniLM-L6 achieved the most favorable balance between precision and recall, while additional hyperparameter optimization resulted in only marginal improvements. These findings highlight the importance of diverse and representative datasets and the need for further research on improving generalization in prompt injection detection across different languages and domains.