Optimization of large language model prompts using the DSPy framework

Abstract

Large language models are sensitive to query formulation, which is why manual prompt engineering often yields unstable and cost-inefficient results. The DSPy framework offers a programmatic approach in which the query is viewed as a parameter that can be systematically learned and optimized. The aim of the work is to examine how much DSPy automatic optimization improves the classification of malicious queries compared to manually defined queries, with a limited budget and a strict output format. For the purposes of the experiment, the GPT-4.1-nano model and the DSPy program (Signature + dspy.Predict), with the MIPROv2 optimizer, with the SIMBA and COPRO variants, were used. Compared to the baseline (macro-F1=0.248), DSPy without optimization increases the performance to 0.689, while MIPROv2 reaches 0.878 (context=empty) and 0.936 (context=true), very close to the upper bound of 0.955. The risk decreases significantly, and the format violation is ≈0%. Ablation shows that rewrite is a stronger individual contribution (0.904) than examples alone (0.885), while their combination gives the best result (0.936), with moderately higher risk. DSPy turns query design into a reproducible, measurable and cost-conscious process. Precise rule reformulation and concise domain context give the highest contribution, and combining with several examples allows to approach the upper bound of performance without modifying the model weights.