Publications
2026
-
Targeting Alignment: Extracting Safety Classifiers of Aligned LLMs
, , , ,
IEEE Conference on Secure and Trustworthy Machine Learning (SaTML), 2026Abstract: Alignment in large language models (LLMs) is used to enforce guidelines such as safety. Yet, alignment fails in the face of jailbreak attacks that modify inputs to induce unsafe outputs. In this paper, we introduce and evaluate a new technique for jailbreak attacks. We observe that alignment embeds a safety classifier in the LLM responsible for deciding between refusal and compliance, and seek to extract an approximation of this classifier: a surrogate classifier. To this end, we build candidate classifiers from subsets of the LLM. We first evaluate the degree to which candidate classifiers approximate the LLM's safety classifier in benign and adversarial settings. Then, we attack the candidates and measure how well the resulting adversarial inputs transfer to the LLM. Our evaluation shows that the best candidates achieve accurate agreement (an F1 score above 80%) using as little as 20% of the model architecture. Further, we find that attacks mounted on the surrogate classifiers can be transferred to the LLM with high success. For example, a surrogate using only 50% of the Llama 2 model achieved an attack success rate (ASR) of 70% with half the memory footprint and runtime -- a substantial improvement over attacking the LLM directly, where we only observed a 22% ASR. These results show that extracting surrogate classifiers is an effective and efficient means for modeling (and therein addressing) the vulnerability of aligned models to jailbreaking attacks.
@inproceedings{noirot_ferrand_targeting_2026, author = {Jean-Charles {Noirot Ferrand} and Yohan Beugin and Eric Pauley and Ryan Sheatsley and Patrick McDaniel}, booktitle = {IEEE Conference on Secure and Trustworthy Machine Learning (SaTML)}, doi = {10.48550/arXiv.2501.16534}, month = {March}, title = {{Targeting} {Alignment}: {Extracting} {Safety} {Classifiers} of {Aligned} {LLMs}}, year = {2026} } -
Adversarial Agents: Black-Box Evasion Attacks with Reinforcement Learning
, , , , ,
Findings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2026Abstract: Attacks on machine learning models have been extensively studied through stateless optimization. In this paper, we demonstrate how a reinforcement learning (RL) agent can learn a new class of attack algorithms that generate adversarial samples. Unlike traditional adversarial machine learning (AML) methods that craft adversarial samples independently, our RL-based approach retains and exploits past attack experience to improve the effectiveness and efficiency of future attacks. We formulate adversarial sample generation as a Markov Decision Process and evaluate RL's ability to (a) learn effective and efficient attack strategies and (b) compete with state-of-the-art AML. On two image classification benchmarks, our agent increases attack success rate by up to 13.2% and decreases the average number of victim model queries per attack by up to 16.9% from the start to the end of training. In a head-to-head comparison with state-of-the-art image attacks, our approach enables an adversary to generate adversarial samples with 17% more success on unseen inputs post-training. From a security perspective, this work demonstrates a powerful new attack vector that uses RL to train agents that attack ML models efficiently and at scale
@inproceedings{domico2026adversarialagents, author = {Kyle Domico and Jean-Charles {Noirot Ferrand} and Ryan Sheatsley and Eric Pauley and Josiah Hanna and Patrick McDaniel}, booktitle = {Findings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)}, doi = {10.48550/arXiv.2503.01734}, title = {Adversarial Agents: Black-Box Evasion Attacks with Reinforcement Learning}, year = {2026} } -
Longitudinal Analyses of SAST Tools: A CodeQL Case Study
, , ,
In Submission, 2026Abstract: Open-source software (OSS) pipelines rely on automated static analysis tools to prevent the introduction of vulnerabilities in code. However, there is limited understanding of the efficacy of these tools across the OSS ecosystem over time. In this paper, we introduce a novel method to evaluate static application security testing (SAST) tools through longitudinal measurements and perform the largest academic study of CodeQL---the most prevalent static analysis tool from GitHub---on OSS codebases. We apply our apparatus on 114 versions of CodeQL over time on 3993 CVEs from 1622 repositories to measure key properties of the tool, culminating in more than 20 billion lines of code analyzed. First, we measure its effectiveness, i.e., its ability to detect vulnerabilities before they are fixed. Then, we determine whether these detections were actionable through two measures of the distance between findings and vulnerability location either over the entire codebase or within the vulnerable file. Finally, we study the stability of CodeQL by examining how vulnerability detections hold across versions and the evolution of CodeQL on the accuracy-precision trade-off. We find that CodeQL identifies a total of 171 CVEs, and that for 83 of them, a CodeQL version prior to the fix could detect it. Such detections are in general actionable if findings are triaged across files, as for 50% of the 171 detections, more than 50% of findings in the vulnerable file are located in the vulnerable location. Finally, we show that CVE detections are not monotonic across versions as 21 CVEs were no longer detected following a version change and 17 that were never redetected. Our study shows that using SAST tools is a matter of best practice as they prevent numerous vulnerabilities from being introduced, but that developers should be aware of changes that may leave blind spots in detections upon updates of the tool.
@misc{ferrand2026longitudinalanalysessasttools, archiveprefix = {arXiv}, author = {Jean-Charles {Noirot Ferrand} and Kyle Domico and Yohan Beugin and Patrick McDaniel}, doi = {10.48550/arXiv.2605.07900}, eprint = {2605.07900}, primaryclass = {cs.CR}, title = {Longitudinal Analyses of SAST Tools: A CodeQL Case Study}, year = {2026} } -
The Role of Learning in Attacking Intrusion Detection Systems
, ,
In Submission, 2026Abstract: Recent work on network attacks have demonstrated that ML-based network intrusion detection systems (NIDS) can be evaded with adversarial perturbations. However, these attacks rely on complex optimizations that have large computational overheads, making them impractical in many real-world settings. In this paper, we introduce a lightweight adversarial agent that implements strategies (policies) trained via reinforcement learning (RL) that learn to evade ML-based NIDS without requiring online optimization. This attack proceeds by (1) offline training, where the agent learns to evade a surrogate ML model by perturbing malicious flows using network traffic data assumed to be collected via reconnaissance, then (2) deployment, where the trained agent is used in a compromised device controlled by an attacker to evade ML-based NIDS using learned attack strategies. We evaluate our approach across diverse NIDS and several white-, gray-, and black-box threat models. We demonstrate that attacks using these lightweight agents can be highly effective (reaching up to 48.9% attack success rate), extremely fast (requiring as little as 5.72ms to craft an attack), and require negligible resources (e.g., 0.52MB of memory). Through this work, we demonstrate that future botnets driven by lightweight learning-based agents can be highly effective and widely deployable in diverse environments of compromised devices.
@misc{domico2026rolelearningattackingintrusion, archiveprefix = {arXiv}, author = {Kyle Domico and Jean-Charles {Noirot Ferrand} and Patrick McDaniel}, doi = {https://doi.org/10.48550/arXiv.2602.10299}, eprint = {2602.10299}, primaryclass = {cs.CR}, title = {{The} {Role} of {Learning} in {Attacking} {Intrusion} {Detection} {Systems}}, year = {2026} }
arXiv
DOI
Slides
Poster2025
-
LibIHT: A Hardware-Based Approach to Efficient and Evasion-Resistant Dynamic Binary Analysis
, , , , ,
Workshop on Software Understanding and Reverse Engineering (SURE), 2025Abstract: Dynamic program analysis is invaluable for malware detection, debugging, and performance profiling. However, software-based instrumentation incurs high overhead and can be evaded by anti-analysis techniques. In this paper, we propose LibIHT, a hardware-assisted tracing framework that leverages on-CPU branch tracing features (Intel Last Branch Record and Branch Trace Store) to efficiently capture program control-flow with minimal performance impact. Our approach reconstructs control-flow graphs (CFGs) by collecting hardware generated branch execution data in the kernel, preserving program behavior against evasive malware. We implement LibIHT as an OS kernel module and user-space library, and evaluate it on both benign benchmark programs and adversarial anti-instrumentation samples. Our results indicate that LibIHT reduces runtime overhead by over 150× compared to Intel Pin (7× vs 1,053× slowdowns), while achieving high fidelity in CFG reconstruction (capturing over 99% of execution basic blocks and edges). Although this hardware-assisted approach sacrifices the richer semantic detail available from full software instrumentation by capturing only branch addresses, this trade-off is acceptable for many applications where performance and low detectability are paramount. Our findings show that hardware-based tracing captures control flow information significantly faster, reduces detection risk and performs dynamic analysis with minimal interference.
@inproceedings{zhao_libiht_2025, address = {Taipei, Taiwan}, author = {Changyu Zhao and Yohan Beugin and Jean-Charles {Noirot Ferrand} and Quinn Burke and Guancheng Li and Patrick McDaniel}, booktitle = {Workshop on Software Understanding and Reverse Engineering (SURE)}, doi = {10.1145/3733822.3764670}, keywords = {workshop}, month = {October}, title = {{LibIHT}: A Hardware-Based Approach to Efficient and Evasion-Resistant Dynamic Binary Analysis}, year = {2025} } -
On the Robustness Tradeoff in Fine-Tuning
, , , , , ,
IEEE/CVF International Conference on Computer Vision (ICCV), 2025Abstract: Fine-tuning has become the standard practice for adapting pre-trained models to downstream tasks. However, the impact on model robustness is not well understood. In this work, we characterize the robustness-accuracy trade-off in fine-tuning. We evaluate the robustness and accuracy of fine-tuned models over 6 benchmark datasets and 7 different fine-tuning strategies. We observe a consistent trade-off between adversarial robustness and accuracy. Peripheral updates such as BitFit are more effective for simple tasks -- over 75% above the average measured by the area under the Pareto frontiers on CIFAR-10 and CIFAR-100. In contrast, fine-tuning information-heavy layers, such as attention layers via Compacter, achieves a better Pareto frontier on more complex tasks -- 57.5% and 34.6% above the average on Caltech-256 and CUB-200, respectively. Lastly, we observe that the robustness of fine-tuning against out-of-distribution data closely tracks accuracy. These insights emphasize the need for robustness-aware fine-tuning to ensure reliable real-world deployments.
@inproceedings{li_robustness_2025, author = {Kunyang Li and Jean-Charles {Noirot Ferrand} and Ryan Sheatsley and Blaine Hoak and Yohan Beugin and Eric Pauley and Patrick McDaniel}, booktitle = {IEEE/CVF International Conference on Computer Vision (ICCV)}, doi = {10.48550/arXiv.2503.14836}, month = {October}, title = {On the {Robustness} {Tradeoff} in {Fine}-{Tuning}}, year = {2025} }
2024
-
Extracting the Harmfulness Classifier of Aligned LLMs
M.S. Thesis, 2024Abstract: Large language models (LLMs) exhibit high performance on a wide array of tasks. Before deployment, these models are aligned to enforce certain guidelines, such as harmlessness. Previous work has shown that alignment fails in adversarial settings through jailbreak attacks. Such attacks, by modifying the input, can induce harmful behaviors in aligned LLMs. However, since they are based on heuristics, they fail in giving a systematic understanding of why and where alignment fails in adversarial settings. In this paper, we hypothesize that alignment embeds a harmfulness classifier in the model, responsible for deciding between refusal or compliance. Investigating the harmfulness and robustness of the alignment of a model then reduces to evaluating its corresponding classifier, which motivates this work: Can we extract it?. Our approach first builds estimations of the classifier from varying parts of the model and evaluates how well they approximate the classifier in both benign and adversarial settings. We study 4 models across 2 datasets and find through the benign settings that the classifier spans at least a third of each model. In addition, the evaluation in adversarial settings shows that it ends before the first half of most models, exhibiting a transferability greater than 80%. Our results show that the classifier can be extracted, which is beneficial from an attack and defense perspective due to the improvements in both efficiency (smaller model to consider) and efficacy (higher attack success rate).
@mastersthesis{noirot_ferrand_extracting_2024, author = {Jean-Charles {Noirot Ferrand}}, booktitle = {M.S. Thesis}, month = {December}, school = {University of Wisconsin-Madison}, title = {{Extracting} {the} {Harmfulness} {Classifier} {of} {Aligned} {LLMs}}, url = {https://jcnf.me/static/publications/ms_thesis_2024.pdf}, year = {2024} }
2023
-
The Efficacy of Transformer-Based Adversarial Attacks in Security Domains
, , ,
IEEE Military Communications Conference (MILCOM), 2023Abstract: Today, the security of many domains rely on the use of Machine Learning to detect threats, identify vulnerabilities, and safeguard systems from attacks. Recently, transformer architectures have improved the state-of-the-art performance on a wide range of tasks such as malware detection and network intrusion detection. But, before abandoning current approaches to transformers, it is crucial to understand their properties and implications on cybersecurity applications. In this paper, we evaluate the robustness of transformers to adversarial samples for system defenders (i.e., resiliency to adversarial perturbations generated on different types of architectures) and their adversarial strength for system attackers (i.e., transferability of adversarial samples generated by transformers to other target models). To that effect, we first fine-tune a set of pre-trained transformer, Convolutional Neural Network (CNN), and hybrid (an ensemble of transformer and CNN) models to solve different downstream image-based tasks. Then, we use an attack algorithm to craft 19,367 adversarial examples on each model for each task. The transferability of these adversarial examples is measured by evaluating each set on other models to determine which models offer more adversarial strength, and consequently, more robustness against these attacks. We find that the adversarial examples crafted on transformers offer the highest transferability rate (i.e., 25.7% higher than the average) onto other models. Similarly, adversarial examples crafted on other models have the lowest rate of transferability (i.e., 56.7% lower than the average) onto transformers. Our work emphasizes the importance of studying transformer architectures for attacking and defending models in security domains, and suggests using them as the primary architecture in transfer attack settings.
@inproceedings{li_efficacy_2023, address = {Boston, USA}, author = {Kunyang Li and Kyle Domico and Jean-Charles {Noirot Ferrand} and Patrick McDaniel}, booktitle = {IEEE Military Communications Conference (MILCOM)}, doi = {10.1109/MILCOM58377.2023.10356372}, keywords = {workshop}, number = {}, title = {The Efficacy of {Transformer-Based} Adversarial Attacks in Security Domains}, volume = {}, year = {2023} }