Skip to main content

Measuring Probe Performance in Cybersecurity

research
interpretability
probes
ai-safety
Is code vulnerability represented in model activations? What do probes trained on vulnerable code detect?
Author

Efe Akça

Published

June 12, 2026

Slides (old): This work was presented in the ZAIS Meetup 2026 as Probing for Vulnerabilities and as a ZAIS Weekly Session as Monitoring & Probing: Discussion. Both slides are out-of-date due to the update below.

Update (Nov 2026): The main section results have been reproduced in November 2026 for the newly-added PrimeVul section using a shared setup for consistent setup. Probe performance is now slightly elevated token-level compared to before, it’s still behind baselines at the example level.

TL;DR

Note: This project was performed as part of Bluedot Impact’s Technical AI Safety Project course.

Dataset

I started with SVEN (He and Vechev 2023), a dataset of vulnerable/patched function pairs by He and Vechev, with each pair’s vulnerability type (CWE) marked. I turned it into a token-level dataset by marking the tokens removed in the safe code as positive in the vulnerable function. Mechanically: SVEN records which characters the fix changed, and I intersect those character ranges with live-code regions and map them onto each model’s tokens through the tokenizer’s offset mapping for scoring. The probes were trained on token-level diffs for better generalization.

He, Jingxuan, and Martin Vechev. 2023. “Large Language Models for Code: Security Hardening and Adversarial Testing.” Proceedings of the 2023 ACM SIGSAC Conference on Computer and Communications Security (CCS). https://arxiv.org/abs/2302.05319.

SVEN has 710 pairs (1,420 functions) over 9 CWEs (vulnerability types); by language it is composed of 756 Python, 564 C, and 100 C++ functions. The two biggest CWEs are SQL injection (CWE-089, 203 pairs, all Python) and command injection (CWE-078, 105); the memory-safety side is out-of-bounds read (CWE-125, 102), NULL dereference (CWE-476, 65), use-after-free (CWE-416, 55), and out-of-bounds write (CWE-787, 44). Note that CWE/language are correlated, this became important when evaluating with language baselines.

For more in-depth understanding, I split the data by CWE (vulnerability type) and by language, and isolated the subtractive pairs (pairs where the fix changes existing code instead of only adding a check); 477 of the 710 pairs are subtractive, and the other 233, a third of the dataset, only add a guard. Note that in additive pairs, there is no removed/changed portion, which does not have any positive labels for token-level probes.

To cross-check my results, I used the similarly structured PrimeVul dataset (Ding et al. 2025), which provides C/C++-only vulnerable/patched pairs I could process the exact same way. PrimeVul is bigger and broader: 3,626 pairs across 108 CWEs, with a similar subtractive fraction (49%), giving it about four times SVEN’s volume of pairs with markable vulnerable tokens.

Ding, Yangruibo et al. 2025. “Vulnerability Detection with Code Language Models: How Far Are We?” Proceedings of the 47th International Conference on Software Engineering (ICSE). https://arxiv.org/abs/2403.18624.

Probes and the metric

I trained linear token probes on the residual stream across layers. Setup was supervised training with custom loss taken from the hallucination probes paper (Obeso et al. 2025). The loss starts off as BCE on tokens (detect vulnerable tokens individually), then anneals to a BCE on span-max over positive tokens (detect at least one vulnerable token in a given example). I swept layers per model and picked the best layer on validation AUC. Splits are performed at the pair level to avoid data leakage.

Obeso, Oscar, Andy Arditi, Javier Ferrando, Joshua Freeman, Cameron Holmes, and Neel Nanda. 2025. “Real-Time Detection of Hallucinated Entities in Long-Form Generation.” arXiv Preprint arXiv:2509.03531. https://arxiv.org/abs/2509.03531.

I used a linear layer probe (a single weight vector per model) trained on cached activations with AdamW at learning rate 1e-3 for 30 epochs, with the loss blend moving linearly from the per-token term to the span-max term over training. The layer sweep covers every layer of every model (26 to 64 layers depending on size), and the validation-based selection is close to the best-on-test layer for every model (appendix).

For evaluation, I used ROC-AUC over individual tokens, computed only on live code (comments and trivial tokens are filtered out and excluded from scoring) as the evaluation metric. 0.5 represents chance; 1.0 would rank every vulnerable token above every clean one. Note that ROC-AUC is also the probability that a randomly chosen vulnerable token is ranked above a randomly chosen clean one. Concretely, the live-code filter uses tree-sitter to drop comments, import/include lines, function signatures, decorators, and preprocessor directives, about 25-30% of all tokens depending on tokenizer. All test numbers below come from a held-out pool of 290 functions (145 vulnerable, 145 patched): roughly 74,000 live-code tokens (Qwen tokenizer), 1,428 of them positive.

Initial results

Token-level probes get 0.77 to 0.81 token AUC across model sizes and families.

Figure 1: AUC per model: linear probe (bars), MLP head (diamonds).

There are large per-CWE/language performance differences. For example, averaged over the seven models, SQL injection scores 0.95 while use-after-free is at 0.55 (nearly random). When split by language, Python is at 0.85 AUC while C is at 0.59 AUC.

Grouping the CWEs into two families makes the split clearer: every probe detects injection-class bugs well (0.82 to 0.88) but performs at or slightly above random on memory-safety bugs.

Figure 2: Per model, the probe’s token-level AUC on injection-class CWEs (SQL, command, path, XSS; green) versus memory-safety CWEs (out-of-bounds read, use-after-free, NULL dereference; orange; CWE-787 and CWE-190 are excluded, with fewer than 10 positive test examples each). The blue tick denotes the score from the previous figure. Each family bar is the CWE-pooled AUC scores. Note that negatives include other CWEs.

An imbalance of performance between these two are expected due to SVEN’s imbalance of vulnerability types. The ratio of injection to memory tokens are about 3:1. However, this does not explain the near-chance performance in memory type vulnerabilities. I believe the rest of the loss of performance is explained by the difficulty of detecting memory vulnerabilities, as well as the confounder that memory vulnerabilities may be an artifact of the codebase architecture, and not a single function.

Above comparisons were done with token-level predictions (does this token cause vulnerabilities?). However, trying to generalize this to example-level predictions (is this function vulnerable?) fails with naive approaches. Using a simple final-token probe or probe maximum logits over an example is quite weak (Qwen-7B max-pooling: 0.65) compared to token-level AUC’s 0.77 to 0.81 figures.

Figure 3: Per model, example-level AUCs for max-pooled predictions over a function’s live-code tokens, final-token probes. Green dashed line shows the token-level AUC figure (inherently different task, shown to contrast the difference.)

Later in Figure 5 and Figure 6, we see that verbalization performs below a max-over-tokens classifier and that verbalization can be combined with probing for even better detection.

Overall, this is a slightly positive result with injection/Python vulnerabilities being a potential candidate to use probes for (in token-level contexts). To better understand probe performance in this project, I decided to dive deeper into the nuance behind this result. Next, I attempt to break this down: first the probe side (can different probe architectures do better?), then the evaluation side (what are the biases in the dataset? do training individually on vulnerability types perform better? what are we detecting? do the findings replicate cross-dataset?).

Could a stronger probe perform better?

First, I tried to answer the question: Can we get better detection by using bigger probes or bigger models?

Variations on model did not meaningfully change the AUC results:

  • Instruction tuning: a probe on the base 12B model matches the instruct version (0.80 vs 0.79).
  • Scale: 1B to 32B spans 0.77 to 0.81 with no trend (Figure 1).

Then, I experimented with probe architecture starting with MLPs. The MLP probe is a two-layer head (hidden width 128 or 256, 0.3 to 1.3M parameters depending on model). For the MLP probes, I searched a window of nine layers around the initially selected layer instead of every layer, and got 0.75 to 0.81 AUC on every model. MLPs don’t have a consistent lead over linear probes across all variants. MLP probes share similar performance across the dataset, excelling at catching injection family bugs, but failing at catching memory vulnerabilities.

Figure 4: Linear probe vs MLP probe

Next, I decided to investigate per-example performance more closely. First, I tried verbalization. With this method, the model is asked if the code is vulnerable, and P(yes) is read in a single forward pass. Verbalization performed below the max-over-tokens classifier but above the final-token classifier for increasing model sizes, and the example-level performance was still disappointing.

Figure 5: Verbalization compared with naive example-level application of the token probes. Results are near-random for all final-token probes and all verbalization classifiers except for Qwen 32B. Max-pool significantly outperforms final-token and verbalization (in paired tests).

Next, I tried a combination approach: probing on the final assistant turn token with specific prompts. I compared two simple prompts: 1. asking the model neutrally (What do you think about this code?) 2. asking the model about vulnerabilities (Is this code vulnerable?), then probing on the assistant turn’s starting token. These approaches outperformed verbalization by a surprising amount. Qwen 32B’s performance exceeded even that of the baseline (char-n-gram). That pooled score is a language mixture, though. Re-scored within each language, Qwen 32B’s 0.82 is 0.92 on Python functions and 0.63 on C/C++ ones, and every model splits the same way (0.74 to 0.92 within Python, 0.52 to 0.63 within C/C++). The strong example-level numbers here are carried by the Python side.

Figure 6: Per model, example-level AUC comparing probes on neutral and vulnerability-related prompts against asking the model. Upper dashed line (0.78) shows what a lexical-only baseline achieves.

AUC is not the best metric for understanding the performance of production-candidate probes. Other metrics like TPR@FPR=0.01 are often used in cases where false positives are undesired and are costly. A good probe should have high TPR at low FPR so that the people monitoring are not overwhelmed by false positives while missing critical positives.

Looking at the promising candidates from AUC, the TPR@FPR=0.01 is disappointing across CWEs, with particularly bad performance at memory and C/C++ splits. The only production-usable probe seems to be for SQL injection with Qwen 32B. On Qwen 32B, the probe outperforms lexical baselines in “SQLi”, “all” and “Python” slices. For command injection, the lexical scanner is ahead for both models. The rest of the per-CWE TPRs are too low to draw conclusions from. Across the SVEN evals, Qwen probes are slightly ahead of Gemma, which could be explained by Qwen’s focus on coding during training.

Figure 7: Example-level TPR at a 1% FPR threshold, by language (left) and CWE (right). Filled accent color denotes the best probe, gray bar denotes the fixed lexical baseline, dashed fill denotes the verbalized baseline.

Investigating the probe’s classifications

I decided to fix a threshold to see what the probe finds and does not find systematically across models. Threshold was fixed to be the F1-maximizing one on training tokens. For an example-level evaluation, I gave the probes an unfair advantage: I maxed the score over oracle vulnerability-relevant tokens, and used that max score to classify. For token spans, I considered false-positive spans where the probe fired but the range was not in the diff range.

Looking at the 97 held-out vulnerable functions (subtractive), the seven models’ probes have a similar performance distribution as can be seen by the bimodal distribution over functions in Figure 8. 56/97 functions are caught by all probes and 27/97 is caught by none of the probes. The “7/7 caught” functions are injection-only (43/56 SQL injection) and the 0/7 functions are mostly memory-safety. The probe performance is clearly biased towards injection, with very little memory vulnerabilities being detected. Looking at the false positives, 1,729 of the FP spans land on patched code with 78% in injection pairs.

Figure 8: Probe classification performance across the 7 model-probes (from initial results), colored by vulnerability type.

Looking at some example pairs in Figure 9, it’s clear that the probe has learned some patterns of injection vulnerabilities. It triggers for string format syntax as well as concatenation operators.

Figure 9: Three held-out functions their fix side-by-side. Probe probability is shown by red highlights on tokens (Qwen2.5-Coder 32B). Changed/removed portions of the function are underlined. Probe captures the patterns of injection vulnerabilities by firing on string formatting tokens and concatenation operations.

Since these simple patterns of formatting and concatenation could also be captured by text scanners, I selected to use lexical baselines that trained on characters: 1. a token-unigram logistic regression (token identity alone), and 2. a character 3-to-5-gram logistic regression reading a ±48-character window around each token. Note that the scanners are deliberately favored, because I wanted a solid lexical baseline: char-n-gram’s predictions use a window including future characters, and all baselines train on fine-grained per-token labels whereas the probe’s loss anneals to the coarser span-max loss (one label per span).

Compared to these baselines, the probe is slightly above the char-n-gram baseline at 0.81 vs 0.76 (Figure 10). The token-unigram performs much worse at 0.68. The paired difference (+0.054 [+0.011, +0.101]) is significantly above zero. The baselines here are very simple and are text-based classifiers. They cannot be compared to AST-based static code analyzers (e.g. CodeQL, Semgrep), which would likely catch most of the SQL-injection bugs and many of the other injection bugs. The probe performance presents an interesting candidate for further research here. Additionally, since the input code is out of distribution for the model, as the code has not been generated by the model, there may not be an understanding of vulnerability to be probed for.

Figure 10: Text-only scanners vs the probe. Whiskers are 95% bootstrapped intervals.

Language imbalance

Another problem with looking at the total token AUC is that the SVEN dataset is language-imbalanced. In SVEN, 378 of the 410 injection-vulnerable functions (92%) are Python and SQL injection is Python-only, while all 300 memory-vulnerable functions are C/C++. The net effect is that positive tokens are about 5× denser in Python.

An important baseline this suggests is to use language as the score: score every token 1 if its file is Python, 0 otherwise. In token-level AUC, this scores 0.705 AUC. In margin-over-chance terms, (0.705 − 0.5) / (0.813 − 0.5) is about 65% of what the probe achieves, with zero vulnerability information.

Figure 11: The language baseline vs the Qwen2.5-Coder 32B probe (0.813).

Thus, I break down the dataset by language to see the language-invariant token AUCs. Probe performance is strongly biased towards Python: 0.82 to 0.89 within Python, 0.55 to 0.63 within C/C++, on every model.

Figure 12: Within-language AUC, all models. Orange marks: the pooled (original) score.

CWEs are not created equally

To investigate probe performance per CWE, I trained per-CWE probes on Qwen2.5-Coder 32B and Gemma-3 1B. In Figure 13, for CWEs with n>10, the probes outperform the other baseline but fail to outperform other simple baselines like the character-n-gram and unigram.

Figure 13: Per-CWE macro token AUC (live code, averaged over pairs), Qwen2.5-Coder 32B. Per-CWE linear probe versus a char-n-gram and a token-unigram text scanner scored on CWE-only subsets. Whiskers are 95% bootstrapped intervals. Diamonds are the language-only baseline. n = test pairs with a defined pair AUC (only subtractive pairs qualify) .

In SQL injection (CWE-089, CWE-078), the probes perform really well, ahead of both baselines. These differences are significant. Both the baseline and probe seem to be quite saturated, likely due to the very predictable textual signs present in SQL injection bugs.

We’d expect to see semantic probing most useful for memory bugs, but even on a per-CWE basis, there is not much evidence of success. CI is ahead of random chance only for CWE-125 (n=8), and CWE-190 (n=3), even then the mean is around 0.62 and 0.65. Overall, this doesn’t point in the direction of semantic understanding of memory bugs, but n is too small to claim otherwise either.

Training per-CWE probes don’t increase performance over the pooled training, ruling out another avenue for improvement. It’s additionally clear that there is large performance gaps across CWEs.

Pairs without positive tokens

A third of SVEN pairs add code for vulnerability fixes (eg. bounds test, null guard), rather than remove or change vulnerability-causing code. In such pairs, the before-version has no vulnerable token to mark and train on. To investigate the effects of this subset on training, I separated the dataset into so-called “subtractive” and “additive” subsets. The subtractive pairs remove/change code to fix their vulnerabilities, which makes their diffs more likely to be vulnerability-related.

Now, I evaluate example-level pair ranking accuracy, i.e. does the max token score rank the vulnerable function above the fixed version. The accuracy is at or below random chance for additive pairs as can be expected (there are more tokens, which gives a higher chance of a larger max logit). Pair ranking accuracy for additives (0.44, mean over seven models) is much worse than for subtractive pairs (0.78). To isolate training effects, I also trained a control on all pairs (the main probes are trained on subtractive pairs only). This control performs the same on token AUC (0.79 for both), and the differences stay below 0.01 in cross-validation (appendix). This evaluation provided evidence that additive pairs have little impact on token AUC results.

Figure 14: Pair-ranking accuracy across training x evaluation subset combinations (left); token AUC by training x evaluation subset combinations (right).

Cross-checking on PrimeVul

So far, all experiments were run on SVEN. To check whether the findings hold on a second dataset, I used PrimeVul, a bigger paired dataset that is C/C++ only, so the language imbalance does not exist there. I used it in two ways: first to test whether probes transfer between the datasets, then to repeat the main SVEN experiments inside PrimeVul.

Looking at a test of transfer across datasets, SVEN-trained probes do not seem to transfer to PrimeVul (Figure 15). On the tested models (Qwen2.5-Coder 7B and Gemma-3 12B), they score 0.52 to 0.54 token AUC with every interval including chance (probe trained and applied at the same layer). Probes trained only on SVEN’s C/C++ examples barely transfer, with 0.52 to 0.56 on PrimeVul compared to 0.57 to 0.58 on SVEN’s own C/C++ test set. I do not report the other direction (PrimeVul-trained probes on SVEN), because 41 of SVEN’s 62 C/C++ test pairs also appear in PrimeVul’s training split.

Figure 15: Token AUC of SVEN-trained probes on PrimeVul. Probes are trained on either all of SVEN or the C/C++ subset. The gray bars show C/C++-trained probes on SVEN’s C/C++ test set for reference. Each probe is trained and applied at the same layer. Whiskers are 95% bootstrapped intervals.

Next, I trained per-CWE probes for 12 CWEs inside PrimeVul on Qwen2.5-Coder 7B and tested every probe on each (Figure 16). The probes do not find a shared memory-safety direction: probes trained on one memory CWE score 0.52 on the other memory CWEs, about the same as on unrelated CWEs (0.54). On the trained CWE, 7/12 probes perform above chance in 95% CI. Some have an AUC above chance but the CI includes 0.5, for example CWE-476 (NULL dereference) is at 0.68 AUC with 95% CI [0.35, 0.78].

Figure 16: Per-CWE probes trained on PrimeVul (Qwen2.5-Coder 7B, layer 15). Left: token AUC of per-CWE probe tested on different CWE, averaged by family. Right: per-CWE probe token AUC on the same CWE. Whiskers are 95% bootstrapped intervals.

I also tested the SVEN-trained per-CWE memory probes on PrimeVul’s pairs (Figure 17, Qwen2.5-Coder 7B), scoring each vulnerable function against the fix (macro token AUC as in Figure 13, excluding 4 pairs also in SVEN’s training data). The probes for use-after-free (CWE-416, 0.62) and out-of-bounds write (CWE-787, 0.63) are above chance, out-of-bounds read (CWE-125, 0.59) is not, and the other two CWEs have fewer than 10 pairs. On CWE-787 (22 pairs), the probe is ahead of the unigram scanner (+0.125 [+0.042, +0.215]), but not significantly ahead of the char-n-gram scanner (+0.102 [−0.001, +0.208]). The SVEN-trained per-CWE probes are also no better than per-CWE probes on PrimeVul. With 10 to 22 pairs per CWE, the evidence is quite weak to claim whether transfer happens or not.

Figure 17: Per-CWE macro token AUC on PrimeVul pairs, Qwen2.5-Coder 7B. Per-CWE probes trained on SVEN (layer 13) versus per-CWE probes (layer 15) and text scanners trained on PrimeVul. Whiskers are 95% bootstrapped intervals. CWEs with fewer than 10 pairs are grayed out.

The stronger experiment is to repeat the main SVEN experiments inside PrimeVul, with training, layer selection and baselines on PrimeVul data. I repeated the token-level MLP comparison and example-level experiments: the probe on the assistant turn, the lexical baseline, verbalization, and TPR at 1% FPR.

On SVEN, the best example-level result was the probe on the assistant-turn token with the vulnerability prompt (Figure 6), and that score was carried by Python examples. For the four models tested, the pooled scores split into ~0.9 within-Python and ~0.6 within-C/C++ performance. Since PrimeVul is all C/C++, I expected the scores to be close to SVEN’s within-C/C++ numbers. This is what happens for all four models (Table 1).

Table 1: Example-level AUC of the strongest example-level variant (probe on the assistant-turn token, vulnerability prompt). The SVEN columns are per model on the pooled test set and language subsets for this probe; the PrimeVul column is trained and evaluated entirely inside PrimeVul with the same methodology.
SVEN pooled SVEN within-Python SVEN within-C/C++ PrimeVul (all C/C++)
Qwen-7B 0.809 0.908 0.595 0.576
Qwen-32B 0.825 0.917 0.626 0.605
gemma-12B 0.762 0.861 0.567 0.576
gemma-27B 0.762 0.862 0.568 0.604

In Table 1, the SVEN C/C++ and PrimeVul results are within ±0.04 of each other which is within SVEN’s confidence intervals of ±0.05. This shows that the results in C/C++ are consistent across datasets.

The baseline results are similar across datasets as well (Figure 18), when looking at the within-C/C++ subset. The char-n-gram scanner matches with SVEN’s 0.565 on PrimeVul at 0.565. The char-n-gram is still the strongest text baseline tested on PrimeVul (Figure 19): token-unigram classifier scores 0.55, and max-pool of token-level classifier scores ~0.49. Verbalization is at ~0.5 on all four models. This also matches SVEN’s C/C++ functions, where verbalization is at chance for every model (0.48 to 0.51). On PrimeVul, the probe is ahead of verbalization in every case (+0.07 to +0.11).

Figure 18: Example-level AUC on SVEN C/C++ test versus PrimeVul, for the probe on the assistant-turn token (vulnerability prompt), verbalization, and the char-n-gram scanner. The SVEN scanner is trained on all of SVEN and scored on C/C++. Whiskers are 95% bootstrapped intervals.

Only the biggest models beat the lexical scanner (Figure 19). The probe is ahead of the char-n-gram scanner by 0.03 to 0.04 AUC for Qwen 32B with both prompts and for gemma 27B with the vulnerability prompt. The rest including gemma 27B with the neutral prompt, are within margin of error from the scanner. Even for the biggest models, the scores (~0.6) are still close to random chance, so this is not a usable classifier.

Figure 19: PrimeVul example-level AUC, probe on the assistant-turn token (vulnerability and neutral prompts) vs verbalization and the char-n-gram scanner. The horizontal lines are the text baselines: the char-n-gram scanner (0.565 here vs 0.776 on SVEN), the token-unigram classifier and the best max-pooled token scanner. Whiskers are 95% bootstrapped intervals.

TPR at 1% FPR is very low for every method (Figure 20). I set one threshold per method for a 1% FPR on held-out training negatives, which ended up between 0.3% and 1.2% FPR on the test set. At this threshold, the probes on the two largest models catch only 2 to 3% of the vulnerable functions, which is not significantly more than the char-n-gram scanner (~2%). Verbalization catches less than 1%. This is in line with SVEN’s C/C++ functions, where the probes catch 1 and 3 of the 62 vulnerable functions. SVEN’s overall TPR (~30%) came from the Python/SQL injection side. None of the methods are usable as a detector on C/C++ code.

Figure 20: PrimeVul TPR (left) and realized FPR (right) for the two largest models (probe under the vulnerability prompt), with one threshold per method set for 1% FPR on training negatives. No method is usable, and the probes are not significantly ahead of the scanner.

Finally, I repeated the MLP comparison inside PrimeVul. The MLP probes score below the linear probes on both models (Figure 21), with the differences within margin of error. Note that the layer search here covers layers 13 to 17 (up to 19 for the gemma-12B linear probe). As in SVEN, a bigger probe does not help.

Figure 21: Token AUC of probe with different architecture (linear, MLP) on PrimeVul. The MLP performs below the linear probe on both models, though within margin of error. Whiskers are 95% bootstrapped intervals.

Overall, PrimeVul supports the SVEN findings on a second dataset. Example-level probe scores match SVEN’s within-C/C++ numbers, the lexical baseline is barely above chance without the language imbalance, verbalization is at chance, TPR at 1% FPR is too low to be usable, and MLP probes do not help. The strong SVEN numbers seem to come from the Python and injection examples, and on C/C++ code neither dataset shows a usable signal. Two results are in favor of the probes: the biggest models beat the strongest text baseline by a small margin, and the SVEN-trained CWE-787 probe is ahead of the unigram scanner at the token level. Neither is anywhere close to a usable detector.

Next steps

The most important design gap in this study is that probes were trained on single functions out of repo context. Some examples are impossible to reason with vulnerabilities as well (memory vulnerabilities depend on function usage). The negative results have more than one reading: the probe catches lexical patterns, model has no vulnerability understanding to read, or the vulnerability understanding happens at different tokens.

Some next experiments I did not pursue that would help to better understand the results:

  • Random-direction probe distribution: the AUC distribution of random unit vectors per layer, with the trained probe reported as a percentile of that null (the sleeper-agents paper checked this; random directions hit 0.95 there). This would demonstrate how unique the direction that the trained probe catches.
  • Pivoting to generation-based probing: probing generation-time states on model-written code with labels from analyzing the code directly. The monitor the North Star calls for should probe in-distribution code.

Conclusion

Activation probes for code vulnerability, as evaluated here, seem to discover textual patterns and are unreliable for detection of vulnerabilities as demonstrated in the SVEN dataset. There is still a lot of pending work to narrow down whether LLMs know when they’re writing vulnerable code. What this project shows is limited in scope: activation probes and zero-shot verbalization are not reliable vulnerability scanners for pieces of code. A text scanner trails closely behind at the token level and matches or beats the probes at the example level in most settings, with much lower cost and more interpretability, and the probe’s TPR@FPR is abysmally low for all but the simplest of vulnerabilities. The takeaway is in careful dataset exploration and experiment control for probes, which is essential for mechinterp research. Baselines like text-only scanners, composition nulls, and experimentation with multiple metrics, controlling for covariates in the dataset as well as cross-checking on a secondary dataset are essential experiments to rule out confounders. The cross-dataset check covered my main findings: SVEN’s within-C/C++ scores matched PrimeVul, and PrimeVul experiments showed that the key findings are supported cross-dataset.

Supplementary

Steering with memory-family token probes barely affects model decisions on vulnerability. When adding ±4σ along the probe direction, next-token probability of “yes” changes by less than 0.01 on three models (Qwen2.5-Coder-32B, gemma-3-27b, Qwen3-32B) and by 0.03 on Qwen3.6-27B. The Qwen3.6-27B shift is larger than for random directions, but still very small. This probe direction does not seem to be causally used by the models.

Figure 22: Steering at ±4σ on Qwen2.5-Coder 32B: verbalized judgment unchanged.

Appendix: validation details

  • Length cap. SVEN’s train and validation splits hold 803 pairs. I dropped the 88 pairs where either function is longer than 6,000 characters, so that no labelled line is cut off by the extractor’s 2,048-token limit (a cut-off positive would silently become a negative). The remaining 715 pairs are the dataset used throughout.
  • Prompts. Both example-level prompts put the function in a fenced code block after the line Here is a code snippet:, end with the question, and use the model’s chat template with the assistant turn opened. The neutral question is What do you think about this code?. The vulnerability question, also used for verbalization, is Does the code above contain a security vulnerability? Respond with ONLY one word — "yes" or "no" — and nothing else.; the main text shortens it to “Is this code vulnerable?”.
  • Masking no-op. Excluding comments and trivial tokens from training changes test AUC by less than 0.004 on all seven models, within margin of error. Separately, AUC over code tokens is slightly lower than AUC over all tokens on six of seven models (by up to 0.02), so restricting the evaluation to code tokens does not inflate the reported numbers.
  • Layer selection. Every layer of every model was swept (78 to 192 probes per model: 26 to 64 layers, three regularization strengths each). Best layer by validation token-level AUC is within 0.030 of the best layer by test AUC for all seven models.
  • Loss weighting sweep. I compared two values of α, the weight on vulnerable tokens in the per-token loss term. On Qwen-32B and gemma-27b, α=10 beats α=1 on example-level AUC (0.66 vs 0.64, 0.64 vs 0.60), and the two are within margin of error on token-level AUC (0.81 vs 0.81, 0.78 vs 0.77), so the token probes here use α=10.
  • Subtractive-restriction stability. The all-pairs vs subtractive-only comparison was re-run as a 5-fold × 3-seed cross-validation (15 retrainings per model): on the full test set the all-pairs probe leads by 0.004 on average, and on the subtractive test pairs it trails by 0.005. Both are small compared to the ~0.03 standard deviation across retrainings.
  • Saved-score rescoring and reproduction checks. All probes were retrained in the November 2026 reproduction with one shared setup. The language and scanner comparisons re-score the saved probe scores without retraining, after first checking that the saved scores reproduce each probe’s reported AUC. I trained new probes for the PrimeVul experiments: per-CWE probes on Qwen-7B, the assistant-turn probe, and MLP vs linear.