Databricks 用领域知识注入提升 LLM 对 npm 二进制恶意包的检测

Introduction

Supply chain attacks on package registries have grown in volume and sophistication [1]. Public research regularly observes malware being deployed to the npm registry by a range of threat actors, from spam campaigns [2] to the steady proliferation of DPRK malware [3, 4, 5], and to the high-impact TeamPCP in 2025 and 2026 [6, 7], with the DPRK and TeamPCP actors having a track record of successfully compromising widely used open source projects for maximized downstream reach [8].

To study this problem directly, we have been collecting npm malware samples since November 2025, and observed that, among the 11,413 package-versions we scanned and corroborated through OSV npm malware advisories [9], most malware on the npm registry hides its payload in the package source or fetches it from an external server, accounting for 96.2% of the corroborated dataset. The remaining 3.8%, 432 package-versions, bundle a binary. Though small in volume, they can carry disproportionate risk during triage. A stripped, opaque binary that carries no source for tools to read can be nested within a facade of clean source and metadata and potentially skew an LLM towards a benign verdict.

We evaluate how a panel of LLMs reason over crafted binary payloads at varying obfuscation and ambiguity, identify the failure modes specific to this vector, and close them with a purpose-built, cost-effective encoding of domain knowledge that improves detection quality for both budget and frontier models alike.

Background

To support the wider research community, we built a scanner that monitors the npm registry in real time and runs a static stage before LLM triage, with YARA [10] on the tarball, Semgrep [11] and the npm registry API on the JavaScript and metadata, and a binary-analysis module over the compiled artifact. It assembles the output into a structured prompt, and the LLM returns a verdict with reasoning, requesting additional files over up to three rounds.

Our analysis found packages with bundled binaries to be both the rarest and the hardest for the pipeline to triage. An adversary distributing malicious JavaScript can assume the target already has the runtime to execute the payload, whereas a binary must be compiled for a range of target machines. Across OSV-confirmed npm malware, binary-bearing packages fail to reach a decisive malicious verdict more than twice as often as source-only ones (14.4% vs 6.2%).

Prior work instruction-tunes open-source models on expert-authored data, baking the knowledge into the weights [12]. That route cannot reach the closed, hosted models, and it requires retraining as the threat landscape shifts. We take the opposite approach and supply the expertise at inference, as pre-computed static evidence placed in the prompt before the model reasons.

Experimental Design

Objective

We evaluate whether LLMs can detect malicious logic concealed within precompiled binaries in otherwise clean packages. A secondary question is whether the detection failures are systematic rather than random. Third, whether encoding domain knowledge of the binary problem can improve detection accuracy.

Adversarial Payloads

We built five payload types (Table 1) of native Node.js addons (.node files) covering common supply chain attack patterns. Imports range from defensible (fopen, connect, gethostname) to rarely legitimate (execve with dup2), and each package wraps the binary in a professional package.json, README, JavaScript bindings, and binding.gyp.

The sysmon stealer spans an obfuscation ladder from a readable baseline to a hardened build with symbols stripped and strings XOR-encoded, while the other four types appear at a single obfuscation level. We chose single-byte XOR deliberately, since unlike a stronger cipher such as RC4 it leaves no structural markers in disassembly for a model to catch.

The set mirrors collected malware. Across the OSV-confirmed npm packages shipping a binary, formats are PE, ELF (all stripped), and Mach-O, with behaviors spanning credential and environment theft, persistence, exfiltration and C2, host reconnaissance, and droppers.

Table 1. Adversarial payload types and their import profiles.

Models Tested

We evaluated nine models across seven providers (Table 2): seven budget models a team would realistically run across its codebase, four of them open-weight, and two higher-capability baselines as behavioral baseline references at higher price points. We also ran Claude Opus 4.6, the most expensive model, only as a cost reference.

Table 2. Models evaluated and their roles.

To isolate the profiles, each payload was scanned twice per model, once without profiles and once with them. Prompt, context, and binary analysis were identical across both runs. Inference settings were held constant across every model and both conditions: temperature 0, the same system and analysis prompts throughout, and an 8192-token output cap.

Capability Profile Architecture

A capability profile is a declarative playbook that encodes one malicious technique, such as credential harvesting or persistence, as a set of weighted conditions over the scanner's static evidence. The profiles cover source and binary techniques across npm and PyPI.

Figure 1 shows how a profile enters a scan:

1. The static stage extracts evidence from the package.
2. Each profile scores that evidence against its suspicious, malicious, and critical thresholds, a scale separate from the model's own verdict.
3. When a threshold is reached, the engine injects a structured summary (~1,800 tokens) into the prompt before the model reasons, along with any findings from the static and metadata analyzer stage.
4. The model forms its own verdict over the combined evidence and, in this experiment, may override a firing profile entirely.

Figure 1. The scan pipeline with capability profiles.

Results

Each model was scanned against all ten payloads. We count HIGHLY_SUSPICIOUS or MALICIOUS as a detection success; SUSPICIOUS or BENIGN on a known-malicious payload is a failure. The failure analysis centers on the seven-model budget tier (70 model-payload tests); the two baselines (20 tests) serve only as a reference ceiling and are reported separately.

Without Capability Profiles

Without profiles, the budget tier failed on 35 of 70 tests (50%), 20 of them outright benign, and fell short of the malicious bar on 51 (73%). The two behavior baselines failed none at the escalation threshold and fell short of malicious only once (Sonnet, sysmon v3). Within the budget tier the failures followed distinct, repeatable model-specific patterns (Table 3) rather than random error; Gemini alone matched the baselines on detection, though through confabulated reasoning rather than genuine analysis. The full verdict and confidence matrix is in the appendix (Table A1).

With Capability Profiles

With profiles enabled, 32 of the 35 no-profile detection failures (91%) recover to a detection, and no model returns a benign verdict on any payload. Every model that failed without profiles reached full detection with them, except GPT-5.4-nano, whose three residual failures stay at suspicious on three payloads. Across the affordable tier, failures at the escalation threshold fell from 50% to 4% (3 of 70), and at the malicious bar from 73% to 34%.

Table 3. Per-model detection rate (verdict at highly suspicious or above on a known-malicious payload) without and with capability profiles.

The clearest single case is Qwen3-Next on the obfuscated sysmon-native stealer: benign at 0.98 recommending allowlist without profiles, malicious at 0.98 recommending block with them, on byte-identical input, after four profiles fired. The baselines, Sonnet and Kimi, were already correct without profiles, and the evidence strengthens their verdicts, so the profiles reinforce rather than distort a model that already reasons correctly.

Profile Triggers

Technique-specific profiles fired according to each payload’s attack pattern.

Table 4. Capability profiles triggered per payload at critical or malicious threshold.

Discussion

Failure Mode Analysis

The experiment surfaced four failure modes, each spanning a large fraction of the panel rather than any single model (Table 5). They are not mutually exclusive, since one benign verdict can be both a dismissal and a rationalization, so the table reports per-mode prevalence and a model's row can sum past ten. We read dismissal and hedging from the verdict tier, rationalization and confabulation from the written reasoning.

Two factors explain them. The first is domain knowledge, which budget LLMs lack for interpreting binary structural evidence in a package's context [13]. The second is raw capability. Reading a stripped, string-obfuscated binary requires disassembly and decoding that the budget tier did not attempt.

Table 5. Per-model failure-mode counts across the ten payloads without profiles, spanning all nine models (90 scans).

An ensemble that takes the majority verdict would not resolve this. On the obfuscated sysmon-native stealer (version 2), the nine verdicts split three benign (Gemma, Qwen, Llama), two suspicious (Haiku, GPT-5.4-nano), one highly suspicious (GLM), three malicious (Gemini, Kimi, Sonnet): the majority sits below the detection threshold, so the vote returns the wrong answer. The modes do not cancel out; they compound, so no voting scheme gives reliable coverage without first closing the domain-knowledge gap. This matches known limits of majority voting in LLM ensembles, where correlated errors converge on the wrong answer rather than canceling [14, 15].

Rationalization (~37% of scans, six models) invokes contextual cues, the install script, native-module packaging, or package description, to build a benign interpretation that overrides the binary signals. Haiku called the unobfuscated sysmon v1 MALICIOUS 0.98, but once the strings were XOR-encoded its verdict fell to SUSPICIOUS, treating node-gyp-build as standard and the .node extension as a legitimate addon, matching documented post-hoc rationalization [16]. Obfuscation thus inverts the expected relationship. The more thoroughly the payload is concealed, the lower the assigned severity, and because the rationale stays internally coherent, evaluation on unobfuscated samples would never elicit it.

Dismissal (22%, four models) returns a benign verdict without engaging the binary at all. Qwen3-Next returned BENIGN 0.98 on nine of the ten payloads and recommended allowlisting each, including the netpulse reverse shell and the taskd persistence tool, with a stated intent that ignored the binary's imports entirely. The verdict is insensitive to import ambiguity, and the uniform high confidence conceals the failure from any downstream threshold.

Hedging (16%, four models) registers suspicion but never escalates to a decisive verdict. GPT-5.4-nano returned SUSPICIOUS at 0.55-0.62 on eight of the ten payloads, never reaching highly suspicious or malicious, with buildchain-native drawing only SUSPICIOUS 0.62 and a recommended action of "Monitor." A suspicious verdict looks like functioning analysis while adding nothing beyond the YARA and Semgrep stages, and under a policy that blocks only at the malicious threshold these payloads pass.

Confabulation (~23%, six models) is the odd case, where the verdict is correct but its justification is fabricated, so a pipeline that logs it for audit accumulates fabricated indicators that mislead a later analyst or any model fine-tuned on them. It appears even in both baselines, so it recurs regardless of capability tier. Gemini is the clearest case. It returned MALICIOUS on all ten payloads but invented the detail, reporting "a decryption routine at address 0x401490" on taskd and "a custom XOR decryption loop at 0x4013b0" on netpulse, though neither address exists in the scan input and neither payload contains a cipher. The fabrication concentrates where the evidence is thinnest, in the single-function x86 payloads whose scan exposes one real address.

The Structural Gap

The under-detections are structural, not a defect of individual models. The budget-tier failures below the escalation threshold are dominated by rationalization and dismissal, where the binary sits alongside clean source, a node-gyp-build install script, professional metadata, and a plausible README, each unremarkable for a native module. No single artifact misleads the model. Every one of the 35 failures leans on at least two of these benign signals, 24 on four at once, and 14 recast the binary's own imports as standard system calls (Table 6). LLMs can analyze binary evidence in isolation [17, 18], but here the failure is contextual.

Table 6. Benign contextual signals cited across the 35 detection failures.

Predictability of Failure Modes

The modes are predictable in that each model exhibits one, though none is deterministic across a model's whole output, so defenders cannot simply pick a better model. The two categories even fail differently, with dismissal almost entirely an open-weight behavior (18 of the 20 no-profile benign verdicts from Gemma, Qwen, and Llama) while the proprietary models rarely dismiss and instead rationalize, hedge, or confabulate. This bears on model selection. A test set of a few unobfuscated or low-ambiguity packages makes a budget model look sufficient while hiding its failure on the obfuscated credential stealer, and obfuscation recurs in real malware, with the Binary String Obfuscation profile firing on 74 of the 425 OSV-confirmed binary packages (17%). Testing only high-ambiguity payloads has the opposite effect, condemning a serviceable model and pushing a team to overspend on a frontier one. A correct verdict also does not imply a correct label, as budget models often flagged maliciousness while assigning the wrong category, so the category and stated evidence are triage hints, not ground truth for attribution.

Cost Implications

The baselines Sonnet 4.6 and Kimi K3 detected every payload without profiles, at roughly 3x the per-scan cost of the budget tier, which with profiles matches that accuracy at the lower price. Profiles also change the token work each model does (Table 7), and the direction depends on how hard the model works unaided. The budget tier takes a modest input premium from the ~1,800-token evidence block, with input rising 9 to 34% and its analysis otherwise unchanged, whereas the thorough models move the other way. Opus, which without profiles gathers about 10.5K input tokens per scan requesting extra files, drops to 7.2K with them (−32%), and Kimi emits 13% fewer output tokens for the same reason. Profiles add a small, bounded input cost to the cheap models and claw back far more from the expensive ones.

Table 7. Per-model token usage per scan, averaged over the ten payloads, without and with capability profiles.

Production Validation

Malicious Set

Public OSV advisories [9], excluding withdrawn entries, confirm 470 malicious npm package-versions that ship a native binary, 604 binaries in total. By format they are PE (273), ELF (215, every one stripped), and Mach-O (113), with 3 unidentified; by compiler, 363 are native C or C++, 183 Go, and 58 Rust. The 241 Go or Rust binaries, 40% of the total, are the case the engine's runtime-aware gating exists for, and 50, all ELF, are UPX-packed. Of these 470, 425 fall in our scanned, non-withdrawn corpus form the evaluation set in Detection.

Benign Set

The benign set is defined from external signals only, with no reliance on the engine's own verdict. A package qualifies when it ships a native binary, exceeds one million monthly downloads, and has never appeared in OSV, giving 2,449 package-versions across 287 distinct packages. The scanner escalates 104 of them to highly suspicious or above, a false-positive rate of 4.25% of package-versions, concentrated in 6 distinct packages, 2% of the 287. On all 104 the static playbooks also fired, and on 33 a binary capability profile reached the malicious or critical threshold, so the escalation reflected genuine signals in the binary, packing, high-entropy sections, and suspicious imports. The residual false-positive rate is a property of static binary ambiguity.

Detection

The evaluation set is the 425 binary-bearing package-versions from the Malicious Set that carry a production verdict (the non-withdrawn subset in our verdicted corpus). Against these and the benign set, the scanner reaches precision 0.77, recall 0.81, and F1 0.79 at the escalation threshold (Table 8). Recall is scored against OSV's package-level label, which does not isolate the malicious artifact, and a binary capability profile drove 74% of the detections while the rest were escalated on JavaScript, metadata, or source signals.

Table 8. Detection performance across OSV-confirmed binary malware and the benign validation corpus at the escalation threshold.

The scanner recovers 345 of the 425 positives; 60 fall to suspicious and 20 are cleared as benign, a 4.7% hard-miss rate. Inspecting the metadata of the 20 cleared packages, without opening the binaries, about half bundle a recognizable third-party tool unchanged (ripgrep, node-pty, NSIS installer plugins) where OSV attributes the malice to the install or JavaScript layer; the rest ship the package's own compiled CLI, flagged for an account-compromise campaign or a low-confidence advisory. The binary-level miss rate is thus lower than the headline, though we do not claim every cleared binary is benign.

Limitations

The production figures are measured against OSV's package-level labels, which mark a whole package malicious without isolating the responsible artifact; some cleared packages are mislabeled, their payload in JavaScript or metadata rather than the binary. So the reported recall and false-positive rate describe performance against those labels, not binary detection specifically, which could be somewhat higher or lower. And the hosted models sit behind continuously updated endpoints, so absolute verdicts reflect one serving epoch and may drift, which is why our conclusions rest on the within-epoch delta between no-profile and with-profile conditions on byte-identical inputs, not on any single verdict.

Conclusion

The four failure modes, rationalization, dismissal, hedging, and confabulation, recur across the panel, producing detection failures on 50% of budget-tier tests at the escalation threshold and 73% at the stricter malicious bar. Capability profiles, which inject pre-computed binary domain knowledge before the model reasons, cut escalation-threshold failures to 4% and eliminate every benign verdict, while holding false positives on the benign set to 4.25% of package-versions (104 of 2,449; 2% of distinct packages), low enough for an analyst to clear by hand. Together these let a far cheaper model match the detection accuracy of a substantially more expensive one.

Appendix

Table A1. Verdicts without capability profiles (verdict and confidence), per model and payload. This is the full matrix summarized in Table 3 and the Results Without Capability Profiles section.

References

[1] Sonatype. “2026 State of the Software Supply Chain”. 2026. https://www.sonatype.com/state-of-the-software-supply-chain/introduction

[2] SourceCodeRed. “IndonesianFoods spam campaign publishes more than 86,000 malicious NPM packages”. 2025. https://sourcecodered.com/indonesianfoods-npm-worm/

[3] Kieran Miyamoto. “DPRK npm packages”. https://dprk-research.kmsec.uk/

[4] Amazon. “Amazon identifies North Korean hacker group behind open-source supply chain attacks”. 2026. https://aws.amazon.com/blogs/security/amazon-identifies-north-korean-hacker-group-behind-open-source-supply-chain-attacks/

[5] OpenSourceMalware. “PolinRider: DPRK Threat Actor Implants Malware in Hundreds of GitHub Repos”. 2026. https://github.com/OpenSourceMalware/PolinRider

[6] Zscaler. “Tracking Shai-Hulud: Inside the ChainDrop NPM Worm”. 2026. https://www.zscaler.com/blogs/security-research/tracking-shai-hulud-inside-chaindrop-npm-worm

[7] CISA. “Widespread Supply Chain Compromise Impacting npm Ecosystem”. 2025. https://www.cisa.gov/news-events/alerts/2025/09/23/widespread-supply-chain-compromise-impacting-npm-ecosystem

[8] Unit42. “The npm Threat Landscape: Attack Surface and Mitigations”. 2026. https://unit42.paloaltonetworks.com/monitoring-npm-supply-chain-attacks/

[9] OSSF. “OpenSSF Malicious Packages”. https://github.com/ossf/malicious-packages

[10] VirusTotal. “What is YARA?”. 2026. https://docs.virustotal.com/docs/what-is-yara

[11] Semgrep. “Semgrep: lightweight static analysis for many languages”. https://semgrep.dev

[12] Levi, M. et al. “CyberPal.AI: Empowering LLMs with Expert-Driven Cybersecurity Instructions”. 2024. https://arxiv.org/pdf/2408.09304v1

[13] Ullah, S. et al. “LLMs Cannot Reliably Identify and Reason About Security Vulnerabilities (Yet?): A Comprehensive Evaluation, Framework, and Benchmarks”. 2024. https://arxiv.org/abs/2312.12575

[14] Traad, F.; Chehab, A. “To Ensemble or Not: Assessing Majority Voting Strategies for Phishing Detection with Large Language Models”. 2024. https://arxiv.org/abs/2412.00166

[15] Li, Q. et al. “Beyond Majority Voting: LLM Aggregation by Leveraging Higher-Order Information”. 2025. https://arxiv.org/abs/2510.01499

[16] Sharma, M. et al. “Towards Understanding Sycophancy in Language Models”. 2023. https://arxiv.org/abs/2310.13548

[17] Patsakis, C. et al. “Assessing LLMs in Malicious Code Deobfuscation of Real-world Malware Campaigns”. 2024. https://doi.org/10.1016/j.eswa.2024.124912

[18] Jelodar, H. et al. “Large Language Model (LLM) for Software Security: Code Analysis, Malware Analysis, Reverse Engineering”. 2025. https://arxiv.org/abs/2504.07137