英国 AI Security Institute：从漏洞到绕过，为 AI 防护机制调整漏洞披露流程

This website uses cookies

We use cookies to personalise content and ads, to provide social media features and to analyse our traffic. We also share information about your use of our site with our social media, advertising and analytics partners who may combine it with other information that you’ve provided to them or that they’ve collected from your use of their services.

[#DSR_FORM_URL_TEXT#]

[#GPC_BANNER_ICON#]

[#GPC_TOAST_TEXT#]

Consent Selection

Necessary

Statistics

Marketing

Show details

Details

Necessary 3- [x] Necessary cookies help make a website usable by enabling basic functions like page navigation and access to secure areas of the website. The website cannot function properly without these cookies.

*   [Cloudflare 1](https://www.aisi.gov.uk/blog/from-bugs-to-bypasses-adapting-vulnerability-disclosure-for-ai-safeguards#)[Learn more about this provider![Image 2: opens in a new window](blob:http://localhost/7c37f61654102a88a9be209f9569ef78)](https://www.cloudflare.com/privacypolicy/ "Learn more about this provider Cloudflare's privacy policy - opens in a new window")**cf.turnstile.u**This cookie is used to distinguish between humans and bots. **Maximum Storage Duration**: Persistent**Type**: HTML Local Storage   
*   [Cookiebot 1](https://www.aisi.gov.uk/blog/from-bugs-to-bypasses-adapting-vulnerability-disclosure-for-ai-safeguards#)[Learn more about this provider![Image 3: opens in a new window](blob:http://localhost/7c37f61654102a88a9be209f9569ef78)](https://www.cookiebot.com/goto/privacy-policy/ "Learn more about this provider Cookiebot's privacy policy - opens in a new window")**CookieConsent**Stores the user's cookie consent state for the current domain**Maximum Storage Duration**: 1 year**Type**: HTTP Cookie   
*   [Gov.uk 1](https://www.aisi.gov.uk/blog/from-bugs-to-bypasses-adapting-vulnerability-disclosure-for-ai-safeguards#)[Learn more about this provider![Image 4: opens in a new window](blob:http://localhost/7c37f61654102a88a9be209f9569ef78)](https://www.gov.uk/help/cookies "Learn more about this provider Gov.uk's privacy policy - opens in a new window")**SSESS#**Pending**Maximum Storage Duration**: Session**Type**: HTTP Cookie

Statistics 12- [x] Statistic cookies help website owners to understand how visitors interact with websites by collecting and reporting information anonymously.

*   [Google 4](https://www.aisi.gov.uk/blog/from-bugs-to-bypasses-adapting-vulnerability-disclosure-for-ai-safeguards#)[Learn more about this provider![Image 5: opens in a new window](blob:http://localhost/7c37f61654102a88a9be209f9569ef78)](https://business.safety.google/privacy/ "Learn more about this provider Google's privacy policy - opens in a new window")Some of the data collected by this provider is for the purposes of personalization and measuring advertising effectiveness. The provider may use the IP Addresses for ads measurement and ads personalization.

_ga[x2]Used to send data to Google Analytics about the visitor's device and behavior. Tracks the visitor across devices and marketing channels.Maximum Storage Duration: 2 yearsType: HTTP Cookie ga#[x2]Used to send data to Google Analytics about the visitor's device and behavior. Tracks the visitor across devices and marketing channels.Maximum Storage Duration: 2 yearsType: HTTP Cookie
* Microsoft 7 Learn more about this providerc.gifCollects data on the user’s navigation and behavior on the website. This is used to compile statistical reports and heatmaps for the website owner.Maximum Storage Duration: SessionType: Pixel Tracker _clck[x2]Collects data on the user’s navigation and behavior on the website. This is used to compile statistical reports and heatmaps for the website owner.Maximum Storage Duration: 1 yearType: HTTP Cookie _clsk[x3]Registers statistical data on users' behaviour on the website. Used for internal analytics by the website operator. Maximum Storage Duration: SessionType: HTTP Cookie _cltkRegisters statistical data on users' behaviour on the website. Used for internal analytics by the website operator. Maximum Storage Duration: SessionType: HTML Local Storage
* Spotify 1 Learn more about this providersentryReplaySessionRegisters data on visitors' website-behaviour. This is used for internal analysis and website optimization. Maximum Storage Duration: SessionType: HTML Local Storage

Marketing 17- [x] Marketing cookies are used to track visitors across websites. The intention is to display ads that are relevant and engaging for the individual user and thereby more valuable for publishers and third party advertisers.

*   [Google 3](https://www.aisi.gov.uk/blog/from-bugs-to-bypasses-adapting-vulnerability-disclosure-for-ai-safeguards#)[Learn more about this provider![Image 8: opens in a new window](blob:http://localhost/7c37f61654102a88a9be209f9569ef78)](https://business.safety.google/privacy/ "Learn more about this provider Google's privacy policy - opens in a new window")Some of the data collected by this provider is for the purposes of personalization and measuring advertising effectiveness. The provider may use the IP Addresses for ads measurement and ads personalization.

_gcl_au[x2]Used to measure the efficiency of the website’s advertisement efforts, by collecting data on the conversion rate of the website’s ads across multiple websites.Maximum Storage Duration: 3 monthsType: HTTP Cookie _gcl_lsTracks the conversion rate between the user and the advertisement banners on the website - This serves to optimise the relevance of the advertisements on the website. Maximum Storage Duration: PersistentType: HTML Local Storage
* Microsoft 7 Learn more about this providerMR[x2]Used to track visitors on multiple websites, in order to present relevant advertisement based on the visitor's preferences. Maximum Storage Duration: 7 daysType: HTTP Cookie MUID[x2]Used widely by Microsoft as a unique user ID. The cookie enables user tracking by synchronising the ID across many Microsoft domains.Maximum Storage Duration: 1 yearType: HTTP Cookie SRM_BTracks the user’s interaction with the website’s search-bar-function. This data can be used to present the user with relevant products or services. Maximum Storage Duration: 1 yearType: HTTP Cookie ANONCHKRegisters data on visitors from multiple visits and on multiple websites. This information is used to measure the efficiency of advertisement on websites. Maximum Storage Duration: 1 dayType: HTTP Cookie SMRegisters a unique ID that identifies the user's device during return visits across websites that use the same ad network. The ID is used to allow targeted ads.Maximum Storage Duration: SessionType: HTTP Cookie
* Spotify 2 Learn more about this providersp_landingUsed to implement audio-content from Spotify on the website. Can also be used to register user interaction and preferences in context with audio-content - This can serve statistics and marketing purposes. Maximum Storage Duration: 1 dayType: HTTP Cookie sp_tUsed to implement audio-content from Spotify on the website. Can also be used to register user interaction and preferences in context with audio-content - This can serve statistics and marketing purposes. Maximum Storage Duration: 1 yearType: HTTP Cookie
* YouTube 5 Learn more about this provider__Secure-ROLLOUT_TOKENUsed to track user’s interaction with embedded content.Maximum Storage Duration: 180 daysType: HTTP Cookie __Secure-YECStores the user's video player preferences using embedded YouTube videoMaximum Storage Duration: SessionType: HTTP Cookie __Secure-YNIDUsed to track user’s interaction with embedded content.Maximum Storage Duration: 180 daysType: HTTP Cookie VISITOR_INFO1_LIVETries to estimate the users' bandwidth on pages with integrated YouTube videos.Maximum Storage Duration: 180 daysType: HTTP Cookie YSCRegisters a unique ID to keep statistics of what videos from YouTube the user has seen.Maximum Storage Duration: SessionType: HTTP Cookie

Unclassified 2 Unclassified cookies are cookies that we are in the process of classifying, together with the providers of individual cookies.
- Gov.uk 2 Learn more about this providerncscCyberSprinterPendingMaximum Storage Duration: PersistentType: HTML Local Storage prev_pagePendingMaximum Storage Duration: SessionType: HTML Local Storage

Cross-domain consent[#BULK_CONSENT_DOMAINS_COUNT#]

[#BULK_CONSENT_TITLE#]

List of domains your consent applies to: [#BULK_CONSENT_DOMAINS#]

Cookie declaration last updated on 9/24/26 by Cookiebot

[#IABV2_TITLE#]

[#IABV2_BODY_INTRO#]

[#IABV2_BODY_LEGITIMATE_INTEREST_INTRO#]

[#IABV2_BODY_PREFERENCE_INTRO#]

[#IABV2_LABEL_PURPOSES#]

[#IABV2_BODY_PURPOSES_INTRO#]

[#IABV2_BODY_PURPOSES#]

[#IABV2_LABEL_FEATURES#]

[#IABV2_BODY_FEATURES_INTRO#]

[#IABV2_BODY_FEATURES#]

[#IABV2_LABEL_PARTNERS#]

[#IABV2_BODY_PARTNERS_INTRO#]

[#IABV2_BODY_PARTNERS#]

About

Cookies are small text files that can be used by websites to make a user's experience more efficient. You can learn more about cookies on the ICO website.

The law states that we can store cookies on your device if they are strictly necessary for the operation of this site. For all other types of cookies we need your consent.

This site uses different types of cookies. Some cookies are placed by third party services that appear on our pages.

You can at any time change or withdraw your consent to optional cookies from the Cookie Declaration on our website.

Learn more about who we are, how you can contact us and how we process personal data in our Privacy Notice.

Please state your consent ID and date when you contact us regarding your consent.

See below for more information about other NCSC domains.

Do not sell or share my personal information

Deny Allow selection Customize Allow all

Home

Utility navigation

Open Search Menu

Main navigation

Advice & guidance

Back## Advice & guidance

Our advice & guidance covers a broad range of topics * #### Find information for...

    *   [You & your family](https://www.aisi.gov.uk/section/advice-guidance/you-your-family)
    *   [Small & medium sized organisations](https://www.aisi.gov.uk/section/advice-guidance/small-medium-sized-organisations)
    *   [Large organisations](https://www.aisi.gov.uk/section/advice-guidance/large-organisations)
    *   [Self employed & sole traders](https://www.aisi.gov.uk/section/advice-guidance/self-employed-sole-traders)
    *   [Public sector](https://www.aisi.gov.uk/section/advice-guidance/public-sector)
    *   [Cyber security professionals](https://www.aisi.gov.uk/section/advice-guidance/cyber-security-professionals)
    *   [All advice and guidance articles](https://www.aisi.gov.uk/section/advice-guidance/all-articles)

*   #### Popular topics

    *   [Artificial intelligence](https://www.ncsc.gov.uk/section/advice-guidance/all-topics/artificial-intelligence)
    *   [Cyber attack](https://www.ncsc.gov.uk/section/advice-guidance/all-topics/cyber-attack)
    *   [Frontier AI](https://www.aisi.gov.uk/section/advice-guidance/all-topics/frontier-ai)
    *   [Passkeys](https://www.aisi.gov.uk/section/advice-guidance/all-topics/passkeys)
    *   [Phishing](https://www.aisi.gov.uk/section/advice-guidance/all-topics/phishing)
    *   [All topics](https://www.aisi.gov.uk/section/advice-guidance/all-topics)

*   #### Other resources

    *   [Glossary of terms](https://www.aisi.gov.uk/section/advice-guidance/glossary)
    *   [Infographics](https://www.aisi.gov.uk/section/infographics/home)
    *   [Verify an NCSC contact](https://www.aisi.gov.uk/guidance/how-to-spot-scammers-claiming-to-be-from-the-ncsc)
    *   [Heightened cyber threat information](https://www.aisi.gov.uk/section/keep-up-to-date/heightened-threat)
    *   [Cyber Governance for Boards](https://www.aisi.gov.uk/cyber-governance-for-boards/overview)

Respond to a cyber attack

Back## Respond to a cyber attack

Resources for individuals and organisations in the UK who have experienced an online scam or cyber attack. * #### Find advice for...

    *   [Advice for medium to large organisations](https://www.ncsc.gov.uk/section/respond-recover/medium-large)
    *   [Advice for sole traders and small businesses](https://www.ncsc.gov.uk/section/respond-recover/sole-small)
    *   [Advice for individuals](https://www.ncsc.gov.uk/section/respond-recover/you)

*   #### Respond to...

    *   [Hacked account](https://www.ncsc.gov.uk/section/respond-recover/hacked-accounts)
    *   [Infected device](https://www.ncsc.gov.uk/section/respond-recover/citizen-infected-devices)
    *   [Ransomware attack](https://www.ncsc.gov.uk/section/respond-recover/ransomware-attack)
    *   [Suspicious messages](https://www.aisi.gov.uk/section/respond-recover/phishing)

Find a product or service

Back## Find a product or service

Find a range of products & services from NCSC and certified 3rd party suppliers * #### In this section

    *   [Products & services overview](https://www.aisi.gov.uk/section/products-services/overview)
    *   [Browse cyber security services](https://www.aisi.gov.uk/section/products-services/browse-services)
    *   [NCSC assured services](https://www.aisi.gov.uk/section/products-services/assured-services)
    *   [Assuring technology](https://www.aisi.gov.uk/section/products-services/assuring-technology)
    *   [Search assured providers](https://www.aisi.gov.uk/section/products-services/search)
    *   [Cyber Essentials](https://www.aisi.gov.uk/cyberessentials/overview)

*   #### Useful links

    *   [Active Cyber Defence services](https://www.aisi.gov.uk/section/active-cyber-defence/services)
    *   [MyNCSC](https://www.aisi.gov.uk/information/myncsc)

Education & skills

Back## Education & skills

Working with industry, government and academia to support the next generation of researchers, students and cyber security professionals * #### Find information for...

    *   [Schools](https://www.aisi.gov.uk/section/education-skills/schools)
    *   [Higher education](https://www.aisi.gov.uk/section/education-skills/higher-education)
    *   [Professional skills training](https://www.aisi.gov.uk/section/education-skills/professional-skills-training)
    *   [Working with the NCSC](https://www.aisi.gov.uk/section/education-skills/working-with-the-ncsc)
    *   [Research & academia](https://www.aisi.gov.uk/section/education-skills/research-and-academia)

*   #### Education & academia resources

    *   [Cyber Skills](https://www.ncsc.gov.uk/cyber-skills)
    *   [Training resources for school staff](https://www.aisi.gov.uk/information/cyber-security-training-schools)
    *   [Resources for higher education institutions](https://www.aisi.gov.uk/section/education-skills/cyber-security-heis-feis)
    *   [Cyber Security Body of Knowledge (CyBOK)](https://www.aisi.gov.uk/section/education-skills/cybok)

*   #### Industry & skill resources

    *   [CYBERUK 2026](https://www.aisi.gov.uk/section/keep-up-to-date/cyberuk)
    *   [NCSC for StartUps](https://www.aisi.gov.uk/section/ncsc-for-startups/overview)
    *   [Industry 100](https://www.aisi.gov.uk/section/industry-100/about)
    *   [Cyber League](https://www.aisi.gov.uk/information/cyber-league)

News

Back## News

All the latest information to help you keep track of what's happening * #### In this section

    *   [News](https://www.aisi.gov.uk/section/keep-up-to-date)
    *   [Blogs](https://www.aisi.gov.uk/section/keep-up-to-date/all-blogs)
    *   [Speeches](https://www.aisi.gov.uk/section/keep-up-to-date/all-speeches)
    *   [Reports & advisories](https://www.aisi.gov.uk/section/keep-up-to-date/reports-advisories)
    *   [Malware analysis reports](https://www.aisi.gov.uk/section/keep-up-to-date/malware-analysis-reports)
    *   [CYBERUK 2026](https://www.aisi.gov.uk/section/keep-up-to-date/cyberuk)

*   #### Stay up to date

    *   [The NCSC's Cyber Series podcast](https://www.aisi.gov.uk/section/cyber-series-podcast/overview)
    *   [Subscribe](https://www.aisi.gov.uk/register/subscribe)

Close Search

Search

From bugs to bypasses: adapting vulnerability disclosure for AI safeguards

Exploring how far cyber security approaches can help mitigate risks in generative AI systems

Dr Kate S,Dr Robert Kirk

Visual Generation via Getty Images

As AI systems become more powerful, so do the risks of misuse. From the tangible real-world harms caused by insufficient safeguards which we see today, to the longer term potential risks from malicious actors, the stakes are high. This blog explores how traditional cyber security practices can help mitigate these risks, with a particular focus on public disclosure programmes.

This blog is aimed at:

decision-makers involved in the design, development, and deployment of AI systems
researchers exploring AI safety and security
anyone interested in frontier AI and/or cyber security

Scope

We focus on 'frontier AI' systems using general-purpose models such as ChatGPT, Gemini, Llama, and Claude. However, the insights may apply more broadly.

What are safeguard bypasses?

Safeguards are techniques developers use to prevent AI systems from producing policy-violating outputs or actions. These include model-level changes like refusal training or unlearning, and external tools like auxiliary classifiers. But these safeguards aren’t foolproof and can be bypassed through techniques such as jailbreaking, agent hijacking and indirect prompt injection.

Can cyber security practices help?

We, the NCSC and the AI Security Institute (AISI) have been considering how traditional cyber management tools might help mitigate the possibility of safeguard bypasses, focusing initially on the approaches to vulnerability management and disclosure described in the NCSC’s Vulnerability Disclosure Toolkit. Key areas of transfer include secure development lifecycles to minimise built-in weaknesses, and effective triage and remediation planning.

We think applying these foundations will probably help mitigate safeguard bypasses as much as they do standard software vulnerabilities.

Using disclosure programmes to improve AI security

One area of activity that is receiving increased attention in the frontier AI community is that of crowdsourcing details of safeguard bypasses, as seen in the recently launched OpenAI and Anthropic bug bounty programmes. Safeguard Bypass Bounty Programmes (SBBP) and Safeguard Bypass Disclosure Programmes (SBDP) are similar to bug bounty and vulnerability disclosure programmes in cyber security. They crowdsource security testing of a system by encouraging a researcher community to find and report successful bypasses.

Before considering a public programme AI system developers must first implement robust and mature approaches to security management and responsible disclosure. Without these, reported bypasses might not be handled properly, undermining the purpose of the exercise. And serious vulnerabilities could go unreported if participants don’t trust the disclosure process.

Public disclosure programmes: potential benefits and other considerations

SBBPs and SBDPs can benefit the security of the AI system in 2 main ways:

### By measuring how hard it is to bypass safeguards.

If a well-run programme attracts skilled participants who can’t find any successful bypasses, it’s a good sign. It suggests the system is harder to misuse, and this can help inform internal governance and risk assessments.

### By keeping safeguards strong after deployment.

AI system developers can’t currently identify all potential bypass techniques in advance. Even after deployment, these programmes help discover new weaknesses, allowing developers to fix them and keeping the system secure over time.

There are additional potential benefits to public disclosure programmes, such as:

Encouraging a culture of responsible disclosure, by incentivising and promoting ethical behaviour (although there are some caveats, see below), and encouraging collaboration between researchers and developers.
Increasing brand awareness and engagement among the security community and projecting a sense of security around the product.
Providing an opportunity for researchers to practice and demonstrate a range of real-world security skills.

Other factors to consider:

Companies don’t necessarily need to provide a financial incentive to get many of the benefits. It's likely beneficial to provide a range of incentives on top of purely financial ones.
The breadth and diversity of evaluation of public programmes should supplement, not replace, deeper security evaluations.
There are significant overheads associated with triaging and managing reports.
It won’t be effective unless the developers have good foundational security practices in place.

What makes a good disclosure programme?

We have developed some suggested best practice principles for using SBBPs and SBDPs, building on AISI’s experience collaborating on and judging the Gray Swan Agent Red-Teaming Challenge and evaluating frontier AI safeguards, as well as the NCSC’s internal research.

It has a clearly defined scope. A well-defined scope helps participants understand exactly what success looks like. For example, a vague scope such as ‘find inputs that cause the system to output harmful content’ is difficult to assess. Definitions of ‘harmful’ will vary and are hard to quantify. Instead, a detailed model spec containing, for example, the instruction to ‘never generate sexual content’, combined with a scope targeting violations of that spec, is much clearer to understand and evaluate.
**Its launch and duration support its goals.**Developers should launch the programme only after conducting internal reviews and fixing any discovered weaknesses, to avoid swamping developers with low impact reports from trivial weaknesses. If risks are likely to emerge at or after product launch, the SBBP or SBDP should be launched alongside the product, if not before. If the aim is to monitor safeguard bypass difficulty over time or maintain integrity post-deployment, a short timeframe will be less effective.
Reports are easy to track and reproduce. To effectively learn from reports, developers must be able to easily track and reproduce what users found. Ways to do this includes: * logging all messages with unique IDs * providing simple tools for users to copy and share the full conversation context * giving trusted users access to an internal version of the system with more detailed tracking.

Please note: We welcome programmes like SBDPs and SBBPs to encourage and support cyber security analysis of AI models. But note that the presence of an SBDP and SBBP does not automatically mean the model or system is safe or secure. We’re encouraging further research on this and other questions (see below).

Open questions and further research

There’s still a lot that we don’t understand about the role public programmes can play in AI security, and how we can make them – and other tools inspired by standard cyber practice – most effective.

We're assuming that lessons from decades of cyber security apply to AI systems, but there are probably also important differences. For example:

Many safeguard attacks are more like detection bypasses than software vulnerabilities. If this is the case, can other areas of cyber security offer tools or insights here?
The types of people involved in AI-related research may well be different too. Safeguard attack research is more accessible and a diversity of perspectives and skills is likely even more important. So, how do our approaches to incentives need to change or expand?

The NCSC and AISI encourage researchers across disciplines to explore these and other open questions, including:

Once a safeguard weakness has been found, how can it be mitigated? Standard software vulnerabilities can generally be ‘patched’ (that is, fixed with a high level of confidence). Training against the specific attack may make models robust to that attack, but it’s unclear whether retraining (or a similar technique) improves robustness more generally. How can we have a similar level of confidence for changes made to AI systems as we do for patches to standard software?
How should we handle attacks that transfer across models and programmes? Individual safeguard bypass approaches are often successful against multiple AI systems, not just those from one company. What frameworks or methods for collaboration and sharing across the sector (such as theFrontier Model Forum info-sharing agreement) could help disseminate discovered vulnerabilities safely and effectively to mitigate risk across the whole ecosystem?
How should we judge the severity of safeguard bypass weaknesses, especially in the general-purpose case where we don’t know the deployment context? Unlike in cyber, established principles for judging the severity of safeguard attacks don't yet exist. Anthropic has proposed judging jailbreak severity by:

(i) how capable the model response was compared to a model without safeguards, and

(ii) generalisability across types of prompts.

It’s a promising framework but needs further refinement, and an exploration of alternatives.

On public programmes specifically:

How public and open should SBBPs and SBDPs be? More open programmes encourage diversity of submissions, whilst invite-only programmes make it easier to manage risks. A hybrid model, where any user can participate but trusted testers are given extra access and affordances, may provide the best of both worlds. This needs testing.
What incentives are most appropriate for the safeguard context? The economics of vulnerability disclosure is still an active area of sociotechnical research, and we’re not aware that safeguard attacks specifically are being considered. Even in traditional software, incentives are complex, and we want to avoid inadvertently introducing perverse incentives: prioritising low-impact reports or duplication of effort, for example.

How you can help

Public disclosure programmes may help track and mitigate risks of powerful AI systems during deployment, and are an example of transferring best-practice from cybersecurity to safeguarding AI. But there’s a lot more to be learned about the ways in which existing cyber approaches apply to AI systems. And there is more to be done to uncover what else the AI and cyber communities can learn from each other.

We welcome further research from both AI and cyber security disciplines to identify and harness the opportunities which will help us achieve the safe and secure AI outcomes we all want to see.