← Library · Concept
AI in Cybersecurity — The New Layer of Attack and Defense
AI can accelerate code analysis, alert triage and attack preparation. Its value depends on tools, data and verification; an agent does not replace patching, access control or the security team's accountability.
What AI changes in security
A language model can help read unfamiliar code, explain a suspicious event or draft a fix. The same capabilities can support attack preparation. Accelerating one step, however, does not prove that a model can handle an entire incident autonomously or reliably find unknown bugs in arbitrary systems.
One concrete result is Big Sleep's discovery of a SQLite vulnerability. Google Project Zero described targeted variant analysis of known bugs, not a universal machine for cheap zero-day discovery. The issue was fixed before the affected code was released. The team's original report.
Defense Through Agents
A reasonable starting point is an assistant that groups related alerts, builds a timeline and links each claim to its source log. An analyst can then decide what to investigate more quickly. The model may also connect unrelated events or invent an explanation. The risk of convincing but false output is discussed in NIST's generative AI profile.
Measure false positives, missed incidents and evidence-backed conclusions as well as time saved. Automatically closing cases or blocking accounts requires different permissions and verification from merely summarising them.
A safe evaluation needs real boundaries
An agent with tools can cause harm even in a task labelled as a test. Define the target systems, network access, account and time budget for your evaluation. Use test data, record tool calls and provide a way to stop the run. The required isolation depends on the agent's powers and the test; an instruction to stay inside a sandbox does not enforce it.
Do not treat text from a log or external page as an operator's command. An attacker may control some of that data. Security across the model, data and operations is covered by the joint NCSC and partner guidelines.
Example: an unusual sign-in alert
The assistant receives read-only access to selected events. It builds a timeline, identifies the source records and states what is missing for a decision. A security analyst checks the context and can revoke the session through a separate authorised procedure. If essential data is missing, the output should say that a decision is not possible, rather than automatically closing the case.
AI supplements basic defence. Patching, limited permissions, segmentation and rehearsed incident response remain important against attackers using models. They are also emphasised in the joint agency statement published by NCSC. Do not substitute promises of confusing an attacker with textual traps for these controls.
Sources
- Google Project Zero: From Naptime to Big Sleep — a specific research result and its conditions.
- NIST AI 600-1 — generative AI risks.
- Guidelines for secure AI system development — secure design, deployment and operation.
- The AI shift in cyber risk — AI and security fundamentals.