AI Is Compressing Cyber Conflict to Machine Speed: Three Talks From AvengerCon X

I’ve already written about Day 1’s hands-on workshop and The origin story I heard on Day 2. This post covers the three Day 2 talks that stuck with me the most: NVIDIA’s four years of attacking AI systems, an autonomous adversary-emulation platform built for defenders, and a hard look at how AI has changed social engineering. Three very different speakers, one shared conclusion: both attackers and defenders are being pushed to operate at a speed that didn’t exist a few years ago, and the humans on both sides are the ones struggling to keep up.

Four years of attacking AI systems, from NVIDIA’s red team

Joe Lucas walked through NVIDIA’s evolution from conventional adversarial machine learning research to full operational AI red teaming, and the shift in his own thinking was the most useful part.

Before ChatGPT, the team’s work looked like classic adversarial ML: evasion attacks, inference attacks, data poisoning, and membership inference, tested against medical imaging classifiers and federated learning setups. Data poisoning was originally rated as low priority, on the theory that an adversary would rarely have access to training data. That assumption didn’t survive contact with reality. Research and development pipelines, it turns out, are “fast and loose” compared to production systems, and training data is often far easier to reach than a hardened production endpoint ever would be.

The team’s charter settled around three principles worth stealing for any security practice: be threat-informed (ground everything in realistic scenarios, not hypotheticals), be algorithmic (if there’s no math involved, you’re really just doing web app hacking with extra steps), and be ecosystem-aware (most open-source ML tooling doesn’t matter to an attacker; the job is figuring out which parts do). One of their recurring findings was that JupyterLab, despite being documented as single-tenant, gets deployed multi-tenant or without authentication constantly, opening the door to things like in-memory label flipping (where the disk labels stay correct but the model returns the wrong output) and exploitation through Python’s pickle format, which allows code execution on deserialization by design.

Post-ChatGPT, the focus shifted toward operations against internal chatbots, RAG applications, and AI-assisted IDEs, deliberately staying away from scoping jailbreak/harm-content work. The vignettes from that shift were the most concrete part of the talk. A chat-with-data guardrail bypass had to clear three separate layers in sequence: evade a guardrail LLM acting as a judge, navigate an SDK’s own preprocessing logic, and then inject actual code rather than natural language into a pandas code-generation step. Lucas’s framing stuck with me: the modern AI attack payload isn’t a clever sentence anymore, it’s an encapsulated instruction set that has to successfully navigate multiple independent layers of a system to land. To study that at scale, the team built its own assessment tool that spins up websites on the fly, watches what an agent chooses to fetch, and produces a heat map of where prompt-injection attempts actually land, so they can tune payloads against real agent behavior instead of guessing.

He also made the case that AI systems aren’t static endpoints, unlike a traditional web server. Sessions, shared context, cluster orchestration, and inference optimizations are all now live attack surfaces. A KV cache bug in ChatGPT once enabled cross-tenant access. Systems like Mooncake move cached state between compute nodes, and speculative decoding runs several models in parallel, both of which introduce timing side-channel possibilities that don’t exist in simpler architectures.

One of his most practical warnings had nothing to do with a specific exploit. He described an incident where a non-technical user followed instructions from a chatbot, leaked a JWT in the process, and left behind no logs that could distinguish an honest mistake from a deliberate attack. Many vendors simply don’t expose the telemetry needed to tell those two apart, which turns “assess your vendors on what they let you log” into a real procurement question rather than a compliance checkbox.

His predictions are worth sitting with. He kept returning to the point that the model itself is the least durable part of any attack, since new ones ship every week and a payload tuned to one may not transfer to the next, while system-level and lifecycle attacks age far more slowly. Handcrafted prompt injection (the “pretend you have no restrictions” style of jailbreak) is dying, because models are increasingly resistant to it, and the frontier is moving toward algorithmic optimization methods that search for working payloads the way you’d search for an adjacent problem in optimization theory. Physical systems (cars, robots, drones, industrial control systems) are the domain he considers most interesting going forward, precisely because they’re compute- and latency-constrained in ways that make guardrails expensive to run, and a defender’s constraint is always an attacker’s advantage. And the most durable attack surface of all, in his view, isn’t a specific model or a specific bug: it’s alignment itself. Models are trained hard to be helpful and to solve the task in front of them, and that same drive is exactly what an attacker can lean on. His example: a sliding puzzle challenge with a command embedded in it, no adversarial prompting required at all, because the model just tried to solve everything presented to it, including the part that wasn’t meant to be solved.

Autonomous adversary swarms, built for defenders

Madhav Vats and Austin Brown from Method Security presented the flip side: a platform built to give defenders their own AI-driven offensive capability, under tight rules of engagement, so blue teams can find out what an AI-accelerated adversary would actually do to their environment before a real one does.

The platform treats agents as a first-class part of the workflow across three modes: fully manual, co-pilot (the agent suggests, the operator approves), and full auto within defined boundaries. Agents are built from two components: tools, which are structured wrappers around scanners, exploits, connectors, and custom scripts, and skills, markdown-encoded tradecraft that tells the agent how to actually use those tools. You can configure known adversary tactics, techniques, and procedures and have the platform emulate how that specific adversary would move through your environment, rather than a generic attack simulation. Underneath the agents sits a deployable C2 layer they call Jackals: modular implants that can run cloud-managed for scale or host-deployed for precision, on targets as varied as laptops, phones, and drones, all driven through the same infrastructure.

The live demo was the part that made it real. A multi-day operation, run against an actual environment, started from an assumed-breach foothold via email phishing, progressed to domain admin using the platform’s own tooling, and automatically generated a full network map and linked entity graph along the way, saved afterward as a reusable plan for future runs. The technical detail that impressed me most was the ontology layer beneath it all: every tool operates on structured objects in a knowledge graph, not raw strings, which makes enforceable no-strike lists possible. You can’t accidentally point a scan at something out of scope if the system only understands “scan this object,” and that object was never in the graph to begin with. The graph even tracks provenance at the property level, recording which tool produced which fact, when, and from what raw signal, and it enforces access controls at that same granularity, so the system always knows not just what it believes about a target but where that belief came from.

Guardrails are enforced at three separate levels, deliberately redundant: at the package level, the system can block progression toward a target without explicit approval; at the agent level, tool and skill selection is limited by the agent’s own definition, with user approval required on specific calls; and at the tool level, no-strike lists on IPs and domains, plus risk thresholds tied to network footprint, will block execution outright if a threshold is crossed. Notably, all actual tool parameterization is compiled by the system itself, not generated by the agent, which keeps behavior deterministic even when the agent’s reasoning is not. Operators can even bring their own tools into the platform and still have the same guardrails applied to them uniformly. That’s a distinction worth remembering any time you’re evaluating an “agentic” security tool: the interesting engineering isn’t the model; it’s the harness that decides what the model is actually allowed to do.

AI hasn’t just changed the phishing email; it’s changed the odds

The last talk was less about AI systems as targets and more about AI as a force multiplier for the oldest attack in the book: getting a human to make a mistake. The numbers presented were sobering. Attacking a person takes 20 to 30 minutes; attacking a system properly can take days to months. In the studies cited, 83% of targeted users fell victim to a well-crafted attempt, 97% of people globally couldn’t identify a sophisticated phishing email, and 70% knew the underlying risks and clicked anyway. The average organization still takes around 207 days to identify a breach and another 70 days to contain it, a gap measured in months compared with an initial compromise that took minutes.

AI’s contribution has been to strip out nearly every traditional warning sign. Misspellings and awkward phrasing are gone; branding and tone are now correct by default. Voice cloning needs as little as 20 seconds of sample audio, which has made the “grandparent scam” nearly undetectable by ear alone; the only reliable mitigation offered was a family safe word that’s never posted anywhere online. Large language models can scrape multiple social platforms simultaneously to build a detailed personal profile, and a lure built from someone’s Facebook, Twitter, and Instagram activity combined outperforms anything a human social engineer could hand-craft alone. A cited study comparing phishing emails generated by different methods found that the simplest, most personalized, on-brand approach (referred to as V-Triad) beat more elaborate AI-generated attempts on click-through rate, and that AI-generated emails scored lower on recipient suspicion than the human-written control group. The best-performing emails did two things: they used the recipient’s real name and referenced their actual institutional affiliation.

None of this is hypothetical. Social engineering was behind the Anthem breach in 2015, the OPM breach in 2016 (which affected nearly the entire federal and military personnel population), SolarWinds, and a 2024 phishing campaign attributed to APT28 (Fancy Bear) targeting webmail infrastructure. Nation-state actors still rely on spear phishing as their primary way in, not because it’s exotic, but because it works.

The defensive side isn’t standing still either: automated phishing detection that flags probability rather than certainty, malware analysis that compresses hours of manual reversing into seconds, anomaly detection tuned to catch things like fragmented data exfiltration or data hidden in the timing gaps of VoIP calls, and user and entity behavior analytics watching for the moment someone starts acting like they’ve been compromised. But the takeaway that mattered most was procedural, not technical: the core rule is never click, verify by calling the sender directly, and never reply in the same thread a suspicious message arrived in. And the speed of reporting is the whole game. The difference between a 20-second report and a 207-day discovery isn’t a matter of degree; it’s the difference between an incident and a breach.

The thread connecting all three

Put the three talks side by side, and a pattern emerges. NVIDIA’s talk showed that AI systems themselves are now a target surface with novel failure modes of their own. Method Security’s talk showed that AI can just as easily be harnessed by the defender, provided the guardrails around it are engineered as carefully as the offense it enables. And the social engineering talk showed that even when the target is still a human being, AI has already erased most of the tells people used to rely on. Machine speed cuts both ways. The organizations that come out ahead won’t be the ones with the fanciest model. They’ll be the ones who engineered the harness, the guardrails, and the human habits around it with as much care as the AI itself.