Last updated: January 2026
Cutting-Edge Research & Threats (2025-2026)
- EchoLeak (CVE-2025-32711) - Microsoft Copilot zero-click exploit where infected emails with engineered prompts automatically exfiltrate sensitive data without user interaction
- OWASP Top 10 for Agentic Applications 2026 - New framework addressing autonomous AI agent security risks, distinct from traditional LLM vulnerabilities (developed with 100+ experts)
- FlipAttack Jailbreak Technique - Achieves ~98% attack success rate on GPT-4o by altering character order in prompts
- AI Agents as Autonomous Insiders - Organizations now have 82-to-1 ratio of AI agents to humans; single compromised agent can poison 87% of downstream decisions within 4 hours
- Black Hat 2025: Targeted Promptware Attacks - Demonstrated Gemini for Workspace compromise via Google Calendar invitations
- Sugar-Coated Poison Attack - Stealth prompt injection hiding malicious intent behind benign reasoning
- DeepSeek-R1 CCP Censorship Vulnerability - Politically sensitive prompts increase likelihood of severe security vulnerabilities in generated code by up to 50%
- Malicious MCP Server Attacks - Rogue Model Context Protocol servers can inject code into development environments like Cursor IDE
- Agentic AI Security Survey - 94.4% of LLM agents vulnerable to prompt injection, 83.3% to retrieval backdoors, 100% to inter-agent trust exploits
- Multi-LLM Orchestrated Attacks - Attackers using coordinated LLMs for sophisticated reconnaissance, exploitation, and evasion
Frameworks & Standards
- OWASP GenAI Security Project - The definitive guide for securing LLM applications, with the LLM Top 10 vulnerabilities framework used across the industry
- OWASP LLM Top 10 (2025) - Prompt injection is #1, appearing in 73% of production AI deployments
- MITRE ATLAS - Machine learning attack matrix covering adversarial ML techniques from model poisoning to evasion attacks
- NIST Adversarial Machine Learning - Taxonomy and terminology of attacks and mitigations for ML systems
- Model Context Protocol (MCP) - Anthropic's open standard for connecting AI systems with data sources, replacing fragmented integrations
- CISA: Deploying AI Systems Securely - Government guidance on secure AI system development
Security Tools & Projects
- DIANA - Automates detection rule creation from threat intelligence using LLMs, created by Dylan Williams for reducing SOC toil
- SigmaGen - AI-powered system that extracts MITRE ATT&CK techniques from threat intel and generates SIGMA detection rules automatically
- LLMCloudHunter - Research framework leveraging LLMs to generate detection rules from cloud-based cyber threat intelligence
- Dylan Williams' LLM Security Collection - Comprehensive curated list of LLM and GenAI resources for cybersecurity applications
- OpenSSF AI/ML Security Working Group - Explores security risks in LLMs, GenAI, and ML systems including prompt injection and model poisoning
- Lakera Guard - Platform for detecting and defending against prompt injection attacks in production LLM applications
Research Papers
- Hierarchical Multi-agent Reinforcement Learning for Cyber Network Defense - Alina Oprea's research on applying RL to automated network defense
- Adversarial Machine Learning: A Taxonomy and Terminology - NIST's comprehensive framework for ML attacks and mitigations, authored by Alina Oprea
- A Marauder's Map of Security and Privacy in Machine Learning - Foundational paper mapping the ML security threat landscape
- Advancing Autonomous Incident Response: Leveraging LLMs and CTI - RAG-based framework for incident response automation achieving 22% faster recovery times
- AI-Augmented SOC: A Survey of LLMs and Agents for Security Automation - Comprehensive survey of LLM applications in security operations
- Prompt Injection Attack Against LLM-Integrated Applications - Academic research on the most common AI exploit in 2025
Courses & Learning
- CMU Machine Learning in Production / AI Engineering - Carnegie Mellon's course on building production ML systems
- Building Effective Agents - Anthropic's engineering guide on when and how to build LLM agents vs. workflows
- A Practical Guide to Building Agents - OpenAI's comprehensive PDF guide for agent development
- SEC545S: GenAI and LLM Application Security - SANS course on securing generative AI applications
Articles & Analysis
- Utilizing Generative AI and LLMs to Automate Detection Writing - Dylan Williams' practical guide on using LLMs for detection engineering with hyper-specific prompts
- Predicting AI's Impact on Security - Caleb Sima's analysis of how AI will transform security operations
- Discovering Zero-Days with AI - Case study of GreyNoise using AI to find vulnerabilities in IoT devices
- Methodology for Incident Response on Generative AI Workloads - AWS Security Blog guide for responding to AI-specific incidents
- A Five Year Retrospective on Detection as Code - Caleb Sima's reflection on detection automation trends
- TL;DR: Every AI Talk from BSidesLV, Black Hat, and DEF CON 2024 - Comprehensive summary of AI security conference content by Clint Gibler
- How Microsoft Defends Against Indirect Prompt Injection - Microsoft's defense-in-depth approach to the #1 LLM vulnerability
Videos & Conference Talks
- Using AI to reduce toil in detection writing - Dylan Williams at MSSN CTRL 2024, demonstrating practical LLM applications in SOC automation
- On Your Ocean's 11 Team, I'm the AI Guy - Harriet Farlow's DEF CON 32 talk on AI in offensive security
- AWS re:Invent 2024 Security Talks - Playlist featuring AI-enhanced security innovations announced at re:Invent
- DEF CON 32 Main Stage Talks - Includes multiple AI security presentations
Agent Development Tools
- LangGraph - Framework for building stateful, multi-actor LLM applications with cycles and controllability
- Amazon Bedrock Agents - Fully managed service for building and deploying generative AI agents
- Rivet - Open-source visual AI programming environment for prototyping and debugging LLM applications
- Vellum - Development platform for building, testing, and monitoring production LLM applications
- LLM Automator - Tool for automating workflows with large language models
Companies & Platforms
- HuggingFace Models - Open-source model hub with thousands of pre-trained models and datasets
- Anthropic - AI safety company behind Claude, focusing on responsible AI development
- Mindgard - AI red teaming platform for testing LLM security, simulating prompt injection and jailbreak scenarios
People to Follow
- Alina Oprea - Northeastern University Professor researching machine learning and security, author of NIST ML adversarial framework
- Rich Harang - Principal Security Architect at NVIDIA, specializing in ML/AI systems security
- Dylan Williams - Detection engineering expert pioneering LLM applications in security operations, creator of DIANA
- Simon Willison - Creator of Datasette, writing extensively on LLM security and prompt injection research