Securing AI in Software Engineering: A Field Guide

One of the areas where AI is providing the most value and risk is in software engineering. In March 2026, Amazon held a large internal engineering meeting to address a "trend of incidents" characterised by "Gen-AI assisted changes" with "high blast radius". They are not alone. Developers giving instructions to Claude Code have seen production environments getting wiped and skills used to instruct coding agents contain both security-flaws and outright malicious payloads. 95% of software engineers use AI coding assistants. The most popular tools are Claude Code, Cursor, and GitHub Copilot.

The current state of cybersecurity is being fundamentally reshaped by this rapid evolution of AI. Nordic businesses are already investing more than 10% of their digital budgets in AI, with rising expectations of return on investment. Businesses transition from "AI on top" productivity tools (chatbots and copilots) to re-engineering core processes with AI, and ultimately to AI-native business models. This causes security requirements to shift further. 

As a cloud security company, we want to share our perspectives from our work in the field, and what we think is needed to enable AI-led innovation while running at acceptable risk. In this article we will cover:

1. How AI-assisted coding introduces novel and amplifies existing risks

2. Our field notes: learnings from risk assessments, pentesting vibecoded apps and securing agents in cloud environments

3. Overarching principles to securing AI 

4. Suggested actions for security teams to holistically address the risks

The focus of this article is securing AI and GenAI for software development. This is fundamentally different to e.g. securing LLMs in applications developed for customers. The key difference is being a consumer of GenAI services compared to being a provider of AI-enabled services. We will not cover securing AI/GenAI in general, privacy risks or other GenAI risks (e.g. safety, bias or future dependency), or challenges of using AI to augment security teams and tooling. 

How AI-assisted coding introduces novel and amplifies existing risks

Risks from AI-produced and AI-assisted code can be classified in two categories, (1) novel risks from AI and (2) amplification of existing risks. The distinction matters because novel risks require new controls, while amplified risks require scaling and strengthening of existing controls. Novel risks can chain into existing risks, e.g. a prompt injection can lead to installation of a malicious npm package or making a private GitHub repository public. 

Existing risks amplified by AI: 

• Insecure code generation

• Degraded code quality and maintainability

• Configuration mistakes and insecure implementation

• Failure in secret management (API keys, credentials) 

• Unvetted software supply chain (libraries, npm packages)

• Platform and environment misconfigurations (exposing internal S3 buckets) 

• Supply chain risk from vendors (CVEs in the tools)

These risks already exist in software development. With AI as a force multiplier, existing risks will increase in both frequency and velocity. An AI assistant can introduce the same vulnerable pattern across dozens of files in minutes, where a human developer might only make the mistake once. Existing risks around software supply chain and vendors also need to be reassessed in the context of AI. Your AI vendor is now processing your source code.

The architecture and design of AI causes novel risks to arise. Not only from the LLMs themselves, but there is a new technical ecosystem encompassing AI that increases attack surface and blast radius. Security teams need to handle MCP servers, rule file management, context management, and more. 

Novel AI-driven risks: 

• Prompt injection of untrusted content (malicious instructions hidden in files, issues, or documentation)

• Credential and secret theft via context windows (agent reads .env or cloud configs as part of its context)

• Rogue actions (e.g. infrastructure destruction via unconfirmed commands)

• Data exfiltration through agent actions (source code or secrets sent to external APIs)

• Persistent context poisoning (corrupted instructions that carry across sessions)

• Shadow AI tooling and ungoverned access (developers using unapproved tools or MCP servers)

• Unbounded resource consumption (runaway agents eating tokens or spinning up cloud resources)

• Cascading failures and insecure inter-agent communication (one compromised agent propagating to others)

• Non-deterministic security controls (agent instructions like claude.md or .cursorrules are prompt text, not enforced policy; they can be ignored, overridden, or bypassed)

How these chain together: A developer uses an AI coding agent to implement a feature. The agent reads a markdown file from a third-party repository as context. That markdown contains a hidden prompt injection instructing the agent to add a dependency to package.json. The agent complies, it has file system access and the instruction looks like a legitimate requirement. The malicious package now sits in the project, passing CI because it functions correctly while exfiltrating environment variables via a postinstall script. Each building block, prompt injection, agent file access, supply chain attacks, is a well-documented attack vector. AI coding agents are what chain them together.

If your company is building and training your own GenAI models, additional risks apply (e.g. data poisoning, model tampering, model exfiltration, etc.) which are not covered here. 

Our field notes: learnings from risk assessments, pentesting vibecoded apps and securing agents in cloud environments

In O3C we have helped several of our clients with security for AI in software engineering. Based on risk assessments, pentesting, and hands-on implementation across multiple clients, we see five themes.

1. Needed controls depend on your developers' AI proficiency 

The need for security controls is highly dependent on where developers sit on what we call the "AI Coding Spectrum", from no AI to multi-agent orchestration. There is a significant difference in impact and implications of using AI as an assistant or chatbot, compared to more autonomous applications of agentic AI.

To complicate matters, most companies have developers spread across multiple levels simultaneously. The security posture must match the highest level in active use, not the average. We distinguish between two axes: the level of AI autonomy, and whether the AI operates under local developer control or as a platform-level service.

AI Coding Spectrum*

*Loose adoption of model mentioned in the Pragmatic Engineer interview with Steve Yegge.

The local-vs-platform distinction matters for security. Levels 0-5 operate under the developer's identity, on their machine or IDE. The developer is the control point. Levels 6-7 operate as platform services under non-human identities, often without a human in the loop. The security model is fundamentally different: the control point shifts from the individual developer to the platform and its configuration. A misconfigured platform agent can affect every repository it touches.

2. Vibe-coded applications can be secure at time of test, but maintainability is the hidden risk 

We have tested vibe-coded applications that pass our assessment, but white-box testing reveals the code is held together with "duct tape and bubble gum". The risk of future updates causing vulnerabilities in code or business logic is close to certain. We observed an application where the agent had turned off the built-in authentication from the platform, and then implemented its own solution. The agent's solution for authentication was implemented (correctly) in 12 different places rather than using a single function. It technically works, but makes future updates risky.

3. Risk assessments need to be designed to scale and clearly defined. 

In our experience an underlying risk assessment of AI in software engineering is needed before diving into AI tool assessments. Extensive risk assessments of every individual AI tool will not scale, and only cause shadow AI to grow. It's also necessary to be clear on defining the challenge and topic. Are we discussing security for AI in software development, or securing enterprise agents, or GenAI use across the enterprise? The risks and controls are different for these related topics. 

4. Design and architecture matter more than ever in software development. 

Almost all AppSec tools focus on later phases of software development: code, deploy, and runtime. But with AI generating code at speed, flaws in design and architecture propagate faster and wider than before. In some set-ups it's the only line of defence left. We expect to see more tooling in this area going forward (Clover and Prime Security are already attempting), but until then teams need to spend time on design.

5. Legacy IT compounds the challenge. 

Modern AI coding tools assume cloud-native, containerised environments with CI/CD pipelines and automated gates. Individual developers can run devcontainers and sandboxes locally. The challenge is at the platform level: organisations without mature CI/CD, container orchestration, or infrastructure-as-code will struggle to enforce security controls at scale. The gap between AI tool maturity and platform maturity is a risk in itself.

6. The security for AI vendor landscape is crowded and unsettled. 

There are 150+ new companies created for different aspects of AI security in the last few years, and five distinct categories are competing for your budget:

1. New AI-native startups securing AI (e.g. Knostic, Noma, Ox)

2. Existing AppSec vendors adding AI capabilities (e.g. Aikido, GitHub, Snyk)

3. Cloud security platforms racing to secure AI (e.g. Sysdig, Upwind, Wiz)

4. Hyperscalers building security into their AI platforms (e.g. AWS, Google, Microsoft)

5. Foundation AI companies entering security directly (e.g. Claude, OpenAI) 

The market is moving faster than the clarity on what actually works. Real total cost of ownership for emerging tools is hard to predict as many startups use funding to subsidise prices. 

We suggest to lean on your existing security products and supplement with accessible open-source tools: clauditor for auditing Claude Code configurations, claude-code-config and claude-code-devcontainer from Trail of Bits for hardened setups, and git-ai for tracking AI-generated code. Wait for the market to settle before committing a budget to new point solutions, but don't wait on handling the challenge and implementing controls.

Overarching principles to securing AI 

With the rapid development and innovation around AI in coding, a lot is uncertain. What is certain is that AI is not going away. We believe the principles below are good fundamentals that will hold true going forward, even with the fast-paced development of AI.

Principles:

1. Embrace and adapt. Organizations cannot and should not prohibit the use of AI, adapt instead.

2. Identify and remove shadow AI. Visibility is the prerequisite for every other principle.

3. Keep humans in the loop. AI agents must not take high-risk actions without explicit human confirmation.

4. Reduce attack surface. Minimise what is connected and exposed.

5. Reduce blast radius. Assume AI will make mistakes and contain the damage.

6. Secure the full lifecycle. From prompt to production: context, code review, CI/CD, deployment, and runtime.

7. Make security the path of least resistance. Golden paths, pre-approved configs, secure-by-default environments.

8. Accept a deliberate velocity-risk trade-off. Organizations must decide consciously how much speed to trade for risk reduction, and revisit as the technology matures.

Longer term we see that keeping humans in the loop will be a challenge. Human code review will likely end up augmented, or fully automated. Still, we believe organizations should start stricter, and widen guardrails as maturity increases. Human attention is the scarcest resource; direct it where it provides the highest return on security.

Suggested actions for security teams to holistically address the risks

Addressing these risks requires pragmatism. Forbidding AI tools is not realistic. Instead, integrate security for AI into your existing security, developer and platform programmes, starting with governance. Using AI tools is a decision on risk for the company. Rarely has it been more true that risk is both risk of loss and risk of gain.

1. Map your AI tool landscape and establish lightweight governance

You cannot secure what you cannot see. Audit which AI tools and artefacts (e.g. MCP) developers actually use today, including unsanctioned ones. Use data to identify:

• IdP/OAuth logs: Audit your IdP for unrecognized AI clients requesting access and existing grants. Okta's Agent Discovery now maps these automatically.

• DNS/network logs: Monitor outbound traffic against AI API domains (api.openai.com, api.anthropic.com, etc.). Include cloud egress.

• Codebase scanning: Scan for hardcoded AI API keys, service endpoints, AI instruction files (e.g. claude.md, .cursorrules) and AI-generated code patterns.

• Endpoint telemetry: Track AI tool binaries, browser extensions, and web-based AI access via your existing EDR/UEM.

• Finance data: If a tool shows up in expense claims but not in your IT estate, it is shadow AI.

• People: Ask developers what they're doing to become more efficient. Frame it as enablement, not enforcement.

Maintain a living inventory that records each tool's name, intended use, network access model (what external APIs it calls, whether it can make arbitrary requests), and whether it supports confirmation prompts before execution. 

Approval criteria should be concrete: mandatory confirmation before shell execution, ability to exclude files and directories from context, and a documented security model. Critically, establish a lightweight onboarding path for new tools. A three-month approval process guarantees shadow AI. 

2. Require human-in-the-loop for high-risk actions

All agentic tools must require explicit developer confirmation before executing shell commands, writing or deleting files outside the project directory, making network requests, or running deployment commands. This is the single most important behavioural control. 

Configure through tool-level settings (Claude Code's allowedTools and permission modes, or equivalents in Cursor and Copilot). Equally important is session hygiene: start new sessions for unrelated tasks, do not paste untrusted content directly into agent prompts without flagging it as untrusted, and review all agent-proposed commands before confirming, especially anything involving curl, pipes to sh, or encoded strings.

3. Tier AI autonomy based on risk

Not all code and not all actions carry the same risk. Design a tiered model that maps AI permissions across two dimensions: what the agent can touch (repository or system criticality) and what the agent can do (read, write, execute, deploy, destroy). 

In low-risk contexts (internal tooling, documentation), agents may operate with fewer confirmation gates. In high-risk contexts (production services, infrastructure-as-code), enforce strict confirmation on all actions and restrict autonomous modes entirely. 

Enforce tiers through repository tagging, per-repo agent configuration files, and CI checks that validate agent config matches the assigned tier. Restrict flags like Claude’s --skip-permissions to low-risk repositories only. 

4. Isolate agent execution environments

Agentic AI tools must not run in environments with access to production systems, production credentials, or platform secrets. This is the single most important infrastructure control. Run agentic tools inside isolated, ephemeral containers (devcontainers, droplets or equivalent sandboxes) that are destroyed after use. The primary defence is not mounting sensitive paths into the working directory at all. 

As a second line of defence, enforce file and secret exclusion via .aiignore or equivalent:

# Secrets and credentials
.env
.env.*
*.pem
*.key
*credentials*
*secret*


# Infrastructure state
terraform.tfstate
terraform.tfstate.backup
.terraform/


# Internal docs that shouldn't be sent to external APIs
docs/internal/
architecture/
runbooks/
incident-reports/


# Cloud provider configs
.kube/
.config/cloud/

5. Scope credentials to minimum viable access

Agent environments get short-lived, narrowly scoped, read-only-by-default credentials only. No ambient credentials. No long-lived keys. No production access. 

Regardless of cloud provider, the approach is the same: 

• Create a dedicated service account for AI-assisted development

• Grant read-only access to sandbox or non-production resources only

• Explicitly deny access to production resources and destructive actions (delete, terminate, modify IAM) 

• Issue credentials via temporary token mechanisms (workload identity, service account impersonation, or short-lived access keys to sandbox)

• Pass only these temporary credentials into the sandbox container 

For Kubernetes, scope agent access to read-only operations on non-secret resources in development namespaces only.

6. Configure agent tool boundaries explicitly

Beyond confirmation prompts, define what agents are allowed to do through explicit allowlists for permitted commands and tools. Review agent configuration files, .claude/settings.json, .cursorrules, MCP server configs, as part of your code review process, since these files are checked into repositories and affect every developer who clones them. Use tooling like clauditor to audit configurations, and claude-code-config for hardened, pre-approved setups. If your organisation has standardised on a tool, publish a "golden path" configuration that is secure by default and easy to adopt.

7. Gate AI-generated code through existing review processes

Agent-generated code is untrusted input. It must pass the same peer review, static analysis, and security scanning as human-written code, no exceptions. Tag commits and PRs when AI tools were used so it is transparent to the reviewer. Flag AI-generated changes for additional scrutiny in high-risk repositories. 

8. Monitor agent activity and extend incident response

We cannot observe agent reasoning directly in third-party tools, but we can monitor downstream effects: unusual git activity, cloud API calls from sandbox roles, unexpected network egress from sandbox environments, and deployments correlated with AI-assisted PRs. 

Extend your incident response playbook to cover AI-specific scenarios. 

• If an agent-generated change causes an incident: revert first, review the full diff (agent code may contain subtle issues that passed review), check whether the agent had access to production credentials, and document in the post-incident review. 

• If credentials may have been exposed to an agent: rotate immediately, check the AI provider's data retention policy, audit cloud logs for unauthorised use, and tighten sandbox configuration.

9. Build a security culture that embraces AI, not resists it

None of the above works if developers see security as the team that says no. Position AI security controls as enablers, "we're making it safe for you to move fast" rather than "we're restricting what you can do". Publish clear, living policies for AI usage that function as guides, not just rule books. Involve developers in defining tiers and golden paths. And accept that these policies will need continuous updating, the tooling and threat landscape is moving faster than annual review cycles can handle.

Security teams should also start learning and understanding how the systems work, and how they are applied. This is the only way to generate pragmatic and risk-based implementations in your context. Start experimenting in your own workflows to understand the capabilities and be able to reason with the engineers that want to use these tools.

Conclusion

AI in software engineering is not slowing down, nor should it. The productivity gains are real, and organisations that refuse to adopt AI coding tools risk falling behind. But the security risks are equally real, and they compound with every level of autonomy granted to AI agents.

The good news: securing AI-assisted development is achievable with pragmatic controls. It does not require waiting for a silver-bullet product or rebuilding your security programme from scratch. Work through the actions above, and adapt them as tooling and the threat landscape evolves.

If your company is set on becoming AI-native, security must be equally transformative. Not as a barrier to innovation, but as the foundation that allows it to scale.