Earnie
ARTICLEAI PROVENANCE

What if AI wrote the next security breach?

AI assistance is becoming part of how software is built. The governance around it hasn't caught up.

Alan Facey
Chief Partnership Officer
3 min read
Two people reviewing code with an AI assistant's suggestion and a checklist

TL;DR: AI models generate code from patterns they were trained on, so they can reproduce vulnerabilities, weak cryptography and licensed code without saying where it came from. Organisations relying on AI assistance need visibility into how it shapes their code, and an AI bill of materials provides it.

There is a growing narrative that AI is becoming a new kind of developer: a silent teammate who writes code on command, accelerates delivery and fills skill gaps. The distinction worth sitting with is that AI emulates code. It predicts patterns based on what it has been trained on, blending fragments of the world's repositories into outputs that often look convincing, sometimes useful and occasionally problematic.

Generated code carries what the model learned from

That distinction is more than academic. When a model generates code from patterns it has absorbed, it may also reproduce the vulnerabilities, outdated practices or weak cryptography embedded within those patterns.

Beyond security, there is authorship. AI-generated code blurs the boundaries of ownership, licensing, accountability and intellectual property in ways that most organisations aren't yet prepared for.

The problem is the lost origin

AI makes mistakes, and so do people. What's different is that AI blurs the boundaries of origin. A model can't tell you whether the pattern it reproduced came from MIT-licensed code, GPL code, proprietary enterprise code or a personal side project on someone's laptop.

It can't explain intent. It can't distinguish an outdated cryptographic routine from a modern one. And it can't describe the obligations that may travel with its output.

Responsible use starts with visibility

If AI assistance is becoming a core part of how we build software, organisations need visibility into how that assistance is shaping the code that enters their supply chain. Innovation keeps its pace when transparency increases. It simply becomes safer to rely on at scale.

This is why I believe we are approaching a new layer of governance: an AI bill of materials (AIBOM). Its purpose is understanding, giving organisations a way to see how AI contributed to their systems. Just as SBOMs helped us catalogue human-produced components, an AIBOM can help track which models were used, what training sources shaped them, and what risks, legal, operational or cryptographic, might follow from their outputs.

Avoiding blind trust is the point

Mistrusting AI would miss the point. Blind trust is what needs avoiding. AI is powerful in the way any tool is powerful: it magnifies both strengths and assumptions. When we understand how it works and what it draws from, we can use it with confidence.

The companies that succeed in the next decade will combine AI-driven acceleration with governance that respects licensing, provenance, contributor identity and cryptographic integrity. Tooling for this layer already exists. Earnie finds the AI models and SDKs in a codebase and records them in an AIBOM. The harder work is organisational.

Clarity has to come with the transformation

AI will continue to transform how software is created. As leaders, our responsibility is to pair that transformation with clarity rather than with assumption.

If AI ever contributes to the next security breach, the cause won't be AI itself. It will be that we used it without the governance that keeps complex systems trustworthy. It's time to build that future openly.

RELATED READING

See it on your kind of codebase.

Request a demo
What if AI wrote the next security breach?
Earnie
ARTICLEAI PROVENANCE

What if AI wrote the next security breach?

AI assistance is becoming part of how software is built. The governance around it hasn't caught up.

Alan Facey
Chief Partnership Officer
3 min read
Two people reviewing code with an AI assistant's suggestion and a checklist

TL;DR: AI models generate code from patterns they were trained on, so they can reproduce vulnerabilities, weak cryptography and licensed code without saying where it came from. Organisations relying on AI assistance need visibility into how it shapes their code, and an AI bill of materials provides it.

There is a growing narrative that AI is becoming a new kind of developer: a silent teammate who writes code on command, accelerates delivery and fills skill gaps. The distinction worth sitting with is that AI emulates code. It predicts patterns based on what it has been trained on, blending fragments of the world's repositories into outputs that often look convincing, sometimes useful and occasionally problematic.

Generated code carries what the model learned from

That distinction is more than academic. When a model generates code from patterns it has absorbed, it may also reproduce the vulnerabilities, outdated practices or weak cryptography embedded within those patterns.

Beyond security, there is authorship. AI-generated code blurs the boundaries of ownership, licensing, accountability and intellectual property in ways that most organisations aren't yet prepared for.

The problem is the lost origin

AI makes mistakes, and so do people. What's different is that AI blurs the boundaries of origin. A model can't tell you whether the pattern it reproduced came from MIT-licensed code, GPL code, proprietary enterprise code or a personal side project on someone's laptop.

It can't explain intent. It can't distinguish an outdated cryptographic routine from a modern one. And it can't describe the obligations that may travel with its output.

Responsible use starts with visibility

If AI assistance is becoming a core part of how we build software, organisations need visibility into how that assistance is shaping the code that enters their supply chain. Innovation keeps its pace when transparency increases. It simply becomes safer to rely on at scale.

This is why I believe we are approaching a new layer of governance: an AI bill of materials (AIBOM). Its purpose is understanding, giving organisations a way to see how AI contributed to their systems. Just as SBOMs helped us catalogue human-produced components, an AIBOM can help track which models were used, what training sources shaped them, and what risks, legal, operational or cryptographic, might follow from their outputs.

Avoiding blind trust is the point

Mistrusting AI would miss the point. Blind trust is what needs avoiding. AI is powerful in the way any tool is powerful: it magnifies both strengths and assumptions. When we understand how it works and what it draws from, we can use it with confidence.

The companies that succeed in the next decade will combine AI-driven acceleration with governance that respects licensing, provenance, contributor identity and cryptographic integrity. Tooling for this layer already exists. Earnie finds the AI models and SDKs in a codebase and records them in an AIBOM. The harder work is organisational.

Clarity has to come with the transformation

AI will continue to transform how software is created. As leaders, our responsibility is to pair that transformation with clarity rather than with assumption.

If AI ever contributes to the next security breach, the cause won't be AI itself. It will be that we used it without the governance that keeps complex systems trustworthy. It's time to build that future openly.

RELATED READING
ARTICLE · AI PROVENANCE
Why EU AI Act compliance starts with an AIBOM
ARTICLE · AI PROVENANCE
A new standard for AI transparency: AIBOM

See it on your kind of codebase.

Request a demo