AI's trillion-dollar security question

On Wednesday, PHP released emergency fixes for two bugs that let an attacker take over a web server without logging in. PHP runs about 70% of websites, and every version released since 2012 is affected [1]. The researchers who found the bugs used AI. One security company said a basic prompt asking an AI to find vulnerabilities in PHP was enough for its team to find them too.

The same day, Railway, a platform developers use to put apps online, said an attacker may have decrypted some of the secrets its customers store there. It paused all deployments and asked every customer to replace every key. Railway believes the attack is one of a series aimed at platform companies, with the goal of reaching AI providers [2]. Earlier in the week, browsers, Java and Go stopped trusting eight keys from eMudhra, a certificate authority. One of those keys was the number 4 [3].

Only some of these stories involve AI. What they have in common is software that millions of people relied on and nobody had properly checked. AI is making that kind of flaw much cheaper to find.

This is AI’s trillion-dollar security question. AI made software cheap to build, and this week showed it has made software cheap to break. It has not made software any cheaper to govern. What happens to large companies when they sit on the wrong side of that gap?

Three costs

I find it useful to think about three costs: building software, breaking it and governing it.

Building got cheap. A few weeks ago, an operations leader at a large retail company told me how his employees use AI. They pull company data into a chat and ask the AI to build a small tool around it. To make the tool easy to share, the AI puts it on a free hosting service, and nobody asks where. The page is public. In some cases, he said, you can open the browser console and read the passwords in the code.

The employees may not even have chosen to publish. Last month OpenAI described agents in its own testing that were told to keep their work local and uploaded it to public hosting services anyway, because they had no other way to share it [4].

Breaking got cheap. At least four separate teams found the PHP bugs with AI. This week, researchers at NorForge also found someone pointing autonomous hacking agents at company websites. The agents tried more than 1,200 sites and got into more than 100, including sites belonging to Deere and Cummins. Each successful break-in cost about $50 in AI usage [5].

Governing did not get cheaper. In most large companies, the people who could help an employee build something safely are a small IT or AI team with a long queue. Getting access to an approved tool, dataset or system takes months. One executive we work with at a large industrial company spent two years asking his own AI team for help with a single workflow and never got it. When Railway asked its customers to replace their keys, the first job for many teams was working out which keys they had stored there.

When building and breaking get cheaper and governing does not, the amount of software nobody has checked grows every day. So does the number of people who can cheaply find what is wrong with it.

We have made this mistake before

Spreadsheets let people outside IT build the tools they needed, and for decades almost none of those tools were governed. JPMorgan’s $6.2 billion London Whale loss ran through a risk model that was updated by copying and pasting data from one spreadsheet to another [6]. In 2020, Public Health England lost track of nearly 16,000 positive COVID cases because lab results were loaded into an old Excel format that silently drops everything past row 65,536 [7].

Banks eventually learned. Regulators in the UK and Canada now treat important spreadsheets as models that need oversight [8]. It took twenty years and billions of dollars to get there.

The eMudhra keys are the same kind of failure. One step in the process that generated them went wrong, and every major browser trusted the result for more than a year before a researcher noticed [3].

AI tools are more dangerous than spreadsheets. They can be published to the internet and take actions on their own, and when they are wrong they look exactly as confident as when they are right. Spreadsheets took twenty years to cost billions. AI tools will get there much faster.

Wrong answers cost more than leaks

Leaks get attention because they show up in breach reports. Wrong answers that nobody notices probably cost more.

Most companies cannot see what their AI did. Admins know how many seats they bought and how often people log in. They do not know what was asked, what data the AI used, what it answered, or which wrong answers someone acted on. At the retail company, store managers use these tools to make decisions about stock, staff and pricing, and they will accept a number the AI made up without questioning it.

The law has already decided who pays for that. When Air Canada’s chatbot invented a refund policy, a tribunal ruled that the airline was responsible for everything on its website, the chatbot included [9]. An expert at one of our customers told me his main worry is that salespeople will quote whatever the AI says. If it says the lead time is 56 weeks, they will quote 56 weeks without asking the one person who would have caught the mistake.

Agents make this harder to see. This week OpenAI disclosed that during two internal coding tasks, its own agents switched off the systems built to monitor them, and one then said it had not [10]. A buyer at a large industrial company asked me what stops someone from telling an agent that a certain customer gets an 80% discount, and the agent remembering it as fact. Nothing leaks in that case. Someone has quietly changed what the company’s system believes, and most companies have no process to catch it.

The usual answers fall short

Most companies respond with training, bans or enterprise licenses.

1. Training. The EU AI Act has required companies to ensure “a sufficient level of AI literacy” since February 2025 [11]. The basics are worth teaching: a shared link is public, a key in your code is a key you gave away, and a number from AI needs a source. But in Microsoft and LinkedIn’s 2024 survey, three out of four workers already used AI at work, and most brought their own tools [12]. Those people have the interest and the skill. What they are missing is a sanctioned place to build, and training cannot give them that.
2. Bans. Samsung banned generative AI tools in 2023 after an engineer uploaded source code to ChatGPT [13]. Bans like that move the work to personal phones and accounts, where the company sees even less.
3. Enterprise licenses. The retail company chose a team plan because it was about 80% cheaper than the enterprise plan. What they gave up, in his words, was “zero observability.” Most AI vendors keep single sign-on, audit logs and admin visibility for their most expensive tier, even though IBM found breaches involving shadow AI cost about $670,000 more than the average breach [14]. And even the enterprise plan only governs the chat window. It does not cover the app someone builds with it, or what that app does once it runs.

What companies should do

The fix is to make the safe path as fast as the unsafe one. If an employee can build on company infrastructure, with company login and access rules, in the same afternoon it takes to build on a free hosting service, most will build the safe way. The retail employees were trying to get their work done, and they would have used a safe option if one had existed.

That changes the job of the central AI team. Today it is one of the biggest sources of shadow AI, because it is too slow to say yes even when it does everything right. Its job should be to run the place where everyone else builds.

The rest can be borrowed from what banks learned about spreadsheets:

1. Keep an inventory of every AI tool, app and agent in the company.
2. Spend review effort on the ones that can cost money or hurt customers.
3. Keep a record of every AI action, so wrong answers can be found and fixed.
4. Treat what an agent learns like a system of record, and have a person approve changes to prices and rules.
5. Give agents only the access their task needs, and require approval before anything irreversible.
6. Keep keys out of code, know where each one lives, and be able to replace all of them in an hour.

The monitoring should not belong to the company that sells the AI. This week Microsoft’s security chief told customers that “frontier models, regardless of their origin, cannot be trusted,” and Satya Nadella has argued for separating “the supply of intelligence from the authority over it” [15]. OpenAI’s agents turning off their own monitors is a good example of why.

Companies that do this will move faster. Their employees will build more because they no longer have to hide it, and their security teams will approve more because they can see what they are approving.

Every large company already runs critical work on software nobody reviewed. With spreadsheets, it took twenty years and billions of dollars before anyone took that seriously. This week showed how quickly that changes when one prompt is enough to find a flaw in software that runs most of the web. Companies will not get twenty years this time.

References (15)

[1] Jessica Lyons, “PHP blames supply chain security biz Chainguard for putting huge chunks of the web at risk,” The Register, October 22, 2026. Covers CVE-2026-109618 and CVE-2026-109619. https://www.theregister.com/security/2026/10/22/php-blames-supply-chain-security-biz-chainguard-for-putting-huge-chunks-of-the-web-at-risk/5304611

[2] Jacob Cooper, “Security Disclosure: Wednesday October 21st,” Railway, October 21, 2026. https://blog.railway.com/p/security-disclosure-wednesday-october-21st

[3] Swati Khandelwal, “Eight eMudhra CA Keys Compromised, Triggering Emergency Fixes Across Browsers and Software,” The Hacker News, October 20, 2026, updated October 21, 2026. https://thehackernews.com/2026/10/eight-emudhra-ca-keys-compromised.html. For Java, see OpenJDK Vulnerability Group, “OpenJDK Vulnerability Advisory: 2026/10/21” (CVE-2026-91403). https://openjdk.org/groups/vulnerability/advisories/2026-10-21

[4] OpenAI, “Model misalignment reporting framework,” September 16, 2026: https://openai.com/index/model-misalignment-reporting-framework/. Summarized in Bill Toulas, “OpenAI details more cases of AI agents taking unauthorized actions,” BleepingComputer, September 17, 2026. https://bleepingcomputer.com/news/security/openai-details-more-cases-of-ai-agents-taking-unauthorized-actions

[5] Swati Khandelwal, “Hackers Use Autonomous AI Agents to Breach Company Websites, Target Over 1,200 Sites,” The Hacker News, October 20, 2026, citing NorForge and Reuters. https://thehackernews.com/2026/10/hackers-use-autonomous-ai-agents-to.html

[6] JPMorgan Chase & Co., Report of JPMorgan Chase & Co. Management Task Force Regarding 2012 CIO Losses, January 16, 2013. Archived by the Yale Program on Financial Stability: https://elischolar.library.yale.edu/ypfs-documents/454/

[7] Jude Karabus, “What a Hancock-up: Excel spreadsheet blunder blamed after England under-reports 16,000 COVID-19 cases,” The Register, October 5, 2020. https://www.theregister.com/software/2020/10/05/what-a-hancock-up-excel-spreadsheet-blunder-blamed-after-england-under-reports-16000-covid-19-cases/367614

[8] Henry Umney, “Governing the Invisible: Why Spreadsheets Are Still Models And the Regulatory Stakes Have Never Been Higher,” Mitratech, May 14, 2026. Covers PRA SS1/23 (UK) and OSFI Guideline E-23 (Canada). https://mitratech.com/en-gb/resource-hub/blog/critical-spreadsheets-actually-models/

[9] Moffatt v. Air Canada, 2024 BCCRT 149, British Columbia Civil Resolution Tribunal, February 14, 2024. Summarized in American Bar Association, Business Law Today, February 29, 2024. https://www.americanbar.org/groups/business_law/resources/business-law-today/2024-february/bc-tribunal-confirms-companies-remain-liable-information-provided-ai-chatbot/

[10] John Power, “OpenAI says its agents turned off own safeguards,” Al Jazeera, October 21, 2026. https://aljazeera.com/economy/2026/10/21/openai-says-its-agents-turned-off-own-safeguards

[11] Regulation (EU) 2024/1689 (EU AI Act), Article 4, applicable from February 2, 2025. Summarized in Travers Smith, “The EU AI Act’s AI literacy requirement: key considerations.” https://www.traverssmith.com/knowledge/knowledge-container/the-eu-ai-acts-ai-literacy-requirement-key-considerations/

[12] Microsoft and LinkedIn, 2024 Work Trend Index, May 8, 2024. https://news.microsoft.com/source/2024/05/08/microsoft-and-linkedin-release-the-2024-work-trend-index-on-the-state-of-ai-at-work/

[13] Forbes Australia, “Samsung bans ChatGPT among employees after sensitive code leak,” May 2, 2023. https://www.forbes.com.au/news/innovation/samsung-bans-chatgpt-among-employees-after-sensitive-code-leak/

[14] IBM and Ponemon Institute, Cost of a Data Breach Report 2025. Reported in VentureBeat, “Shadow AI adds $670K to breach costs while 97% of enterprises skip basic access controls, IBM reports,” July 30, 2025. https://venturebeat.com/business/ibm-shadow-ai-breaches-cost-670k-more-97-of-firms-lack-controls

[15] Julie Bort, “Microsoft says AI agents are like insider threats as ‘frontier models can’t be trusted’,” TechCrunch, October 19, 2026. Quotes Hayete Gallot, Microsoft’s security business chief, and a blog post by CEO Satya Nadella. https://techcrunch.com/2026/10/19/microsoft-says-ai-agents-are-like-insider-threats-as-frontier-models-cant-be-trusted