AI security will not be won with a bigger budget or a new dashboard. It will be won by disciplined engineering, clear governance and a culture that treats intelligent systems as powerful but fallible tools. The phrase ‘trust but verify’ is not a slogan for AI security; it is the operating model.
Michael Burch does not romanticise risk. Before he became Director of Application Security at Security Journey, he served as a US Army Green Beret medic. The habit of scanning a changing environment, anticipating failure modes, and building layered mitigations follows him from field medicine to code. He maps that habit to the current rush into intelligent systems with a blunt clarity that executives will recognise.
“Risk assessment and mitigation are daily work, not special moments,” Burch explains, reflecting on the continuity between elite operations and security engineering. “On a mission you never know exactly which threat will show up, but you design controls that hold up across scenarios. That is how I approach application security: assume attack paths will vary, build resilience in layers, and keep the objective in sight. The impacts are financial, they are reputational, and, in sectors like healthcare, they are very real for people’s lives.”
From battlefield risk to software resilience
Executives often ask for a list of specific AI threats, as if the ‘weapon’ could be named in advance. Burch argues for a different frame: design for resilience regardless of the exact technique. The analogy is practical rather than poetic. In the field, risk comes from many angles with no perfect forecast. In software, the same uncertainty applies. Attacks may be automated with sophisticated tooling or driven by a single, persistent human. Controls should degrade gracefully across both.
“We do not control the form of the next attack, so we do not anchor our defences to a single scenario,” Burch continues, drawing a straight line from operational discipline to engineering choices. “It might be an automated probe, it might be a person testing inputs manually. The answer is layered controls that make sense across situations. Validate what comes in. Separate sensitive actions from public interfaces. Log and monitor so you can respond. If you build that way, the specific weapon matters less.”
This is not a call to ignore model-specific risks. It is a reminder that most early failures occur where basic discipline is missing. The organisations that withstand pressure are the ones that embed security decisions in design rather than gatekeeping at the end of a release. The ones that struggle, by contrast, treat security as a quality check to be survived before launch.
The first breach is trust
Large language models are persuasive systems. They answer with fluency and a tone that feels collaborative. Burch regards unchecked trust as the most dangerous misconception at executive level. “It feels like a capable colleague is giving you confident advice,” Burch adds, describing the allure of conversational AI. “The model will confidently lie with the same tone it uses to tell the truth. I once spent twenty minutes discussing a security scanner it described in detail—outputs, use cases, everything. The tool did not exist. The lesson is not to fear the technology; it is to treat it as a powerful, probabilistic system. Trust, then verify. Always verify.”
The rush to instrument customer touchpoints with generative AI has a second-order effect that leadership teams underestimate: expanded attack surface for little incremental value. A basic chatbot that used to serve static menu answers becomes a dynamic agent with access to live catalogues, prices and stock systems. The customer experience may feel smoother. The security exposure is measurably larger.
“Productivity gains are real, and so is the price you pay if you wire AI into places that never needed it,” Burch explains, using a small retail example to make a broader point. “Integrating a model to answer simple questions can quadruple the threat surface, because you add prompt-handling risks and new data paths. That trade-off might not be worth it.”
Democratisation cuts both ways. Tools that help junior developers ship more capability also lift the floor for attackers. Burch’s description of the modern reconnaissance phase should give anyone comfort with legacy controls pause.
“We made sophisticated tasks easier for defenders and attackers,” Burch continues, outlining how social engineering scales. “A model can scrape public profiles at speed, generate personalised emails, and even support deepfake interactions that mimic executives on a video call. The barrier to running credible phishing and pretexting campaigns is lower. The answer is not panic; it is to raise our own baseline faster.”
Back to basics before agents
The conversation about AI security can sound exotic: adversarial prompts, jailbreaks, model collapse, and autonomous agents. Burch repeatedly drags the discussion back to fundamentals. Most organisations are not fine-tuning frontier models or building autonomous systems that act without human approval. Most are connecting pre-trained models to business data and exposing new interfaces to users. The failure modes look suspiciously familiar.
“The biggest threat surface has not changed; the impact has,” Burch explains. “A compelling talk on AI threats spent most of its time on traditional application security because that is still where break-ins start. You will still see missing input validation. You will still see sensitive functions without proper authentication. You will still see weak separation between public and privileged actions. Become an expert at the basics if you want to keep AI products safe.”
This is not a counsel of perfection. It is an engineering checklist executives can understand and measure: confidentiality, integrity and availability applied to model inputs and outputs; authentication on actions that change state; authorisation for data access; logging that allows incident response; and monitoring that catches drift and abuse. The hardest part is cultural rather than technical: insisting that a feature is not complete unless it is secure.
“Special operations are not special because of flashy tactics; they are special because they master fundamentals,” Burch continues, drawing the parallel without ceremony. “The equivalent in software is building security into normal work. If you cannot trace what a system did, if you do not validate inputs, and if you do not separate sensitive actions from public ones, you did not finish the job.”
Education is the lever most leaders underuse. Developers are rarely required to learn secure coding as part of their training. Code-generation tools amplify that gap when engineers move from writing code to reviewing code the model produces on their behalf.
“Secure coding is not elective if you expect resilience,” Burch explains, highlighting a structural issue. “Most developers are taught to implement features, not to defend them. When models write more of the code, reviewers need even more security literacy because they are validating rather than authoring. Without that, teams ship functions that work but do not withstand pressure.”
Examples of missing security are depressingly consistent, from unauthenticated sensitive endpoints to database calls built without sane input handling. Models are improving at producing safer boilerplate, but they remain probability engines that hallucinate detail and ignore context. The responsibility to verify does not move.
Culture, governance and recovery in the age of shadow AI
The neatest security policy will fail if it ignores how people work. Engineers and analysts use AI whether or not an organisation approves it. Burch argues that leadership teams must stop pretending to block usage and instead create controlled paths that balance productivity and risk.
“Shadow AI is already inside your organisation,” Burch explains, describing the pattern with a practical tone. “If you ban it, people will use unvetted tools anyway. The right move is to write a clear policy, select supported tools with your security team, and ask developers what genuinely helps them. Say ‘yes, and here are the guardrails’ rather than ‘no’. You will reduce risk and friction.”
Governance starts with visibility. An executive should be able to answer three questions at short notice: where AI is used in the business, what data it can touch and what actions it is allowed to perform. Without that, risk managers are blind, and product leaders cannot explain their own systems under pressure. Lightweight controls do better than heavy frameworks that nobody follows. Integrating scanning and policy checks into existing pipelines is more effective than expecting manual processes to run at the end of a sprint.
Burch handles the tension between ambition and discipline with a pragmatic lens. Security is not a universal veto; it is a set of deliberate choices about what to protect now and what to accept as risk. The problem is not that leaders accept risk. The problem is when they ignore it.
“Accepted risk is honest,” Burch adds. “Ignored risk is negligence. You will never implement every control. The job is to document what you will and will not do, why you chose that, and what the financial and operational implications are. That is very different from hoping for the best.”
Preparation for failure is part of resilience rather than a negative mindset. Incident response is not glamorous, and it is the difference between a contained event and a prolonged outage. “You should know exactly who gets called and what they do when something goes wrong,” Burch continues. “Write the activation list. Decide the thresholds. Run the drill. The act of planning improves design because it forces you to see how the system behaves under stress.”
The more adventurous frontiers, autonomous agents that can take actions without human oversight, require a stricter interpretation of ‘trust but verify’. Boards should treat capability boundaries, human approval steps and policy engines as non-negotiable. Legal ownership of outcomes will remain contested across jurisdictions. Process clarity is insurance against that uncertainty.
Building security thinking into everyday work is the long play. Burch recommends an offensive lens in code reviews and design, backed by practice that makes the mindset stick. The technique is unpretentious and effective: demonstrate how an application is exploited, then fix it, then re-run the attack to prove the fix. The point is not to turn every developer into a red teamer. The point is to make attack paths visible, so defensive code feels necessary rather than ornamental.
“Thinking like an attacker becomes a habit if you nudge it into the workflow,” Burch explains, pointing to small tactics that add up. “Add a review prompt that asks ‘how would this be attacked’. Let people see exploitation in a safe environment, then have them implement the fix and watch the attack fail. It changes how you build.”
Security conversations often drift toward existential questions of public trust. Burch’s answer is disarmingly practical. A single catastrophic breach may not permanently change user habits, but it will force regulators and boards to close the gap between headline ambition and operational reality. The faster route to meaningful change is simpler: show that building it right the first time is cheaper than rework and make the secure path the easy one.
Innovation pressure versus risk management is not a binary choice in this telling. The decisive factor is how easy the security team makes it for product teams to do the right thing. If controls are embedded in pipelines, if telemetry is visible where decisions are made, and if policy is written for the work people already do, adoption improves. If controls are slow and manual, they will be bypassed.
What boards should ask for next
A board cannot rewrite code or tune a model, but it can demand the conditions that keep intelligent systems honest. The checklist is short enough to hold in memory. Start with inventory and policy. Every business unit should declare where AI is used, what data it touches and what actions it can take. Provide approved tools and make it easy to request alternatives. Publish the guardrails and keep them current.
Insist on education that is accurate and light. People do not need a graduate seminar on model architecture. They do need to know that outputs are probabilistic, that confidence is simulated, and that verification is part of the job. When users stop ascribing authority to tone, the quality of decisions improves.
Make observability a requirement by capturing prompts and responses where law and policy permit, tracking hallucination rates, bias indicators and prompt-injection attempts, and routing these signals to the teams who can act on them rather than confining insight to a security operations centre.
Prioritise automation that removes friction by embedding scanners and policy checks into CI/CD pipelines rather than relying on manual checklists, triaging the first wave of alerts before they reach product teams, and recognising groups that eliminate whole classes of issues instead of merely burning down queues. Treat recovery as a design discipline by naming the incident lead and contributors, defining escalation paths, and rehearsing without theatrics; the practice itself will surface weaknesses and drive improvement.
Above all, treat ‘trust but verify’ as a habit, not a slogan. The most dangerous shift in the AI era is not a jailbreak or a novel prompt. It is the temptation to defer judgement to systems that speak with fluency. Executives should respect the leverage those systems provide without outsourcing responsibility for outcomes. That balance, clear ambition paired with disciplined verification, is what turns intelligent software into a reliable part of the enterprise rather than a headline risk.
Burch’s journey from field medic to application security leader underlines a consistent lesson. Resilience arises from mastering basics, rehearsing failure, and building controls that hold under changing pressure. Intelligent systems do not change that principle; they make it more urgent. The organisations that internalise it will move faster and safer than those that keep security as an end-of-line inspection. The ones that carry on trusting without verification will discover that probability is not a defence strategy.
