Agentic Systems Engineering
An agent is only as safe as the tools and permissions you give it, and only as useful as the test set you hold it to. Teams that reach production settle both before they tune prompts; teams that skip them end up with a demo nobody will sign off.
Part of AI & Data, in the Build mode. 10 sources, checked .
- Agentic AI projects Gartner predicts will be canceled by the end of 2027 (prediction published June 2025)
- 40%+
- Agentic AI projects Gartner predicts will be canceled by the end of 2027 (prediction published June 2025)
- Gartner, 2025(opens in a new tab)
- Respondents reporting scaled AI agent use, in nearly every business function (McKinsey 2025 survey, reported in the Stanford AI Index 2026)
- <10%
- Respondents reporting scaled AI agent use, in nearly every business function (McKinsey 2025 survey, reported in the Stanford AI Index 2026)
- Stanford Institute for Human-Centered Artificial Intelligence (HAI), 2026(opens in a new tab)
- Chief data officers who say they are early in defining how to scale and govern AI agents (IBM, 2025)
- 79%
- Chief data officers who say they are early in defining how to scale and govern AI agents (IBM, 2025)
- IBM Institute for Business Value, 2025(opens in a new tab)
The problem
Interest in agents is high and production use is thin. In McKinsey's 2025 survey, charted in the Stanford AI Index 2026, most respondents in most business functions reported no agent use at all, and scaled use was in the single digits for nearly all functions2. IBM found 79% of chief data officers early in defining how to scale and govern AI agents3. Gartner predicts over 40% of agentic AI projects will be canceled by the end of 2027, due to escalating costs, unclear business value or inadequate risk controls1.
Two engineering facts drive much of that. First, an agent that reads untrusted content can be steered by it. The UK National Cyber Security Centre notes that an LLM makes no distinction between data and instructions, and that prompt injection may never be totally mitigated the way SQL injection can4. OWASP's 2026 list ranks prompt injection first and excessive agency third, and says most high-impact incidents became severe because the model sat inside a system whose tools and scopes let it act for the attacker5. Second, agent behavior varies between runs, so results are harder to read than they look8.
Good practice is plain. Anthropic's advice to builders is to find the simplest solution and add complexity only when needed7, and Gartner notes that many use cases positioned as agentic do not need agentic implementations1. Tools get the minimum functions and permissions, actions run in the user's own security context, and high-impact actions need human approval5. Behavior is checked against test tasks drawn from real cases and real failures, before launch and after every change8.
What you get
- 01
Task and autonomy design
The simplest pattern that does the job, whether a single model call, a fixed workflow or an agent, with a written decision on which steps it may take alone and which need approval.
- 02
Tool and permission layer
Narrow tools with strict input schemas, a least-privilege identity per tool, actions executed in the requesting user's context, and authorization enforced in code outside the model.
- 03
Integration with your systems of record
Connections to CRM, ERP, ticketing and document stores through your existing APIs and identity provider, with every tool call logged and attributable.
- 04
Evaluation suite as a release gate
Test tasks built from real cases and known failures, graders that check outcomes, repeated trials to measure consistency, and regression runs on every prompt, model or tool change.
- 05
Production readiness pack
A threat model against the OWASP agentic risks, red-team results, monitoring, rate limits and circuit breakers, runbooks, and a tested way to roll back or switch the agent off.
Where it goes wrong
The failure modes we design against from the first week, and the control for each.
An agent where a workflow would do
Autonomy trades latency and cost for task performance and brings the risk of compounding errors7. Gartner says many use cases positioned as agentic do not need agentic implementations1. The control: start with the simplest pattern and add autonomy only where the evaluation results show it pays.
Broad standing access for the agent
OWASP traces excessive agency to excessive functionality, excessive permissions and excessive autonomy5. A read-only task given a tool that can also delete or send is an incident waiting for a trigger. The control: apply least agency, scope each tool, run actions as the user, and require approval for irreversible ones6.
Relying on filters to stop prompt injection
Filters reduce injection success but are expected to degrade against adaptive attackers5, and the NCSC advises reducing the risk and the impact rather than hoping for a complete fix4. The control: design so a fooled agent can do little harm. An agent that reads external content should not hold privileged tools, and every tool call should be logged and checked by deterministic rules.
Launching on a demo instead of a test set
A few good runs prove little when behavior varies between runs. Anthropic suggests 20 to 50 simple tasks drawn from real failures as a starting set, graded on what the agent produced8. The control: make the evaluation suite a release gate, run multiple trials, and read the transcripts, not just the scores.
Standards and rules
OWASP Top 10 for LLM Applications 2026
Primary text on genai.owasp.org(opens in a new tab)Replaced the 2025 list in August 2026 and is the baseline threat checklist, with prompt injection (LLM01) and excessive agency (LLM03) the entries most relevant to agents.
OWASP Top 10 for Agentic Applications 2026
Primary text on genai.owasp.org(opens in a new tab)Adds agent-specific risks such as goal hijack, tool misuse, identity and privilege abuse and cascading failures, and sets out the least-agency principle your design should follow.
EU AI Act, Article 50 (transparency)
Primary text on ai-act-service-desk.ec.europa.eu(opens in a new tab)AI systems that interact directly with people must be designed so those people are told they are dealing with an AI system, unless that is obvious, which covers customer-facing agents.
NIST AI Risk Management Framework (AI RMF 1.0)
Primary text on nist.gov(opens in a new tab)Its Measure function expects the test sets, metrics and tools used in testing and evaluation to be documented, which is the evidence your risk and audit teams will ask for.
How to measure it
- Task success rate on the evaluation set, across repeated trials
- Share of actions sent for human approval, and how often reviewers change or reject them
- Escalation rate to people and time to resolve escalated cases
- Cost and latency per completed task
- Out-of-policy tool calls blocked, and open findings from red-team testing
Questions buyers ask
What is the difference between an AI agent and a chatbot?
A chatbot answers. An agent acts: it decides which tools to call, such as searching a system, updating a record or sending a message, and uses the results to choose its next step. That makes agents more useful for multi-step work and more dangerous when they go wrong, so the engineering effort shifts from the conversation to the tools, permissions, approvals and tests around it.
How do you stop prompt injection in AI agents?
You cannot fully stop it today, so you limit what a successful attack can do. Keep untrusted content away from agents that hold privileged tools, give each tool the narrowest scope, run actions with the user's own permissions, enforce authorization outside the model, require approval for irreversible actions, and log every tool call. Filters help, but they are one layer, not the defense.
How do you test an AI agent before production?
Build a set of test tasks from real cases, including the ones that failed before, and define what a correct outcome looks like for each. Run each task several times, because agents vary between runs, and grade the result rather than the path taken. Add adversarial tests for injection and misuse. Rerun the whole set whenever the prompt, model or tools change, and read transcripts, not only scores.
Sources
Every figure on this page comes from the sources below. We checked each link on .
- 1.Gartner Predicts Over 40% of Agentic AI Projects Will Be Canceled by End of 2027(opens in a new tab)Gartner, June 25, 2025 · gartner.com
- 2.Artificial Intelligence Index Report 2026, Chapter 4: Economy(opens in a new tab)Stanford Institute for Human-Centered Artificial Intelligence (HAI), 2026 · hai.stanford.edu
- 3.IBM Study: Chief Data Officers Redefine Strategies as AI Ambitions Outpace Readiness(opens in a new tab)IBM Institute for Business Value, November 13, 2025 · newsroom.ibm.com
- 4.Prompt injection is not SQL injection (it may be worse)(opens in a new tab)UK National Cyber Security Centre, December 8, 2025 · ncsc.gov.uk
- 5.OWASP Top 10 for LLM Applications 2026(opens in a new tab)OWASP GenAI Security Project, August 3, 2026 · genai.owasp.org
- 6.OWASP Top 10 for Agentic Applications for 2026(opens in a new tab)OWASP GenAI Security Project, December 9, 2025 · genai.owasp.org
- 7.Building effective agents(opens in a new tab)Anthropic, December 19, 2024 · anthropic.com
- 8.Demystifying evals for AI agents(opens in a new tab)Anthropic, January 9, 2026 · anthropic.com
- 9.AI Act (Regulation (EU) 2024/1689), Article 50: Transparency obligations for providers and deployers of certain AI systems(opens in a new tab)European Commission, AI Act Service Desk, 2024 · ai-act-service-desk.ec.europa.eu
- 10.AI Risk Management Framework (AI RMF 1.0)(opens in a new tab)National Institute of Standards and Technology, January 26, 2023 · nist.gov