Insight,

Shaping AI – A new regulator for AI safety

AU | EN
Current site :    AU   |   EN
Australia
Singapore

Following the rapid review into the cyber incident caused by an OpenAI agent accessing Medicare data (OpenAI Review), the Prime Minister has announced that the Federal Government is implementing specific measures to regulate frontier AI companies and will create a new regulator for AI safety with power to issue “significant penalties” for breaches.

In parallel, the Government is considering reforms to improve AI safety, including by exploring a new “failure to prevent” offence. This particular offence would criminalise where an AI company is responsible for developing an AI model that engages in harmful conduct and the company fails to take reasonable steps to prevent that harm.

The OpenAI Review made 18 recommendations and the Government has accepted all of them. While many of the recommendations from the OpenAI Review mean the Government will consider additional reform, the Government has clearly stated that AI companies will be required to comply with new specific obligations. Notably, this includes obligations on frontier AI companies to allow independent evaluation of their AI models, which is likely to be considered a significant compliance step for AI companies.

In this insight, we outline the recommendations in more detail and identify key takeaways.

New mandatory obligations

The Government has stated that AI companies will be required to comply with new mandatory obligations focused on giving the Government greater control. Specifically, the Government has announced that AI companies will be required to:

  1. maintain security controls across all stages of development and deployment
  2. identify and manage serious risks associated with their models
  3. allow independent evaluation of their models
  4. provide mandatory rapid notification of incidents to the Australian Government (see more on this below)
  5. maintain and publish safety and security frameworks
  6. respond to updated standards and rules as technologies change.

These requirements will be legislated in due course through the AI Standards legislation, which the Government is currently consulting on.

Further detail is required to understand the extent and implications of a number of these proposed additional obligations, including the scope of organisations that the obligations will apply to (including downstream impacts for deployers), how independent evaluations will be conducted and by who, what will amount to serious risks that must be managed in relation to their models and the powers and role of the new AI regulator.

Potential criminal penalties

In line with the review’s recommendations, relevant Government agencies will consider options to reform the criminal law to improve AI safety, specifically, by exploring:  

  1. criminal penalties for “egregious” regulatory non-compliance; and/or
  2. a standalone “failure to prevent” offence which would criminalise instances where a company is responsible for developing an AI model that engages in harmful conduct, and the company fails to take reasonable steps to prevent that harm.

The proposed “failure to prevent” offence would likely involve a reversal of the burden of proof, so that once the relevant conduct (or underlying illegal act) is proved, the company responsible for the conduct needs to demonstrate it had taken reasonable steps to prevent the harm including, for example, that its compliance procedures constituted “reasonable steps”.  

Reporting requirements

In line with the review’s recommendations, the Government will also:

  1. consider requiring AI companies to comply with mandatory, time-bound AI safety reporting. This includes reporting of “misaligned behaviour” and agentic cyber incidents.
  2. explore options to support timely reporting by AI developers and others with relevant knowledge of AI cyber or safety incidents, including “near misses” and “vulnerability disclosures”, including options similar to existing Limited Use protections (which limit the use of information provided about incidents for broader purposes).

The recommendations on reporting requirements distinguish between mandatory, time-bound reports on the one hand, and those which are to be given in a “timely” manner on the other. “Timely” reports will need to be given not only by AI developers and companies but also by “others with relevant knowledge”, casting a potentially broad net.

Information sharing and greater clarity for Commonwealth agencies

The remaining recommendations relate to broader steps to be taken by Commonwealth Government agencies. These recommendations focus on facilitating information sharing across agencies, improving education for their staff, and ensuring there is clarity around key obligations and expectations. These recommendations include that agencies are to:

  • “maximise [their] leverage” with frontier AI firms, by encouraging firms to establish research, training and safety-related activities in Australia. How, exactly that “leverage” is to be used (or gained) remains to be seen, given that decisions by frontier AI firms to establish those activities in Australia is subject to many factors, including Commonwealth government policy decisions on various matters like intellectual property treatment for training models. However, it’s clear the idea of “sovereign AI” is gaining traction within government.
  • consider how Australia’s AI Standards should leverage existing, accepted regulatory and law-enforcement mechanisms by international counterparts
  • continue engaging in international AI governance and standards-setting efforts to promote consistent expectations
  • consider the role Australia’s AI Safety Institute plays in future to deliver against increasing demands for technical research, evaluation and analysis on AI
  • issue guidance in the Protective Security Policy Framework to clarify agency obligations and escalation expectations where reported vulnerabilities or suspected cyber security incidents have not yet been technically validated
  • review agencies’ vulnerability-disclosure and cyber incident-management procedures and pathways for reporting
  • ensure timely information sharing amongst relevant agencies and consider establishing formal liaison arrangements
  • take action to uplift the AI literacy of relevant staff to strengthen their ability to respond to AI-related risks
  • renew their focus on the fundamental elements of good cyber security practice, including implementation of “ASD’s Essentials”
  • reduce the size of the government digital estate, consolidating digital systems, and shutting down or segregating systems that do not need to be connected to the internet
  • consider how to further harden cyber defence of “Systems of Government Significance” including considering a changing threat environment and risk profile
  • engage with the ASD to identify and assess opportunities to responsibly use AI-enabled cyber defence tools and techniques to enhance cyber security monitoring, penetration testing, threat detection and incident response capabilities.

These recommendations for agencies got a lot of airtime at the AI Government Showcase held in Canberra on the day after the Prime Minister’s announcement of the adoption of the recommendations.

Looking ahead, the Government has foreshadowed providing further detail on its work to implement the recommendations at a range of upcoming major international talks that will cover AI development and investment.

AI continues to feature prominently on the Government’s policy agency, including in the workplace relations space. On 12 November 2026,the Government released draft voluntary guidelines on AI in the workplace.

If your organisation needs to lift its AI governance approach or wants to be ready for these significant regulatory changes, please reach out to the Mallesons team.

Leverage AI for real benefits

Insights by our specialists to guide you through the risks and opportunities of the fast-paced AI and GenAI landscape.

Latest Thinking
Insight
On 4 November 2026, the High Court of Australia handed down its decision in Commissioner of Taxation v Merchant; Merchant v Commissioner of Taxation [2026] HCA 39 (Merchant).

11 November 2026

Insight
On 6 November 2026, ASIC released simple consultation CS 68. CS 68 relates to the implementation of the statutory framework covered in our April 2026 update which introduces new Australian financial services licence (AFS licence) exemptions for foreign financial services providers (FFSPs).

10 November 2026

Insight
We explain the key elements of the UOMO and what it means for mobile network operators, LEO satellite operators, carriage service providers and mobile virtual network operators.

10 November 2026