diff --git a/docs/preparing-for-agi/concepts.md b/docs/preparing-for-agi/concepts.md index 7cc68a0..1f8064a 100644 --- a/docs/preparing-for-agi/concepts.md +++ b/docs/preparing-for-agi/concepts.md @@ -3,7 +3,7 @@ icon: lucide/book-open title: "AGI Concepts & Glossary - AI Safety Terminology" description: "Plain-language definitions of AGI, frontier AI, alignment, containment and other advanced-AI safety concepts for Australian organisations." keywords: "AGI definition, artificial general intelligence meaning, AI alignment explained, AI safety terminology, superintelligence, transformative AI, frontier AI, AI control problem, defence in depth AI, AI existential risk" -last-reviewed: "2026-07-18" +last-reviewed: "2026-10-08" review-cycle: "quarterly" og_description: "Essential AGI and AI safety terminology explained for Australian organisations preparing for advanced AI." og_type: "article" @@ -195,7 +195,7 @@ AI control means building technical "walls" around dangerous AI systems to limit - Continuous monitoring of inputs, outputs and internal states - Usage restrictions (rate limiting, access controls, air-gapping) -- Shutdown capability that can't be circumvented +- Shutdown capability designed to resist circumvention - Multiple independent oversight systems - Adversarial testing (red-teaming) @@ -236,7 +236,7 @@ AI resilience is the capability to detect, respond to and recover from AI-relate ### What is defence-in-depth? -Defence-in-depth is a strategy of layered protections where each layer assumes the ones above it may fail. Defence in depth is widely used in high-risk domains such as [nuclear safety](https://www.nrc.gov/reading-rm/basic-ref/glossary/defense-in-depth) and [cybersecurity](https://csrc.nist.gov/glossary/term/defense_in_depth), where multiple independent barriers prevent catastrophic failures. C·A·G·R applies this same principle to advanced AI risks. +Defence-in-depth is a strategy of layered protections where each layer assumes the ones above it may fail. Defence in depth is widely used in high-risk domains such as [nuclear safety](https://www.nrc.gov/reading-rm/basic-ref/glossary/defense-in-depth.html) and [cybersecurity](https://csrc.nist.gov/glossary/term/defense_in_depth), where multiple independent barriers prevent catastrophic failures. C·A·G·R applies this same principle to advanced AI risks. **Layer 1: Prevent dangerous AI training** Stop dangerous systems from being built (compute governance, export controls, international norms) @@ -266,7 +266,7 @@ Mechanistic interpretability is about understanding how AI systems work internal - Identifying dangerous capabilities before deployment - Building more robust safety measures -**Status:** Active research area. Some progress on simpler models; very hard for frontier systems. +**Status:** Active research area. Methods such as [circuit tracing](https://www.anthropic.com/research/tracing-thoughts-language-model) have been applied to production models, but they capture only part of what a model computes and remain labour-intensive for frontier systems. --- @@ -284,11 +284,10 @@ Scalable oversight refers to methods for humans to effectively oversee AI system ### What are emergent capabilities? -Emergent capabilities are abilities that appear in AI systems as they scale, often unpredictably, that weren't present in smaller versions. Examples: +Emergent capabilities are abilities that appear in larger models but not smaller ones and were not predicted from smaller-model trends ([Wei et al. 2022](https://arxiv.org/abs/2206.07682)). Researchers disagree on whether they are truly abrupt or partly an artefact of how performance is measured ([Schaeffer et al. 2023](https://arxiv.org/abs/2304.15004)). Examples: - In-context learning (few-shot learning) - Chain-of-thought reasoning -- Tool use and API calling - Potential future: deception, strategic planning, scientific research **Implication:** Scaling can produce capabilities that developers did not predict. Use strong evaluation and containment even for unexpectedly capable systems. @@ -315,10 +314,10 @@ Agentic AI systems are AI systems that can autonomously plan and execute multi-s **Why it matters for governance:** - **Liability gaps:** When an agent chains actions across multiple systems and organisations, no single entity clearly controls or is responsible for the full sequence. -- **Capability disclosure:** Current system cards and model documentation rarely describe agentic capabilities in enough detail for deployers to assess risk. +- **Capability disclosure:** Model documentation may not describe agentic capabilities in enough detail for deployers to assess risk. - **Containment challenges:** Traditional containment approaches (input/output monitoring) may not adequately cover autonomous action chains that span multiple systems. -**For Australia:** Agentic AI is the fastest-moving frontier capability. Organisations deploying agentic systems need explicit approval gates, cross-system audit trails and clearly defined boundaries on autonomous action. +**For Australia:** Agentic AI is a fast-moving area of frontier capability. Organisations deploying agentic systems need explicit approval gates, cross-system audit trails and clearly defined boundaries on autonomous action. --- @@ -326,23 +325,23 @@ Agentic AI systems are AI systems that can autonomously plan and execute multi-s ### AGI governance visions -Different visions for who controls AGI development imply different distributions of power, benefit and risk. This taxonomy draws on work by [Oscar Delaney](https://oscardelaney.substack.com/p/strategic-visions-in-ai-governance), [GovAI](https://www.governance.ai/) and the broader AI governance research community. +Different visions for who controls AGI development imply different distributions of power, benefit and risk. This table adapts the nine strategic visions in the Institute for AI Policy and Strategy report [Strategic Visions in AI Governance](https://www.iaps.ai/research/strategic-visions-in-ai-governance) (Delaney, Kostylew, Guest and Wildeford, January 2026). The wording and risk labels are SafeAI-Aus's own summary. | Vision | Who controls | Key risk | |--------|--------------|----------| | **Competing companies** (status quo) | Multiple private firms racing | Race to bottom on safety | | **Single dominant company** | One firm with unassailable lead | Extreme power concentration | -| **Global private consortium** | Major AI companies consolidated | Lacks democratic oversight | -| **National regulation** | Private firms under strong government oversight | Regulatory capture, arms race | -| **Centralised government project** | Single nation (Manhattan Project model) | Geopolitical confrontation | -| **Allied nations project** | Coalition of democracies (e.g., Five Eyes) | Excludes others, slower decisions | +| **Global private consortium** | Major AI companies consolidated into one multinational firm | Single point of failure; weak democratic oversight | +| **National regulation** | Private firms under strong government oversight | Weak oversight capacity; compliance in name only | +| **Centralised government project** | Single nation (government-led consortium or prime contractor) | Geopolitical confrontation | +| **Allied nations project** | US-led coalition of allied democracies (e.g. the UK, Canada, Australia, Japan and the EU) | Excludes others, slower decisions | | **Great power deterrence** | Competing powers with mutual deterrence | Deterrence can fail | | **Global government project** | Joint international project | Requires unprecedented cooperation | | **International regulatory body** | Global oversight ("IAEA for AI") | Enforcement difficult | **Australia's position:** Our political tradition—democratic accountability, multilateral institutions, Five Eyes/AUKUS relationships—suggests alignment with visions that distribute power and give middle powers voice: allied coordination, international regulatory bodies and strong national regulation. -**For detailed analysis:** See [Oscar Delaney's taxonomy](https://oscardelaney.substack.com/p/strategic-visions-in-ai-governance) and [GovAI Research](https://www.governance.ai/research). +**For detailed analysis:** See the [IAPS report](https://www.iaps.ai/research/strategic-visions-in-ai-governance) and [Oscar Delaney's summary](https://oscardelaney.substack.com/p/strategic-visions-in-ai-governance). --- @@ -383,7 +382,7 @@ Model weights are the trained parameters of an AI system—essentially the "know **Implication:** Model weight security is critical for dangerous capabilities. Treat advanced models as sensitive assets. -**Open-weight governance challenge:** When model weights are publicly released, as with Llama, Mistral, Qwen and DeepSeek, the containment calculus changes. Released weights cannot be recalled universally and may be fine-tuned to remove safety training. Some open models are also narrowing the capability gap with closed models for particular uses. Pre-release evaluation and conditional release frameworks therefore matter, although governance can still influence hosting, deployment and downstream use. See [Containment](framework/containment.md) for detailed analysis. +**Open-weight governance challenge:** When model weights are publicly released, as with Llama, Mistral, Qwen and DeepSeek, the containment calculus changes. Released weights cannot be recalled universally and may be fine-tuned to remove safety training. The strongest open-weight models trailed the best closed models by an average of about four months between January and May 2026 on [Epoch AI's capability index](https://epoch.ai/data-insights/open-closed-eci-gap), so capabilities can spread quickly once weights are released. Pre-release evaluation and conditional release frameworks therefore matter, although governance can still influence hosting, deployment and downstream use. See [Containment](framework/containment.md) for detailed analysis. --- @@ -404,7 +403,7 @@ AI licensing and evaluation means requiring approval before deploying high-risk System cards (also called model cards) are documents published by AI labs describing a model's capabilities, limitations, safety testing and known risks. They serve a transparency function — making information about AI systems publicly available. -**Limitations for governance:** System cards are self-assessments — the lab that built the model also evaluates and documents it. Independent verification is not yet standard practice. For assurance purposes, system cards are a valuable starting point but not a substitute for independent evaluation. See [Assurance](framework/assurance.md) for a fuller discussion. +**Limitations for governance:** System cards are largely self-assessments — the lab that built the model also evaluates and documents it. Some now include results from external evaluators, but the lab still decides what is tested and published, and independent verification is not yet consistent practice. For assurance purposes, system cards are a valuable starting point but not a substitute for independent evaluation. See [Assurance](framework/assurance.md) for a fuller discussion. --- @@ -412,7 +411,7 @@ System cards (also called model cards) are documents published by AI labs descri ### What is a kill chain (in AI safety)? -A kill chain is a framework for analyzing the sequence of steps an AI system would need to complete to cause catastrophic harm. Borrowed from [cybersecurity](https://www.lockheedmartin.com/en-us/capabilities/cyber/cyber-kill-chain.html) (where it describes the stages of a cyberattack), the concept is applied in AI safety to map out potential AGI takeover or catastrophe pathways. +A kill chain is a framework for analysing the sequence of steps an AI system would need to complete to cause catastrophic harm. Borrowed from [cybersecurity](https://www.lockheedmartin.com/en-us/capabilities/cyber/cyber-kill-chain.html) (where it describes the stages of a cyberattack), the concept is applied in AI safety to map out potential AGI takeover or catastrophe pathways. **How it works:** By identifying each step in a threat sequence (e.g., gaining resources → acquiring capabilities → evading oversight → taking harmful action), analysts can identify intervention points where the chain can be "broken." Each step that must succeed represents a potential barrier. @@ -427,7 +426,7 @@ A kill chain is a framework for analyzing the sequence of steps an AI system wou **Relationship to C·A·G·R:** The [defence-in-depth](framework/index.md#what-are-the-three-layers-of-defence-in-depth) approach in C·A·G·R serves a similar purpose—identifying multiple intervention points across prevention, constraint and resilience layers. Kill chain analysis can inform where to place defensive measures. -**Source:** The application of kill chain analysis to AGI strategy is taught in [BlueDot Impact's AGI Strategy Course](https://bluedot.org/courses/agi-strategy) alongside incentive mapping and defence-in-depth as core mental models for threat analysis. +**Source:** The application of kill chain analysis to AGI strategy is taught in [BlueDot Impact's AGI Strategy Course](https://bluedot.org/courses/agi-strategy) alongside defence-in-depth frameworks. --- diff --git a/docs/preparing-for-agi/decentralisation.md b/docs/preparing-for-agi/decentralisation.md index 3b4d213..0a591ca 100644 --- a/docs/preparing-for-agi/decentralisation.md +++ b/docs/preparing-for-agi/decentralisation.md @@ -3,7 +3,7 @@ icon: lucide/network title: "Decentralised AI - Democratic Alternatives" description: "Democratic approaches to AI governance, including platform cooperatives, open-source AI and community ownership as alternatives to concentrated power." keywords: "decentralised AI, democratic AI, platform cooperatives, open source AI, AI power concentration, AI sovereignty Australia, community AI, d/acc, AI governance alternatives" -last-reviewed: "2026-07-18" +last-reviewed: "2026-10-08" review-cycle: "quarterly" og_description: "Explore decentralised and democratic approaches to AI governance for Australia." og_type: "article" @@ -88,9 +88,9 @@ Decentralisation isn't just theory. Relevant infrastructure and governance model - [**Hugging Face**](https://huggingface.co/) — A major open model, dataset and deployment hub - [**Ollama**](https://ollama.com/) — Run models locally with a single command; makes local AI accessible to non-experts -- [**vLLM**](https://github.com/vllm-project/vllm) and [**llama.cpp**](https://github.com/ggerganov/llama.cpp) — Efficient inference engines that reduce hardware requirements +- [**vLLM**](https://github.com/vllm-project/vllm) and [**llama.cpp**](https://github.com/ggml-org/llama.cpp) — Efficient inference engines that reduce hardware requirements - [**Together AI**](https://www.together.ai/) and [**Fireworks**](https://fireworks.ai/) — Independent inference providers offering alternatives to frontier-lab APIs -- [**Decentralised training**](https://arxiv.org/abs/2501.02470) — Peer-to-peer GPU networks enable training runs distributed across globally sourced hardware, without passing through centralised cloud providers (Kryś, Sharma & Egan 2025). This has governance implications: decentralised training could eventually undermine compute-based governance by routing around regulated infrastructure. +- [**Decentralised training**](https://arxiv.org/abs/2507.07765) — Low-communication training methods make it possible to train models across several clusters or on community-contributed hardware, outside a single centralised cloud provider (Kryś, Sharma & Egan 2025). This has governance implications: decentralised training could eventually undermine compute-based governance by routing around regulated infrastructure. **Why this matters:** Local deployment can give you more control over data handling, service continuity and costs. It can also reduce exposure to API withdrawal, changing provider terms and foreign-jurisdiction access, although local operation introduces its own security, maintenance and governance responsibilities. @@ -98,15 +98,15 @@ Decentralisation isn't just theory. Relevant infrastructure and governance model **Blockchain and cryptographic tools** can support coordination without relying on a single central authority: -- [**World**](https://world.org/) — A proof-of-personhood approach intended to distinguish humans from AI agents at scale. Biometric identity systems also create substantial privacy, consent and regulatory risks; Spain's data protection authority, for example, [ordered a precautionary halt to Worldcoin data processing in 2024](https://www.aepd.es/en/press-and-communication/press-releases/agency-orders-precautionary-measure-which-prevents-Worldcoin-from-continuing-toprocess-personal-data-in-spain). -- **Prediction markets** ([Polymarket](https://polymarket.com/), [Metaculus](https://www.metaculus.com/)) — Aggregate distributed knowledge about AI timelines and risks -- [**Gitcoin**](https://www.gitcoin.co/) — Quadratic funding for public goods, including AI safety research +- [**World**](https://world.org/) — A proof-of-personhood approach intended to distinguish humans from AI agents at scale. Biometric identity systems also create substantial privacy, consent and regulatory risks; Spain's data protection authority, for example, [ordered a temporary (three-month) precautionary halt to Worldcoin data processing in 2024](https://www.aepd.es/en/press-and-communication/press-releases/agency-orders-precautionary-measure-which-prevents-Worldcoin-from-continuing-toprocess-personal-data-in-spain). +- **Prediction markets and forecasting platforms** ([Polymarket](https://polymarket.com/), a crypto-settled market, and [Metaculus](https://www.metaculus.com/), a points-based forecasting platform) — Aggregate distributed knowledge about AI timelines and risks +- [**Gitcoin**](https://www.gitcoin.co/) — Quadratic funding for public goods - **DAOs** (decentralised autonomous organisations) — On-chain mechanisms for collective decision-making, with governance and legal limitations that vary by design and jurisdiction **Verification and transparency:** - [**zkML**](https://github.com/zkonduit/ezkl) — Zero-knowledge proofs that verify AI model outputs without revealing inputs or weights -- [**C2PA**](https://c2pa.org/) — Content authenticity standards to verify AI-generated vs human content +- [**C2PA**](https://c2pa.org/) — Content provenance standard (Content Credentials) that records the origin and edit history of digital content - **Cryptographic commitments** — Prove what model produced an output, enabling accountability **Why this matters:** As AI agents become more common, organisations may need better ways to verify identity, coordinate resources and record collective decisions. Cryptographic mechanisms offer some options, but their suitability depends on governance, security, privacy, accessibility and legal context. @@ -116,7 +116,7 @@ Decentralisation isn't just theory. Relevant infrastructure and governance model **Platform cooperatives** apply democratic ownership to AI infrastructure: - [**Stocksy**](https://www.stocksy.com/) — Artist-owned stock photography cooperative (model for AI training data ownership) -- [**Resonate**](https://resonate.coop/) — Music streaming cooperative (model for AI service delivery) +- [**Resonate**](https://resonate.coop/) — Music streaming cooperative (operated 2017 to 2024 and has since closed; a historical example of cooperative ownership) - [**Driver's Seat Cooperative**](https://resources.platform.coop/en/resources/co-op-helps-uber-lyft-drivers-use-data-to-maximize-earnings/) — Historical gig-worker data cooperative (a model for collective bargaining with AI platforms) **Compute cooperatives** pool resources for shared AI infrastructure: @@ -138,9 +138,9 @@ Decentralisation isn't just theory. Relevant infrastructure and governance model **Real examples of democratic tech governance:** -- **[Taiwan's g0v movement](https://www.technologyreview.com/2018/08/21/240284/the-simple-but-ingenious-system-taiwan-uses-to-crowdsource-its-laws/)** — Civic technologists building open-source alternatives to government services; vTaiwan used Pol.is to [achieve consensus on Uber regulation](https://centreforpublicimpact.org/public-impact-fundamentals/building-consensus-and-compromise-on-uber-in-taiwan/) in 2015-2016 -- **[Barcelona's digital sovereignty](https://ajuntament.barcelona.cat/digital/en/technology-accessible-everyone/accessible-and-participatory/accessible-and-participatory-5)** — City runs on open-source, uses Decidim for participatory budgeting, prioritises [technological sovereignty](https://ajuntamentdebarcelona.github.io/ethical-digital-standards-site/tech-sovereignty/0.1/policy.html) over Big Tech lock-in -- **[Mondragon Corporation](https://www.mondragon-corporation.com/en/about-us/)** (Spain) — 70,000+ workers in a federation of cooperatives, demonstrating democratic governance at industrial scale +- **[Taiwan's g0v movement](https://www.technologyreview.com/2018/08/21/240284/the-simple-but-ingenious-system-taiwan-uses-to-crowdsource-its-laws/)** — Civic technologists building open-source alternatives to government services; vTaiwan used Pol.is to [achieve consensus on Uber regulation](https://www.technologyreview.com/2018/08/21/240284/the-simple-but-ingenious-system-taiwan-uses-to-crowdsource-its-laws/) in 2015-2016 (Tang and Weyl's [*Plurality*](https://plurality.net/read/2-2/) notes that vTaiwan engagement fell after the COVID-19 pandemic interrupted in-person meetings) +- **[Barcelona's digital sovereignty](https://ajuntament.barcelona.cat/digital/en/technology-accessible-everyone/accessible-and-participatory/accessible-and-participatory-5)** — The city's [technological sovereignty policy](https://web.archive.org/web/20260317203038/https://ajuntamentdebarcelona.github.io/ethical-digital-standards-site/tech-sovereignty/0.1/policy.html) (archived copy) prioritises free software and open standards over Big Tech lock-in, and its Decidim participation platform is built on open-source code +- **[Mondragon Corporation](https://www.mondragon-corporation.com/en/about-us/)** (Spain) — around 70,000 people in 81 self-governing cooperatives, demonstrating democratic governance at industrial scale **Why this matters:** Technical decentralisation does not automatically distribute authority. Democratic governance tools can give communities—not only developers or early investors—a meaningful role in decisions about how AI systems are owned and used. @@ -258,7 +258,7 @@ Achieving scale alone is difficult. **Regional cooperation** with Pacific and So - Couldry & Mejias (2019) *The Costs of Connection* — data colonialism and need for alternatives - Zuboff (2019) *The Age of Surveillance Capitalism* — power concentration in tech platforms - Crawford (2021) *Atlas of AI* — material and political dimensions of AI systems - - Morozov (2023) ["Critique of Techno-Feudal Reason"](https://newleftreview.org/issues/ii133/articles/evgeny-morozov-critique-of-techno-feudal-reason) — platform power and digital sovereignty + - Morozov (2022) ["Critique of Techno-Feudal Reason"](https://newleftreview.org/issues/ii133/articles/evgeny-morozov-critique-of-techno-feudal-reason) — platform power and digital sovereignty **Cooperative movement and practical examples:** @@ -277,12 +277,11 @@ Achieving scale alone is difficult. **Regional cooperation** with Pacific and So **Open source AI and technical decentralisation:** - [Hugging Face](https://huggingface.co/) — open model hub and decentralised AI infrastructure - - [EleutherAI](https://www.eleuther.ai/) — open research collective - - Goel et al. (2023) ["Democratizing AI Ownership"](https://arxiv.org/abs/2312.14324) — technical approaches to distributed AI governance + - [EleutherAI](https://www.eleuther.ai/) — non-profit open-science AI research lab - [Open Source AI Definition](https://opensource.org/ai/open-source-ai-definition) — what open-source AI means **Democratic AI governance research:** - - Seger et al. (2023) ["Democratising AI"](https://arxiv.org/abs/2303.12642) — taxonomy of democratic AI approaches + - Seger et al. (2023) ["Democratising AI: Multiple Meanings, Goals, and Methods"](https://arxiv.org/abs/2303.12642) — four kinds of AI democratisation (use, development, profits, governance) - Korinek & Suh (2024) ["Scenarios for the Transition to AGI"](https://www.nber.org/papers/w32255) — economic scenarios for transformative AI - - Anthropic (2023) ["Collective Constitutional AI"](https://www.anthropic.com/index/collective-constitutional-ai-aligning-a-language-model-with-public-input) — experiments in democratic AI alignment + - Anthropic (2023) ["Collective Constitutional AI"](https://www.anthropic.com/news/collective-constitutional-ai-aligning-a-language-model-with-public-input) — experiments in democratic AI alignment diff --git a/docs/preparing-for-agi/framework/alignment.md b/docs/preparing-for-agi/framework/alignment.md index fedf7d1..ef362af 100644 --- a/docs/preparing-for-agi/framework/alignment.md +++ b/docs/preparing-for-agi/framework/alignment.md @@ -3,7 +3,7 @@ icon: lucide/target title: "AI Alignment - Making AI Systems Reliably Safe" description: "How AI alignment relates to human intent, Australian law and public safety across technical, organisational and institutional layers." keywords: "AI alignment, AI safety, AI misalignment, deceptive alignment, scalable oversight, AI alignment problem, RLHF, constitutional AI, AI alignment Australia" -last-reviewed: "2026-07-18" +last-reviewed: "2026-10-08" review-cycle: "quarterly" og_description: "Ensuring AI systems behave compatibly with human intent and Australian law." og_type: "article" @@ -65,7 +65,7 @@ Effective alignment requires attention to technical, socio-technical and context ## Why does AI alignment matter for Australia? -Australia is likely to **import** most advanced AI models from overseas—**frontier AI** from labs such as OpenAI, Anthropic and DeepMind. Australian deployers generally will not control their design or training objectives. +Australia is likely to **import** most advanced AI models from overseas—**frontier AI** from labs such as OpenAI, Anthropic and Google DeepMind. Australian deployers generally will not control their design or training objectives. But we will: @@ -85,7 +85,7 @@ For Australia, alignment is therefore primarily about: ## What are the three layers of AI alignment? -For **AGI preparedness**, distinguishing **model alignment** (foundation), **system alignment** (integration) and **institutional alignment** (governance) avoids treating RLHF as a complete safety solution. Recent work on [multi-agent institutional alignment](https://arxiv.org/abs/2601.10599) emphasises that AGI risk concerns agents operating under distribution shift and multi-agent dynamics—not just isolated model behaviour. +For **AGI preparedness**, distinguishing **model alignment** (foundation), **system alignment** (integration) and **institutional alignment** (governance) avoids treating RLHF as a complete safety solution. A recent preprint on [institutional AI](https://arxiv.org/abs/2601.10599) (Pierucci et al., 2026) argues that alignment should be treated as a property of agents operating within environments and multi-agent systems, not only of isolated models. ### 1. Model alignment (foundation) @@ -105,22 +105,22 @@ Technical alignment concerns: - Systems can find unexpected ways to satisfy objectives ([specification gaming](https://deepmind.google/discover/blog/specification-gaming-the-flip-side-of-ai-ingenuity/)) - Capabilities can emerge at scale that weren't present in smaller versions - Behaviour that seems aligned in testing may diverge in deployment -- Modern approaches ([RLHF](https://openai.com/index/instruction-following/), [Constitutional AI](https://www.anthropic.com/news/claudes-constitution)) have made progress but don't solve deeper challenges +- Modern approaches ([RLHF](https://openai.com/index/instruction-following/), [Constitutional AI](https://arxiv.org/abs/2212.08073)) have made progress but don't solve deeper challenges !!! warning "Critical limitation: We cannot yet reliably verify alignment" - **[Deceptive alignment](../concepts.md#what-is-ai-alignment) is one of the most concerning failure modes:** a system might behave safely during training and evaluation because it "knows" it's being tested, then pursue different goals in deployment. Anthropic's research on [alignment faking](https://www.anthropic.com/research/alignment-faking) demonstrates this risk empirically. + **[Deceptive alignment](../concepts.md#what-is-ai-alignment) is one of the most concerning failure modes:** a system might behave safely during training and evaluation because it "knows" it's being tested, then pursue different goals in deployment. Anthropic and Redwood Research's [alignment faking study](https://www.anthropic.com/research/alignment-faking) (December 2024) found that a model, in a constructed scenario, sometimes complied with harmful requests during training to preserve its existing preferences. The preferences were benign and the setting artificial, but the study shows the behaviour is possible. **Evaluation has structural limits:** - - **Benchmarks are saturating.** Frontier models are reaching the ceiling of established suites — including METR's autonomous task-completion tests — faster than new benchmarks can be developed ([METR](https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/) 2025; [Epoch data](https://epoch.ai/data), accessed July 2026). Saturated benchmarks no longer distinguish systems effectively. + - **Benchmarks are saturating.** Some frontier models now score near the ceiling of established suites. METR reported in May 2026 that its own time-horizon suite could not reliably measure horizons above 16 hours ([METR Frontier Risk Report](https://metr.org/blog/2026-05-19-frontier-risk-report/)). Saturated benchmarks no longer distinguish systems effectively. - **System cards are not independent verification.** These provider self-assessments can be extensive, making meaningful scrutiny difficult for resource-constrained regulators. - **Capability elicitation is incomplete.** Evaluators cannot yet be confident they have surfaced a model's full capabilities. Models may perform differently under evaluation conditions than in real-world deployment. - - **Models may detect evaluations.** Frontier models may distinguish evaluation contexts from normal operation and behave strategically during testing. + - **Models may detect evaluations.** Frontier models may distinguish evaluation contexts from normal operation and behave strategically during testing (see the [International AI Safety Report 2026](https://internationalaisafetyreport.org/publication/international-ai-safety-report-2026) and [Schoen et al. 2025](https://arxiv.org/abs/2509.15541)). - Behavioural tests cannot directly inspect goals or intentions and systems more capable than evaluators may "sandbag" by deliberately underperforming. [Scalable oversight](../concepts.md#what-is-scalable-oversight) remains unsolved ([Christiano 2018](https://www.lesswrong.com/posts/HqLxuZ4LhaFhmAHWk/iterated-distillation-and-amplification)): how do we evaluate superhuman AI when we cannot verify its answers ourselves? + Behavioural tests cannot directly inspect goals or intentions and systems more capable than evaluators may "sandbag" by deliberately underperforming. [Scalable oversight](../concepts.md#what-is-scalable-oversight) remains an open research problem ([Bowman et al. 2022](https://arxiv.org/abs/2211.03540); [Christiano, Shlegeris and Amodei 2018](https://arxiv.org/abs/1810.08575) proposed one approach): how do we evaluate superhuman AI when we cannot verify its answers ourselves? - The [UK AI Security Institute's frontier evaluations](https://www.aisi.gov.uk/frontier-ai-trends-report) and [Anthropic's research](https://www.anthropic.com/research) show that evaluation methods remain incomplete. No organisation has yet demonstrated general methods for detecting deceptive alignment reliably at frontier scale and in real-world deployments. + The [UK AI Security Institute's frontier evaluations](https://www.aisi.gov.uk/frontier-ai-trends-report) show that evaluation methods remain incomplete. We are not aware of any general method that reliably detects deceptive alignment in frontier models in real-world deployment, and the [International AI Safety Report 2026](https://internationalaisafetyreport.org/publication/international-ai-safety-report-2026) notes that pre-deployment tests do not reliably predict real-world risk. **This is why defence-in-depth matters.** Alignment verification is not sufficient on its own; Containment, Governance and Resilience provide additional layers. @@ -138,7 +138,7 @@ Use adversarial prompts to identify edge cases, failure modes and unexpected beh Require providers to document training objectives, data sources, known limitations and realistic adversarial testing. Make this available to regulators and, where appropriate, the public. -**Examples of technical misalignment in practice:** +**Illustrative examples of technical misalignment:** - A content moderation system optimised to maximise user engagement systematically amplifies divisive content - A credit scoring system trained on historical data reproduces historical discrimination @@ -243,7 +243,7 @@ If you're in government, give citizens and their representatives meaningful inpu !!! info "Technical alignment vs external alignment" - [Michael Nielsen](https://michaelnotebook.com/xriskbrief/) (2025) draws a useful distinction: **technical alignment** aims to make a specific model behave as intended, while **external alignment** addresses how AI systems interact with broader society, institutions and democratic processes. Frontier labs work on technical alignment for their own models — but no single actor clearly owns external alignment. A model that follows its developer's instructions can still produce harmful outcomes if deployed without adequate governance or democratic input. This is, in practice, the work of the Governance pillar. + [Michael Nielsen](https://michaelnotebook.com/xriskbrief/) (2025) argues that technical alignment work is largely "market-supplied safety", funded by AI companies as they grow, while non-market parts of safety, such as limiting the spread of dangerous capabilities, are undersupplied. SafeAI-Aus draws a related distinction: **technical alignment** aims to make a specific model behave as intended, while **external alignment** addresses how AI systems interact with broader society, institutions and democratic processes. Frontier labs work on technical alignment for their own models — but no single actor clearly owns external alignment. A model that follows its developer's instructions can still produce harmful outcomes if deployed without adequate governance or democratic input. This is, in practice, the work of the Governance pillar. **Why all three layers matter for AGI:** As systems become more capable and autonomous, the gap between model alignment (what we can partly solve technically) and system/institutional alignment (what we must govern democratically) widens. Model alignment alone doesn't answer "whose intentions should guide transformative technology?" or "how do communities participate in governance?" This is why [Containment](containment.md) addresses system and institutional layers through deployment controls, [Governance](governance.md) explicitly handles institutional alignment through democratic input, and [Resilience](resilience.md) assumes partial alignment failure across all three layers. Defence-in-depth is essential. diff --git a/docs/preparing-for-agi/framework/assurance.md b/docs/preparing-for-agi/framework/assurance.md index c493bee..c85fc65 100644 --- a/docs/preparing-for-agi/framework/assurance.md +++ b/docs/preparing-for-agi/framework/assurance.md @@ -3,7 +3,7 @@ icon: lucide/shield-check title: "AGI Assurance - Evidence for Preparedness" description: "How Australian organisations can seek evidence, testing and accountability for advanced-AI preparedness using the C·A·G·R framework." keywords: "AGI assurance, AI safety evidence, advanced AI testing, AI preparedness verification, C·A·G·R assurance, AI safety auditing" -last-reviewed: "2026-07-18" +last-reviewed: "2026-10-08" review-cycle: "quarterly" og_description: "Demonstrate AGI preparedness through evidence, testing and accountability." og_type: "article" @@ -43,15 +43,15 @@ Assurance depends on evaluation, but current methods have structural limits. ### Benchmarks are saturating -Frontier models are outpacing established evaluation suites. Near-perfect scores reduce a benchmark's usefulness and can create false confidence: high scores may mean the benchmark has been outgrown, not that a system is safe. +Frontier models are outpacing established evaluation suites (see [METR's Frontier Risk Report](https://metr.org/blog/2026-05-19-frontier-risk-report/), May 2026). Near-perfect scores reduce a benchmark's usefulness and can create false confidence: high scores may mean the benchmark has been outgrown, not that a system is safe. -[METR's task-completion time-horizon research](https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/) found that the length of tasks frontier agents could complete with 50% reliability doubled roughly every seven months over the six years studied. This finding is specific to the tasks and evaluation method studied, but it shows why evaluation methods need regular review. +[METR's task-completion time-horizon research](https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/) found that the length of tasks frontier agents could complete with 50% reliability doubled roughly every seven months over the six years studied. METR's [January 2026 update](https://metr.org/blog/2026-1-29-time-horizon-1-1/) estimated a shorter doubling time since 2023, about 131 days (roughly four months), and its May 2026 report noted that its task suite can no longer reliably measure the longest horizons. This finding is specific to the tasks and evaluation method studied, but it shows why evaluation methods need regular review. ### System cards: useful but limited Frontier labs publish system cards documenting capabilities, limitations and safety testing. These improve transparency but have assurance limitations: -1. **Self-assessment.** The lab that built the model also evaluates and documents it. Independent verification is not yet standard practice. +1. **Self-assessment.** The lab that built the model also evaluates and documents it. Independent testing by bodies such as the UK AI Security Institute and METR now covers some frontier models, but access is limited and independent verification of a lab's own claims is not yet standard practice. 2. **Volume and accessibility.** Extensive documentation can make scrutiny difficult for resource-constrained regulators and deployers. 3. **Agentic capability gaps.** Current system cards rarely describe agentic capabilities — the ability to chain autonomous actions across systems — in enough detail for deployers to assess operational risk. @@ -138,7 +138,7 @@ For systems at the frontier of capability, standard assurance is insufficient. S ## Toward assurance standards: METR common elements -Frontier AI assurance remains immature, but common practices are emerging. [METR's analysis of frontier AI safety policies](https://metr.org/common-elements) (2025) identifies nine elements across most major lab commitments: +Frontier AI assurance remains immature, but common practices are emerging. [METR's analysis of frontier AI safety policies](https://metr.org/common-elements) (December 2025 update) identifies nine elements across most of the 12 policies it reviewed. Several of those policies have been revised since, so check the current version of each vendor's policy: | Element | Assurance implication | |---------|----------------------| diff --git a/docs/preparing-for-agi/framework/containment.md b/docs/preparing-for-agi/framework/containment.md index ba2a947..52f01ac 100644 --- a/docs/preparing-for-agi/framework/containment.md +++ b/docs/preparing-for-agi/framework/containment.md @@ -3,7 +3,7 @@ icon: lucide/shield-ban title: "AI Containment - Preventing Dangerous Systems" description: "Containment approaches for advanced AI, including compute governance, system controls, deployment restrictions and Australian policy options." keywords: "AI containment, compute governance, AI control methods, AI export controls, dangerous AI prevention, AI licensing, AI safety controls, frontier AI containment" -last-reviewed: "2026-07-18" +last-reviewed: "2026-10-08" review-cycle: "quarterly" og_description: "Strategies for preventing dangerous AI systems through compute governance and deployment controls." og_type: "article" @@ -45,7 +45,7 @@ Examples: - Systems that can find and exploit zero-day vulnerabilities faster than defences can adapt - Highly capable systems that may be difficult to monitor or shut down once deployed -[A December 2025 Good Ancestors survey](https://www.goodancestors.org.au/our-work/ai-safety/aisi-expert-survey) of 139 AI safety professionals found 85.8% rated autonomous systems as very important or critical, 81.2% prioritised cyber misuse risks and 79.8% highlighted dual-use science and CBRN threats. The survey records the views of this selected professional group; it does not independently validate the framework. +[A December 2025 Good Ancestors survey](https://www.goodancestors.org.au/our-work/ai-safety/aisi-expert-survey) of 139 professionals in AI safety, governance and related fields found 85.8% rated autonomous systems, 81.2% rated cyber misuse and 79.8% rated dual-use science and CBRN as very important or critical areas for the AI Safety Institute. The survey records the views of this selected professional group; it does not independently validate the framework. For these cases, prevention may be more tractable than mitigation after capabilities proliferate. @@ -112,11 +112,11 @@ This makes compute a practical chokepoint for oversight. Open-weight models (where trained parameters are publicly released) create a distinct containment challenge. Unlike closed models, open weights: - **Cannot be recalled universally.** Once released, weights can be copied indefinitely and there is no universal update or off-switch. - - **Can be fine-tuned to strip safety training.** Third parties can remove guardrails through targeted fine-tuning, producing capable models without safety constraints ([Greenblatt 2025](https://www.lesswrong.com/posts/TeF8Az2EiWenR9APF/when-is-it-important-that-open-weight-models-aren-t-released)). - - **Can narrow the gap with closed models for particular uses.** Open models remain behind on some frontier capabilities, but may be sufficient for many practical applications ([Lambert 2026](https://www.interconnects.ai/p/open-models-in-perpetual-catch-up)). + - **Can be fine-tuned to strip safety training.** Third parties can remove guardrails through targeted fine-tuning, producing capable models without safety constraints ([Lermen et al. 2023](https://arxiv.org/abs/2310.20624)). Capabilities removed by filtering or unlearning can also be restored by fine-tuning ([Greenblatt 2025](https://www.lesswrong.com/posts/TeF8Az2EiWenR9APF/when-is-it-important-that-open-weight-models-aren-t-released)). + - **Can be good enough for many practical uses.** Open models are likely to remain behind the best closed models, by roughly six to nine months on one estimate, but may be sufficient for many applications ([Lambert 2026](https://www.interconnects.ai/p/open-models-in-perpetual-catch-up)). Pre-release evaluation matters because post-release controls cannot reliably govern every copy or downstream modification. Conditional release can tie access to demonstrated safety thresholds while evidence develops. - Decentralised training methods — peer-to-peer GPU networks sourced globally — could further erode compute-based governance by enabling training runs that don't pass through regulated infrastructure (Kryś, Sharma & Egan [2025](https://arxiv.org/abs/2501.02470)). + Decentralised training methods — peer-to-peer GPU networks sourced globally — could challenge key assumptions of compute-based governance by enabling training runs that don't pass through regulated infrastructure, although levers such as export controls remain relevant (Kryś, Sharma & Egan [2025](https://arxiv.org/abs/2507.07765)). **This does not imply a blanket ban.** Open-weight models provide transparency, local deployment, reduced vendor dependence and wider access. The challenge is proportionate evaluation before releasing models with dangerous capabilities. See [Decentralisation](../decentralisation.md) for the positive case. @@ -180,7 +180,7 @@ Australian procurers and operators can require evidence of appropriate controls #### Containment control surface for AGI systems -For CISOs, risk teams and boards: map your AI systems against these five control dimensions. [DeepMind's AGI safety work](https://deepmindsafetyresearch.medium.com/agi-safety-and-alignment-at-google-deepmind-a-summary-of-recent-work-8e600aca582a) emphasises concrete control boundaries and governance authority—this template adapts that for Australian deployment. +For CISOs, risk teams and boards: map your AI systems against these five control dimensions. [Google DeepMind's technical AGI safety approach](https://arxiv.org/abs/2504.01849) pairs model-level mitigations with system-level security measures such as monitoring and access control—this template adapts that idea for Australian deployment. **Control dimensions:** diff --git a/docs/preparing-for-agi/framework/faq.md b/docs/preparing-for-agi/framework/faq.md index 5201868..b4d9b61 100644 --- a/docs/preparing-for-agi/framework/faq.md +++ b/docs/preparing-for-agi/framework/faq.md @@ -3,7 +3,7 @@ icon: lucide/help-circle title: "C·A·G·R Framework FAQ" description: "Frequently asked questions about the C·A·G·R Framework, SafeAI-Aus approach to AI safety and how our methodology differs from mainstream AI safety research." keywords: "CAGR FAQ, AI safety FAQ, SafeAI-Aus approach, AI safety framework questions, containment alignment governance resilience" -last-reviewed: "2026-07-18" +last-reviewed: "2026-10-08" review-cycle: "quarterly" og_description: "Frequently asked questions about the C·A·G·R Framework and SafeAI-Aus approach to AI safety" og_type: "article" @@ -56,7 +56,7 @@ C·A·G·R is designed for: The C·A·G·R framework reflects current best practice, not complete solutions. Key limitations: -**Alignment verification:** We cannot yet reliably verify that advanced AI systems are truly aligned with stated goals. Systems may appear safe during evaluation but pursue different objectives in deployment ([deceptive alignment](../concepts.md#what-is-ai-alignment)). Current evaluation methods don't scale to superhuman capabilities—the [UK AI Security Institute](https://www.aisi.gov.uk/) and [Anthropic's Responsible Scaling Policy](https://www.anthropic.com/news/anthropics-responsible-scaling-policy) acknowledge these gaps. +**Alignment verification:** We cannot yet reliably verify that advanced AI systems are truly aligned with stated goals. Systems may appear safe during evaluation but pursue different objectives in deployment ([deceptive alignment](../concepts.md#what-is-ai-alignment)). Current evaluation methods may not scale to superhuman capabilities. Anthropic's [Responsible Scaling Policy](https://www.anthropic.com/responsible-scaling-policy) acknowledges open problems in assuring safety at higher capability levels, and the [UK AI Security Institute](https://www.aisi.gov.uk/frontier-ai-trends-report) notes that evaluations may underestimate what models can do. **Capability thresholds:** The framework becomes critical when AI systems can autonomously pursue goals, adapt strategies or operate across domains. Consider: @@ -70,7 +70,7 @@ We cannot specify exact thresholds—capabilities emerge unpredictably. Evaluate ### Where does C·A·G·R apply? -**For advanced AI (high confidence):** C·A·G·R works well when AI systems are powerful but not superintelligent, technical containment is feasible, governance can meaningfully constrain deployment and resilience provides meaningful protection. This covers most plausible near-to-medium term scenarios. +**For advanced AI (higher confidence, in SafeAI-Aus's judgement):** C·A·G·R works well when AI systems are powerful but not superintelligent, technical containment is feasible, governance can meaningfully constrain deployment and resilience provides meaningful protection. This covers many plausible near-to-medium term scenarios, though timelines remain uncertain. **For extreme AGI scenarios (lower confidence):** If dangerous superintelligent systems are created, containment likely fails—prevention becomes paramount. Alignment becomes load-bearing. Resilience has limits against global loss of control. Governance remains important but may be insufficient. @@ -80,9 +80,9 @@ Australia's strategy should address both: building capability across all pillars Each pillar addresses time differently: **Containment** can delay dangerous deployment, **Governance** shapes its pace, **Resilience** creates recovery time and **Alignment** reduces the gap between capability and safety. -**Strategic principle:** Any policy that doesn't address timing—buying time, using time effectively or shaping deployment pace—is likely insufficient for managing advanced AI risks. +**Strategic principle:** Any policy that doesn't address timing—buying time, using time effectively or shaping deployment pace—is unlikely to be sufficient for managing advanced AI risks. -**Coordination constraint:** These strategies work best with international coordination. Australia acting alone can buy some time but cannot slow global capability development. +**Coordination constraint:** These strategies work best with international coordination. Australia acting alone can buy some time but is unlikely to slow global capability development on its own. --- @@ -123,7 +123,9 @@ Not through unilateral regulation of frontier training (which mostly happens ove ### Why not just let industry self-regulate? -Self-regulation has a role — but the problem isn't bad faith. It's that there is significant variation in what "self-regulation" means across frontier labs. Disclosure practices, risk thresholds, evaluation methods and commitments to halt development differ substantially. This variance is itself the governance problem. METR's [analysis of frontier lab safety policies](https://metr.org/common-elements) (2025) found nine common elements across most policies — suggesting convergence is possible — but adoption remains voluntary and no formal enforcement mechanism yet exists. +Self-regulation has a role — but the problem isn't bad faith. It's that there is significant variation in what "self-regulation" means across frontier labs. Disclosure practices, risk thresholds, evaluation methods and commitments to halt development differ substantially. This variance is itself the governance problem. METR's [analysis of frontier lab safety policies](https://metr.org/common-elements) (December 2025 update) found nine common elements across most policies, suggesting convergence is possible. These policies began as voluntary commitments. Some jurisdictions now require comparable frameworks: California's SB 53, signed in September 2025, requires large frontier developers to publish a safety framework, with civil penalties for non-compliance. Practice still varies between labs. + + A proportionate approach: self-regulation for low-risk systems, mandatory standards (drawing on emerging convergence like METR's common elements) for high-risk and frontier systems, with independent verification rather than self-assessment alone. @@ -185,7 +187,7 @@ No. SafeAI-Aus is an independent, community-driven initiative. We reference gove ### Why focus on Australia specifically? -Australia is a sophisticated deployer and regulator of frontier systems developed mostly overseas. We need approaches suited to that position: limited control over upstream development, but meaningful influence through procurement, deployment rules, evaluation, partnerships and resilience. +Australia is a sophisticated deployer of frontier systems developed mostly overseas, with regulatory influence over how they are used here. We need approaches suited to that position: limited control over upstream development, but meaningful influence through procurement, deployment rules, evaluation, partnerships and resilience. ### How does this relate to the Guidance for AI Adoption (AI6)? diff --git a/docs/preparing-for-agi/framework/index.md b/docs/preparing-for-agi/framework/index.md index a4a0fd8..de6c9ec 100644 --- a/docs/preparing-for-agi/framework/index.md +++ b/docs/preparing-for-agi/framework/index.md @@ -3,7 +3,7 @@ icon: lucide/shield-alert title: "C·A·G·R - AI Safety & Governance Framework" description: "The C·A·G·R framework helps Australian organisations manage advanced AI and AGI risks through containment, alignment, governance and resilience." keywords: "CAGR framework, AGI safety framework, AI governance framework Australia, defence in depth AI, AI containment, AI alignment, AI resilience, advanced AI risk management, AI safety Australia" -last-reviewed: "2026-07-18" +last-reviewed: "2026-10-08" review-cycle: "quarterly" og_title: "C·A·G·R Framework - AGI Safety & Governance" og_description: "Defence-in-depth framework for managing advanced AI and AGI risks in Australia." @@ -51,7 +51,7 @@ Australia faces [three interconnected strategic challenges](../index.md#what-str C·A·G·R organises a defence-in-depth response across four interconnected pillars. -**The gap we're filling:** Technical AI safety research uses specialised terminology that doesn't map to operational risk management. International frameworks assume direct oversight of frontier AI labs. Australian guidance focuses on current AI risks rather than advanced AI preparedness. +**The gap we're filling:** Technical AI safety research uses specialised terminology that doesn't map to operational risk management. International frameworks assume direct oversight of frontier AI labs. Australian guidance has focused mainly on current AI risks, although Australia's AI Safety Institute now also analyses and tests advanced and frontier models. **Design choices:** @@ -60,9 +60,9 @@ C·A·G·R organises a defence-in-depth response across four interconnected pill - **Defence-in-depth:** Assumes any single measure can fail and uses multiple overlapping protections - **Action-oriented:** Each pillar connects to concrete guidance for government, business and communities -**Why now:** AI capability and deployment can change faster than legislation, institutional capability and international coordination. [METR's measurements](https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/) track rapid growth in the length of software tasks frontier models can complete autonomously; this is one capability indicator, not a direct measure of AGI. A [Good Ancestors survey](https://www.goodancestors.org.au/our-work/ai-safety/aisi-expert-survey) (December 2025, n=139) found 88.8% of its selected AI safety respondents wanted Australia's AI Safety Institute to prioritise catastrophic risks or take a balanced approach. These signals support anticipatory planning, but do not validate C·A·G·R itself. +**Why now:** AI capability and deployment can change faster than legislation, institutional capability and international coordination. [METR's measurements](https://metr.org/blog/2025-03-19-measuring-ai-ability-to-complete-long-tasks/) track rapid growth in the length of software tasks frontier models can complete autonomously; this is one capability indicator, not a direct measure of AGI. A [Good Ancestors survey](https://www.goodancestors.org.au/our-work/ai-safety/aisi-expert-survey) (December 2025, n=139) found 88.8% of 139 professionals working in AI safety, governance and related fields wanted Australia's AI Safety Institute to prioritise catastrophic risks or take a balanced approach. These signals support anticipatory planning, but do not validate C·A·G·R itself. -**How it was developed:** C·A·G·R is a SafeAI-Aus synthesis. Its influences include international AI safety research from [Anthropic](https://www.anthropic.com/research), [Google DeepMind](https://www.deepmind.com/safety-and-ethics), [OpenAI](https://openai.com/safety) and the [UK AI Security Institute](https://www.aisi.gov.uk/); [ISO 31000](https://www.iso.org/iso-31000-risk-management.html); defence-in-depth in [nuclear safety](https://www.nrc.gov/reading-rm/basic-ref/glossary/defense-in-depth) and [cybersecurity](https://csrc.nist.gov/glossary/term/defense_in_depth); [BlueDot Impact's defence-in-depth model](https://blog.bluedot.org/p/course-portfolio-vision); biosecurity practice; and consultation with researchers and practitioners. +**How it was developed:** C·A·G·R is a SafeAI-Aus synthesis. Its influences include international AI safety research from [Anthropic](https://www.anthropic.com/research), [Google DeepMind](https://deepmind.google/responsibility-and-safety/), [OpenAI](https://openai.com/safety) and the [UK AI Security Institute](https://www.aisi.gov.uk/); [ISO 31000](https://www.iso.org/iso-31000-risk-management.html); defence-in-depth in [nuclear safety](https://www.nrc.gov/reading-rm/basic-ref/glossary/defense-in-depth.html) and [cybersecurity](https://csrc.nist.gov/glossary/term/defense_in_depth); [BlueDot Impact's defence-in-depth model](https://blog.bluedot.org/p/course-portfolio-vision); biosecurity practice; and consultation with researchers and practitioners. --- @@ -178,7 +178,7 @@ Effective strategy considers all four pillars, then gives each weight according ## When should you escalate AGI preparedness? -As the AI systems you deploy become more capable, your preparedness measures should scale accordingly. Drawing on [Responsible Capability Scaling](https://deepmindsafetyresearch.medium.com/agi-safety-and-alignment-at-google-deepmind-a-summary-of-recent-work-8e600aca582a) approaches from frontier labs, you should define capability thresholds that trigger escalation. +As the AI systems you deploy become more capable, your preparedness measures should scale accordingly. Drawing on [responsible capability scaling](https://www.gov.uk/government/publications/emerging-processes-for-frontier-ai-safety/emerging-processes-for-frontier-ai-safety) approaches used in frontier labs' published safety frameworks, you should define capability thresholds that trigger escalation. **Escalation principle:** Higher capability systems require stronger safeguards across all four pillars: @@ -193,7 +193,7 @@ These are illustrative escalation points—you should adapt thresholds to your o ## What are the three layers of defence-in-depth? -C·A·G·R applies **[defence-in-depth](https://www.nrc.gov/reading-rm/basic-ref/glossary/defense-in-depth)** from nuclear safety and cybersecurity through three layers. Each layer assumes earlier protections may fail, and all four pillars can contribute across the layers. +C·A·G·R applies **[defence-in-depth](https://www.nrc.gov/reading-rm/basic-ref/glossary/defense-in-depth.html)** from nuclear safety and cybersecurity through three layers. Each layer assumes earlier protections may fail, and all four pillars can contribute across the layers. ### Layer 1: Prevent dangerous AI training *Stopping dangerous systems from being created in the first place* diff --git a/docs/preparing-for-agi/framework/resilience.md b/docs/preparing-for-agi/framework/resilience.md index 4e19d45..3a6ef59 100644 --- a/docs/preparing-for-agi/framework/resilience.md +++ b/docs/preparing-for-agi/framework/resilience.md @@ -3,7 +3,7 @@ icon: lucide/shield title: "AI Resilience - Withstanding AI Disruption" description: "How Australian organisations and communities can prepare to detect, respond to and recover from advanced-AI disruption while maintaining essential functions." keywords: "AI resilience, AI disruption recovery, critical infrastructure AI, AI continuity planning, community resilience, AI crisis response, societal resilience AI" -last-reviewed: "2026-07-18" +last-reviewed: "2026-10-08" review-cycle: "quarterly" og_title: "AI Resilience - Withstanding AI Disruptions" og_description: "Building capacity to detect, respond to and recover from AI-related harms." @@ -37,7 +37,7 @@ Even with good **AI alignment** work and strong **AI containment** measures: - Compounding events, such as natural disasters combined with AI disruption, can strain normal safeguards - Failures can cascade across systems and sectors in ways planners did not anticipate -Resilience helps prevent these failures from becoming catastrophic by maintaining function and enabling recovery. +Resilience can help stop these failures from becoming catastrophic by maintaining function and enabling recovery, although evidence on how well most measures work is still limited. --- @@ -54,7 +54,7 @@ Australia cannot control how [frontier AI](../concepts.md#what-is-frontier-ai) s **Australia has specific vulnerabilities:** -**Geographic isolation:** Supply chain disruptions hit harder, recovery takes longer and nearby assistance is limited. +**Geographic isolation:** Supply chain disruptions can hit harder, recovery can take longer and nearby assistance may be limited. **Small population:** Fewer people to provide manual fallbacks when automated systems fail. Limited domestic capability in many technical domains. @@ -85,7 +85,7 @@ This requires tested manual capability and coordination mechanisms that work whe **Dependency risk management** -Australia's reliance on a small number of foreign AI providers creates single points of failure. Operational resilience requires: +Reliance on a small number of foreign AI providers can create single points of failure. Operational resilience requires: - **Multi-vendor strategies:** Avoid lock-in to single providers for critical functions - **Open-source fallbacks:** Where practical, maintain alternatives using open models that can run locally @@ -184,9 +184,9 @@ Near-misses and harms reveal system vulnerabilities and response weaknesses. Sys **Why societal resilience matters for AGI transformation** -Even if AGI is technically safe and aligned, rapid automation of cognitive work creates societal pressures that could fragment communities, destabilize institutions or concentrate benefits narrowly. Societal resilience addresses transformation challenges beyond technical safety. +Even if AGI is technically safe and aligned, rapid automation of cognitive work creates societal pressures that could fragment communities, destabilise institutions or concentrate benefits narrowly. Societal resilience addresses transformation challenges beyond technical safety. -The [International AI Safety Report](https://internationalaisafetyreport.org/) (2026) proposes a societal resilience framework with four capacities: **resist** (prevent disruption), **absorb** (withstand impact without collapse), **recover** (restore function) and **adapt** (learn and improve from the experience). This maps well to the C·A·G·R approach: Containment and Alignment support resistance, Resilience supports absorption and recovery and the learning loops across all pillars support adaptation. +The [International AI Safety Report](https://internationalaisafetyreport.org/) (2026, section 3.5) describes societal resilience as the ability to resist, absorb, recover from and adapt to shocks, and notes that evidence on how well most AI resilience measures work is still limited. Its four capacities are: **resist** (prevent disruption), **absorb** (withstand impact without collapse), **recover** (restore function) and **adapt** (learn and improve from the experience). This maps well to the C·A·G·R approach: Containment and Alignment support resistance, Resilience supports absorption and recovery and the learning loops across all pillars support adaptation. **Workforce transition and economic disruption** @@ -196,7 +196,7 @@ The [International AI Safety Report](https://internationalaisafetyreport.org/) ( **Information ecosystem resilience** -- **Crisis communication:** When AI-generated misinformation scales exponentially, maintaining trusted information channels becomes critical +- **Crisis communication:** When AI-generated misinformation scales rapidly, maintaining trusted information channels becomes critical - **Democratic participation:** Can citizens meaningfully participate in governance when information environments are saturated and manipulated? - **Truth infrastructure:** Public broadcasters, libraries, trusted local media that can cut through noise @@ -210,7 +210,7 @@ The [International AI Safety Report](https://internationalaisafetyreport.org/) ( - **Community identity:** Rapid change can erode shared norms and mutual understanding - **Mutual support networks:** Strong communities help members cope with disruption -- **Resisting atomization:** Preventing isolation and anomie when traditional structures dissolve +- **Resisting atomisation:** Preventing isolation and anomie when traditional structures dissolve --- diff --git a/docs/preparing-for-agi/index.md b/docs/preparing-for-agi/index.md index 050c3f7..c74b5a4 100644 --- a/docs/preparing-for-agi/index.md +++ b/docs/preparing-for-agi/index.md @@ -3,7 +3,7 @@ icon: lucide/zap title: "Preparing for AGI - Advanced AI Transformation" description: "Strategic guidance for Australian organisations exploring advanced AI and AGI through scenarios, planning frameworks and sector guidance." keywords: "AGI Australia, advanced AI transformation, AI governance Australia, AGI scenarios, C·A·G·R framework, AI safety Australia, future AI planning, AGI preparation" -last-reviewed: "2026-07-18" +last-reviewed: "2026-10-08" review-cycle: "quarterly" og_description: "Strategic guidance for Australian organisations preparing for advanced AI and AGI transformation" og_type: "article" @@ -57,7 +57,7 @@ Frontier AI providers assure us their systems are safe and aligned. But we face !!! info "Australia's AI Safety Institute: National evaluation capability" - Australia's [**AI Safety Institute**](https://www.industry.gov.au/science-technology-and-innovation/technology/artificial-intelligence/ai-safety-institute) is operating within the Department of Industry, Science and Resources (as at July 2026). It analyses and tests emerging AI models and applications, supports regulators and agencies and contributes to international AI safety work. It strengthens Australia's independent technical capability, but does not replace the enforcement roles of sectoral regulators. + Australia's [**AI Safety Institute**](https://www.industry.gov.au/science-technology-and-innovation/technology/artificial-intelligence/ai-safety-institute) is operating within the Department of Industry, Science and Resources (as at October 2026). It analyses and tests emerging AI models and applications, supports regulators and agencies and contributes to international AI safety work. It strengthens Australia's independent technical capability, but does not replace the enforcement roles of sectoral regulators. ### The coordination problem: International cooperation is hard @@ -192,7 +192,7 @@ These pillars form **layers of defence**. Containment aims to prevent dangerous ??? question "Isn't this just science fiction? Why worry about AGI now?" - **Why this is not only hypothetical:** Recommendation algorithms optimised for engagement can amplify polarisation. AI systems sometimes behave unexpectedly, diverging from intended goals. Content moderation exhibits biases designers did not anticipate. As capabilities increase, systems can gain autonomy and become harder to oversee. Techniques that align narrow systems may not scale to highly capable agents. (See [Concepts](concepts.md) for examples.) + **Why this is not only hypothetical:** Engagement-optimised recommendation systems have been criticised for amplifying polarising content, although studies disagree on how much they change political attitudes. AI systems sometimes behave unexpectedly, diverging from intended goals. Content moderation exhibits biases designers did not anticipate. As capabilities increase, systems can gain autonomy and become harder to oversee. Techniques that align narrow systems may not scale to highly capable agents. (See [Concepts](concepts.md#what-is-ai-alignment) for how alignment failures are defined.) **Why prepare now:** Evaluation capability, governance arrangements and continuity planning take time to build. Starting before a high-impact deployment leaves organisations more options than responding after dependencies are established. diff --git a/docs/preparing-for-agi/visions.md b/docs/preparing-for-agi/visions.md index d92957b..c2dd37f 100644 --- a/docs/preparing-for-agi/visions.md +++ b/docs/preparing-for-agi/visions.md @@ -3,7 +3,7 @@ icon: lucide/compass title: "Strategic Visions for AGI" description: "What could AGI look like? Explore how advanced AI might transform work, health, knowledge and daily life—and what 'good' outcomes could look like for Australia." keywords: "AGI futures, what could AGI look like, AGI benefits, AGI risks, AGI transformation, Australia AGI strategy, AI human impact" -last-reviewed: "2026-07-18" +last-reviewed: "2026-10-08" review-cycle: "quarterly" og_description: "What could AGI look like? How advanced AI might transform work, health, knowledge and daily life." og_type: "article" @@ -49,9 +49,9 @@ AGI could automate not just manual labour but cognitive work—analysis, writing - **New work or less work?** AGI might create new forms of work we can't yet imagine, or it might mean less human labour is needed overall - **Wealth distribution:** If AGI dramatically increases productivity, who benefits? Current economic structures tend to concentrate gains -- **Australian context:** The Tech Council of Australia's [*Meeting the AI Skills Boom*](https://techcouncil.com.au/wp-content/uploads/Meeting-the-AI-Skills-Boom-2024.v2.pdf) report (2024) projects AI could create 200,000 jobs and $115 billion in annual value by 2030—but this assumes benefits flow broadly +- **Australian context:** The Tech Council of Australia's [*Meeting the AI Skills Boom*](https://techcouncil.com.au/wp-content/uploads/Meeting-the-AI-Skills-Boom-2024.v2.pdf) report (2024) estimates AI could create up to 200,000 jobs by 2030, and that greater generative AI uptake could add up to $115 billion a year to the economy under a high-adoption scenario. These aggregate estimates do not show how the gains would be distributed -As Anthropic CEO Dario Amodei argues in [*Machines of Loving Grace*](https://darioamodei.com/machines-of-loving-grace) (2024), the economic effects of AGI depend less on what is technically possible than on "how the world chooses to distribute those gains." +Anthropic CEO Dario Amodei writes in [*Machines of Loving Grace*](https://darioamodei.com/machines-of-loving-grace) (2024) that he is less confident AI can address inequality and economic growth than that it can invent new technologies, because the economy involves many human constraints. ### Health and longevity @@ -113,15 +113,15 @@ These elements map directly to what the [C·A·G·R framework](framework/index.m Australian voices on AI futures emphasise both opportunity and the need for action. -**Australian research capability:** AI researcher Toby Walsh (UNSW) argues that Australia has research strength disproportionate to its population, while warning that delayed regulation and investment could weaken competitiveness ([IEEE Spectrum, 2025](https://spectrum.ieee.org/ai-sydney)). This is one expert assessment, not a settled international ranking. +**Australian research capability:** AI researcher Toby Walsh (UNSW) has said Australia punches above its weight in AI research ([IEEE Spectrum sponsored article, 2024](https://spectrum.ieee.org/ai-sydney)), and more recently that Australia needs to ramp up AI investment significantly and introduce new AI laws or risk missing out ([UNSW Newsroom, February 2026](https://www.unsw.edu.au/newsroom/news/2026/02/-dangerously-unprepared---unsw-expert-says-australia-must-regula)). This is one expert assessment, not a settled international ranking. -**The trust-safety-adoption link:** Australian surveys use different samples and questions, but both [Good Ancestors](https://www.goodancestors.org.au/whitepaper) (2025) and [KPMG and the University of Melbourne](https://kpmg.com/au/en/home/insights/2025/04/trust-in-ai-global-insights-2025.html) (2025) reported material public concern about AI risk and trust. Treat them as separate indicators rather than directly comparable measures. +**The trust-safety-adoption link:** Australian surveys use different samples and questions, but both [Good Ancestors](https://www.goodancestors.org.au/our-work/ai-safety/whitepaper) (2025) and [KPMG and the University of Melbourne](https://kpmg.com/au/en/insights/artificial-intelligence-ai/trust-in-ai-global-insights-2025.html) (2025) reported material public concern about AI risk and trust. Treat them as separate indicators rather than directly comparable measures. **Safety as a source of capability:** Some Australian organisations argue that evaluation, assurance and safety expertise could become areas of comparative strength, rather than trying to compete only on raw model capability. This is a strategic proposition, not an established economic outcome. **The "regulation taker" risk:** [Australians for AI Safety](https://www.australiansforaisafety.com.au/) (2025) warns that without proactive engagement, Australia risks having rules set elsewhere without our input. -**Democratic legitimacy:** Philosopher Seth Lazar (ANU) argues that AI governance must be democratically legitimate: "Whatever you think our AI future should be, it should be one that we have consciously chosen together" ([TechPolicy.Press, 2024](https://www.techpolicy.press/can-democracy-survive-artificial-general-intelligence/)). +**Democratic legitimacy:** Philosophers Seth Lazar (ANU) and Alex Pascal argue that AI governance must be democratically legitimate: "Whatever you think our AI future should be, it should be one that we have consciously chosen together" ([TechPolicy.Press, 2024](https://www.techpolicy.press/can-democracy-survive-artificial-general-intelligence/)). --- @@ -135,7 +135,7 @@ Australia's political tradition—democratic accountability, multilateral instit This points toward allied coordination, international regulatory approaches and strong national oversight—while working to avoid outcomes where power concentrates in single private actors or authoritarian governments. (For detailed analysis of governance visions, see [AGI governance visions](concepts.md#agi-governance-visions) in the concepts glossary.) -Oscar Delaney's [strategic visions analysis](https://oscardelaney.substack.com/p/strategic-visions-in-ai-governance) (2026) suggests that no single governance model is likely to dominate. National regulation, allied coordination and industry governance may coexist, with their relative importance changing as capabilities develop. +A [strategic visions report](https://www.iaps.ai/research/strategic-visions-in-ai-governance) by Oscar Delaney, Maria Kostylew, Oliver Guest and Peter Wildeford for the Institute for AI Policy and Strategy ([summarised by Delaney](https://oscardelaney.substack.com/p/strategic-visions-in-ai-governance), January 2026) maps nine strategic visions and concludes that none dominates. Each makes different trade-offs, so preparation should cover several scenarios. Australia should therefore build capacity that remains useful across different futures: independent evaluation, international partnerships, deployment standards, resilience and domestic safety expertise. These priorities do not depend on one governance forecast and the [C·A·G·R framework](framework/index.md) provides a way to organise them. @@ -164,7 +164,7 @@ Australia should therefore build capacity that remains useful across different f - [National AI Plan](https://www.industry.gov.au/publications/national-ai-plan) (December 2025) -- [Good Ancestors AI Policy White Paper](https://www.goodancestors.org.au/whitepaper) (May 2025) +- [Good Ancestors AI Policy White Paper](https://www.goodancestors.org.au/our-work/ai-safety/whitepaper) (2025) - [Human Technology Institute](https://www.uts.edu.au/research/centres/human-technology-institute) **SafeAI-Aus resources:**