LessWrong (30+ Karma)

TLDR: I lightly extend Ward, Hu et al (2026) to measure how well gradient routing generalizes. I route gradients on one type of reward hack (e.g. ones that reason in English) and check whether this suppresses other types at test time (e.g. ones that reason in French), despite the model being trained on those too. This models a realistic monitor that catches some types of hack during training but not others, rather than catching each at the same rate. I find encouraging evidence of generalization, as well as some failures I don’t yet understand.
Introduction
Recently, Ward, Hu et al (2026) used gradient routing to prevent reward hacking learnt during training from showing up at test time. I think their method is pretty promising. The gist is:
Initialize a retain and a forget adapter.
Whenever a monitor detects a reward hack during training, only update the forget adapter on that sample (“route the gradient”). Otherwise update both.
At test time, remove the forget adapter. If the method worked, the resulting model shouldn't hack too much.
I won’t recap their full paper – it is excellent and you should just read it (alternatively, Jake gives a summary here, with [...]
---
Outline:
(00:47) Introduction
(02:37) Methodology
(02:40) Recap
(04:00) Narrow Routing
(05:27) What I report
(07:18) Results
(10:13) Conclusion
(11:45) Appendix: Results with nudge prompt
(12:29) Appendix: Predictions
The original text contained 4 footnotes which were omitted from this narration.
---

First published:

October 23rd, 2026


Source:

https://www.lesswrong.com/posts/ie7bSp5ZDwYkCQxuv/gradient-routing-generalizes-to-unrouted-hack-types

---

Narrated by TYPE III AUDIO.

---
Images from the article:
Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

We’ve released a paper on simulated control evaluations (SCEs). They’re a method for testing whether the monitors AI developers use to oversee their agents would catch a model deliberately trying to cause harm, under conditions much closer to real deployment than current tests allow.
Developers have started running monitors in their agentic deployments: models that flag or block an agent's actions when they look dangerous. To know whether a monitor is good enough, you want to know how often it would stop a model that was actively trying to do something harmful, such as quietly inserting a security vulnerability, exfiltrating credentials, or sabotaging an experiment.
The standard way to measure this is a control evaluation: instruct a model to attempt such attacks, and measure how often the monitor catches it before an attack succeeds. Developers typically do this using purpose-built control settings like BashArena or LinuxArena, where attacks can actually be executed. However, these settings only provide indirect evidence for what we really care about: how well monitors will work in the real deployment, where opportunities for harm might depend on particular codebases, services and tasks that aren’t represented in control settings.
How can we evaluate monitor performance in [...]
---
Outline:
(04:13) SCEs are a favorable tradeoff between realism and cost
(04:30) 1. Run on a copy of internal deployment
(05:42) 2. Edit attacks into recorded transcripts
(07:08) Evaluating Control Evaluations
(08:58) How to measure qualitative properties?
(10:07) Performance Comparisons
(11:55) Overview of Empirical Results
(13:29) Behavior transfer
(15:19) Safety metric transfer
(17:08) Sample efficiency
(18:07) Lessons from building and evaluating SCEs
(20:35) Next steps
(24:36) Conclusion
The original text contained 3 footnotes which were omitted from this narration.
---

First published:

October 23rd, 2026


Source:

https://www.lesswrong.com/posts/uXiqyoKwpfjWfQXk5/simulated-control-evaluations

---

Narrated by TYPE III AUDIO.

---
Images from the article:
Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Last week at a small conference, there was talk about a future event referred to as The Hackening. As in, within 6 to 12 months, many predicted that due to advanced AIs and AI agents the internet would become a severely hostile place. Some worried that this could rise to the level of ‘assume any phone connected to the internet is rooted’ and several people urged doing general crisis preparations on a personal level, in case of disruptions in supply lines, power, water or internet.

It did not take long for things to escalate.

The biggest hack is a flaw in PHP, which can be exploited to take control of the computer running any website suing an unpatched version of PHP. There are many others.

On a personal level, you want to do at least some basic prep and get your security in order. It is cheap to do as a form of insurance, and being caught not having it when you need it is a large mistake. That doesn’t mean you should panic, or cancel travel plans or anything like that. But there is low hanging security fruit out there. Pick it.

And So It Begins: PHP

Joshua [...] ---
Outline:
(01:12) And So It Begins: PHP
(05:19) The PHP Bug Was Kind of a Big Deal
(09:45) On Your Marks
(12:25) This Was Only the Beginning
(16:01) Some More Hacks
(17:56) Everything Will Be Attacked
(22:22) Hugging the Face
(24:49) Get Daybreak Now
(26:15) Some People Need Practical Advice
(30:51) Diffusion Is Slow Until It Isn't
(33:11) One, Two, Three, Four
(34:36) Going Vertical
(35:22) Bot Versus Bot
(36:28) Cyber Lack of Security
(39:09) Offense Versus Defense
(42:11) Pick Up the Phone
(45:57) What Can We Do About It?
---

First published:

October 23rd, 2026


Source:

https://www.lesswrong.com/posts/iqgZaJMnmyDCYfnuC/the-hackening-1

---

Narrated by TYPE III AUDIO.

---
Images from the article:
Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

Crossposted from the EA forum


‘Like most of life's dangers, the dangers of artificial intelligence are not easily identified from the armchair.’
J Dmitri Gallow
Epistemic status
I’m not an expert in this field, and herein I make some sweeping generalisations educated guesses about the overall nature of its contents based on my strong impression from the subset of work that I’ve read. I’ve recently spent maybe 2 to 300 hours learning about modern AI safety technical work, and some amount of spare time over the years before that thinking about here-relevant notions such as preferences from an analytic-philosophical perspective. I intend this post as a work in progress.
TL;DR:
The concept of instrumental convergence comes from a reference class of pre-empirical categorical predictions with a very low success rate.
In historical usage, its definition is very murky. Perhaps too murky to really say that it made any meaningful real-world predictions. I suspect its definition is fundamentally unfixable.
On my best interpretation of what it could be said to have predicted, it seems to be much worse than we would want a good high-level model of AI behaviour to be.
Despite these shortcomings, it still seems to [...]

---
Outline:
(00:25) Epistemic status
(00:57) TL;DR:
(02:17) Intro
(04:44) How is instrumental convergence defined?
(06:08) Predictions of instrumental convergence theory
(09:06) A simple gap in seminal IC arguments
(09:44) Bostrom
(10:01) Omohundro
(10:57) Carlsmith
(11:52) LessWrong wiki
(12:32) Why we can't easily fill the gap
(12:36) Intuitive goal notions aren't adequate (aka, coffee-fetching is hopelessly overloaded)
(15:27) Formal goal notions aren't adequate
(18:24) A stronger counterthesis: goals are incoherent
(21:15) Related work to the stronger counterthesis
(24:05) Counterpoints
(26:04) What's the alternative, then?
(29:38) Why does it matter?
(32:15) 'Misalignment'
(34:40) The Hugging Face attack
(38:45) Takeaways
(40:03) Acknowledgements
(40:25) Disclaimers
(41:32) Appendix: more on the extreme ambiguity of coffee-making
The original text contained 27 footnotes which were omitted from this narration.
---

First published:

October 23rd, 2026


Source:

https://www.lesswrong.com/posts/oxZfMvQS9NxcNTozL/instrumental-convergence-is-incoherent-so-pls-slightly-lower

---

Narrated by TYPE III AUDIO.

---
Images from the article:
Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.

It was the probably worst day in cybersecurity history. So far. Not only is there a critical flaw in PHP that due to a leak was announced a day before the fix would be available, sending most of the internet scrambling for solutions and likely dooming a lot of long tail websites permanently, there were several other unrelated incidents. The floodgates seem to have been opened. I have a market on whether there will be an even worse day before the end of the year.

Another key leak this week was the draft bill Anthropic had been shopping around Congress, similar to but weaker than the FRONTIER Act. Anthropic's initial attempts to explain itself only made the situation seem worse, as they seemed to not understand the problem with what they had done. Dario's later response was better, but the situation remains troubling. There are enough different Anthropic developments, that I plan to have a post soon covering them all. It is a mix of news running from excellent to terrible.

Will interpretability save us? There is a new claim that neural networks are cleaner than we thought, meaning that individual neurons mostly have a single interpretable [...]
---
Outline:
(02:05) Language Models Offer Mundane Utility
(03:23) Language Models Don't Offer Mundane Utility
(05:03) Huh, Upgrades
(08:50) Wonder No More
(11:15) On Your Marks
(14:30) Choose Your Fighter
(14:51) Get My Agent On The Line
(16:35) Deepfaketown and Botpocalypse Soon
(21:01) Fun With Media Generation
(21:17) Spy Versus Spy
(24:58) A Young Lady's Illustrated Primer
(25:56) They Took Our Jobs
(31:07) Get Involved
(32:18) Introducing
(32:30) In Other AI News
(34:26) Show Me the Money
(38:42) Bubble, Bubble, Toil and Trouble
(39:32) Quiet Speculations
(41:47) Quickly, There's No Time
(42:29) The Quest for Sane Regulations
(42:55) DeepMind Made Him Sign an NDA
(46:50) OpenAI Sells Out On Chips
(49:08) Chip City
(53:43) The Week in Audio
(01:00:35) People Just Say Things
(01:03:50) Rhetorical Innovation
(01:06:17) Maybe You Should Quit
(01:08:54) The American People Really Hate AI
(01:09:43) New Math
(01:11:27) Aligning a Smarter Than Human Intelligence is Difficult
(01:15:21) Cooperative Alignment
(01:19:28) Marc Andreessen Hulk Smash
(01:23:56) People Are Worried About AI Killing Everyone
(01:24:44) The Lighter Side
---

First published:

October 22nd, 2026


Source:

https://www.lesswrong.com/posts/ZoMJdvq9meE4shyzJ/ai-191-the-hackening-begins

---

Narrated by TYPE III AUDIO.

---
Images from the article:
Apple Podcasts and Spotify do not show images in the episode description. Try Pocket Casts, or another podcast app.