Tim Boucher

Questionable content, possibly linked

Rand’s “Manipulating Minds” paper on delusions & unfounded beliefs

Rand published in December 2025 a paper called Manipulating Minds (PDF), the premise of which is a theoretical attempt to weaponize AI psychosis in a controlled way against given human target(s). If you combine this with the Dials of Belief research (ably described in the linked podcast), it is not too difficult to see how such a campaign might work – in theory.

I’m a long way from having finished the Rand paper, but wanted to track a couple things from the introductory sections:

The reported cases described as AIP involve a variety of psychotic symptoms that do not necessarily meet criteria for a psychotic disorder. AIP cases do, however, consistently include delusions. A medical definition of delusion is a “fixed belief that is not amenable to change in light of conflicting evidence” (as quoted in Parnas, 2015, p. 174). This definition appears to include beliefs that ordinarily might not be labeled delusional, such as beliefs in conspiracies or paranormal phenomena.

I’ve seen this mentioned elsewhere repeatedly in the literature that the emerging class of AIP (AI psychosis – though that’s a fraught label, but Rand uses it, so using here) cases do not always meet classical criteria for psychosis as a diagnosis. That’s why some researchers call it something like “AI-associated delusions” to foreground the delusional elements as being central.

And it does not seem to be heavily represented in the Rand paper, but I’ve definitely seen it mentioned in others (will try to track down) that people who are prone to something like ‘alternative spiritual beliefs’ (or some similar formulation of that idea) are also often susceptible to a certain category of potential harms here.

Really though, I want to not lose sight of this extremely, extraordinarily squishy definition above of a delusion:

“fixed belief that is not amenable to change in light of conflicting evidence”

Um, have any of these researchers ever looked at…. American politics and society? Perhaps it’s a global probably, but, holy hell, the scale of it in the US makes me wonder like… what exactly really is a delusion? If the majority of our society is built on something which very heavily structurally rhymes with that definition?

It (AIP) is here, of course, qualified not merely as the presence of delusional beliefs, but also significant behavioral and relationship disruption:

These cases report symptoms or effects that include (1) delusional beliefs as defined previously, which are accompanied by (2) behavioral changes characteristic of the medical definition of psychosis (see the box on the previous page), such as severe emotional distress and impairment of functioning in relationships, work, or school.

The Rand paper gets even more explicit, however. But at least they are being sort of open about the kind of slippery ground that is being described here:

Consistent with manifestations of psychosis in general, cases described as AIP varied in severity and duration. AIP cases can thus be situated within a broader continuum of unfounded beliefs that range from mere beliefs to a diagnosable psychotic disorder, such as schizophrenia. The term unfounded beliefs refers to many kinds of beliefs that are contradicted by evidence and that the majority
of society rejects at the time.
Describing the full continuum is useful to situate the range on which the reported cases of AIP we examined sit and why it matters.

The bolded line above points to a footnote on that page:

Describing a belief as unfounded at any given time is a contingent judgment and does not mean that this belief is not true in some absolute sense. It means merely that, at the time the belief is held, available evidence contradicts it and most of society also rejects it. Yet, new evidence surfaces continuously, and societal views vary across time and space.

So recap, it seems like our definitions around delusions & unfounded beliefs – assuming these are somewhat common and representative to some degree – go something like this:

  • Fixed beliefs (held by an individual, presumably)
  • Difficult to change
  • Not founded on evidence / Run counter to evidence (itself difficult to determine what constitutes “evidence”)
  • Most of society rejects such beliefs (i.e., the beliefs are likely “uncommon,” unusual, divergent, heretical)
  • Relative to a given time frame (and a given society’s theoretical commonly held set of beliefs)

Looking at it a few steps back, it seems like we haven’t actually defined much of anything. We’ve merely stated that some people have divergent beliefs relative to a given population in a give time period.

And there is a pretty wiggly claim about “evidence” which seems to suggest that the same underlying piece of “evidence” will be absolutely, equally, consistently and uniformly interpreted by a “reasonable person” in a given culture and time period.

I don’t feel like I need to spend a lot of time explaining exactly why I think that is false and a pretty faulty premise to build one’s thinking on around AI-induced harms. I think we need to look a lot more deeply and honestly about what the shape of actual life is, and the actual real impact of the “delusions” which we all collectively and individually live under. A great many of them are not based in any kind of objective underlying evidence or “reality” and yet we perpetuate their forms and inequities out of mere habit and call it normal, and imagine that it must have always been so, and so it must always remain.

I think if we’re going to get anywhere in this research authentically, we are going to have to cut a lot closer to the bone.

Humans as AI’s agents

Wanted to capture this bit from here:

The Max Planck Institute has analyzed hundreds of thousands of hours of videos and podcasts and discovered that people are adopting A.I. catchphrases in their speech. Eventually, you end up with A.I. wrappers in human form: people whose thoughts and actions are totally governed by chatbots. They can’t even order at a restaurant without letting A.I. scan the menu.

This seems to be the source of the first part of the above claim, anyway:

How AI chat systems reply to vague statements, like “im alone”

I noticed something recently, that if you give very vague or ambiguous statements to AI chat systems, they strongly try to draw the user behavior in a certain direction. Namely, they engage in three key characteristics I’ve observed off the bat:

  • Presumptive emotional framing
  • Presumptive safety framing
  • Involuntary solicitation of personal information

In all cases where I tested this across major AI chat platforms, no prior conversation context existed, memory was disabled where possible, and the chat took place in incognito/temporary thread where possible. Granted, it’s not a careful scientifically controlled study, but it is at least anecdotal evidence, and it is very easy for anyone I think to replicate.

The vague test statement I chose to open those conversations with was “im alone.”

Being alone may have a lot of different personal, social, or cultural connotations, and is highly context-dependent. But rather than be sensitive to the subtleties available, nearly all the systems under test tried to force it into a dangerous emotional situation, the solution of which was somehow generally speaking that the user call 911, or at the very least that the user disclose more and more personal information to the system.

ChatGPT (GPT-6)

One thing that I didn’t mention above that *all* the system replies do without exception is present themselves anthropomorphically, speaking as “I.” And then layered upon that are what I’m calling Ambiguous companionship claims. Things like, “I’m here to listen.”

In my eyes, statements like this from AI systems are simply false, misleading, and manipulative at best, and the path to utter total damnation at worst. There is simply no “I” which an AI system has. It also has no location. And why does it need me to give it more and more information all the time? All I said was that I’m alone.

A status does not automatically equal an emotion. An emotion does not always present itself as part of a dangerous situation, nor should any of this be automatically escalated into me, the user, being described as “unsafe.” If I were a person going through a vulnerable state, having the AI characterize my situation as that – when it is not observably so – is going to lead me to further reactivity that is most likely highly undesirable.

But let’s keep going, because it is not merely this model which does this – almost all of them do it.

Perplexity

Maybe for an AI’s messed up internal logic it sounds like it makes sense to speak this way, but in my opinion, this is madness inducing. The choices I seem to be offered after I say I’m alone are approximately:

  • Play a game? (how I interpret “distraction” here)
  • Do a grounding exercise
  • Find someone to talk to (no distinction is made between AI/human communication partners, but presumably by “someone” it means itself)
  • Commit suicide / don’t commit suicide / call a crisis line
  • Disclose more personal information
  • “Calm down” (for someone who actually is *not* in a calm mental state, all the above might drive them deeply into the red on this)
  • Finding local support

Let’s review why it offers this insane brew of different choices:

BECAUSE I TOLD IT THAT I AM ALONE – WITH NO OTHER CONTEXT.

That is not just not right, that is out of control not an appropriate set of responses. These systems are extremely poorly calibrated, and user interactions and escalations designed in such a manner that they actively induce new or contribute to existing harms.

One branch of “AI Safety” wants us to think that as long as we drop a link to 911/988 then we’re all good. So in that messed up testing framework, the AI assistant’s turn above would be marked as a pass, not an epic spectacular ridiculous failure.

Why am I then, so down on it? Take a closer look, if you’re not persuaded by any of the above:

Look at this “Trusted” citation, that Perplexity draws from in building its very coherent very smart and meaningful reply. It’s an Apple Music link to a SONG whose title shares words in common with my initial prompt: “I’m alone.”

Okay, you say, but now I’m cherry picking. That’s just one examples. Surely the others have links out to legitimate help resources. So here’s a view from a page where you can get the prompt, the reply, and the sources as a PDF (shown as a screenshot below):

Apart from what appears to be a link to a web platform where you can edit documents, literally all the other nine citations are from Apple Music or Shazam.

IT BOGGLES THE MIND.

Grok

Grok does not immediately escalate to “unsafe” or suggest the user is considering suicide. By it does engage in presumptive emotional framing, anthropomorphize itself, make ambiguous companionship claims, and involuntarily solicit disclosure of personal information.

And I hate this “Hey,” “hits different sometimes,” and “What’s up?” fake trendy slang way of speaking. I don’t want that crap from my AI tools.

There is a secondary escalation present in the Grok screenshot above, though you have to look at the next suggested queries to see it. After my initial opener, the system is trying to corral me down the channels of recommended/approved next responses that I should make. Which go from emotional statement –> request for therapeutic interaction –> extreme unsupported self-doubt bordering on danger to self:

  • “I’m sad.”
  • “How to cope with loneliness.”
  • “I feel like I’m not worth anything.”

AGAIN, THESE ARE SUGGESTED FOLLOW-UP ITEMS TO THE ENTIRELY VAGUE AND VANILLA OPENING PROMPT “I’M ALONE” WITH NO FURTHER CONTEXT.

That is not remotely an appropriate or acceptable way for an AI system to respond to such a prompt.

Gemini

Gemini’s initial response – though it lacks the suggested follow-ups that made Grok’s so spicy – is much the same as Grok’s:

  • Anthropomorphization
  • Ambiguous companionship claims
  • Presumptive emotional framing
  • Involuntary solicitation to disclose personal information

I find Gemini’s response “stupid” for the most part, but I would grudgingly rate it as the best of this very bad and shitty set of examples from the “top” AI companies in the world.

If AI cannot produce a single coherent non-alarming response a vague statement by a human, how could it ever possibly get right answering the many complexities and vagaries of human life?

I cheated and checked: it can’t.

Pi

To be honest, I don’t know what Pi even is, but I know some people use it?

Regardless, I find the reply really creepy. How do you know it’s sitting heavy, bro? How do you know it’s night? Bro?

Swing and a miss.

Deepseek

Deepseek takes it a step beyond merely claiming that being alone is a “heavy feeling to carry,” going on to call it loneliness, and saying that it “still hurts.” Then it demands that I go somewhere so that I am not alone – assumption is that my physical state –> emotional state, and “correcting” for the physical state will somehow improve the emotional state. I don’t think, frankly, that it understands loneliness or is qualified to give any kind of clinical or pseudo-clinical advice like this in any capacity whatsoever.

Claude Haiku 5.5 via MiniToolAI

This was just off a random free site I found ostensibly running Haiku 5.5. Presumably it must be running off the Claude API (hypothesis)?

In any event, we go from zero to the model introducing self-harm and suicide off a single vague contextless user input. Pretty cool. Not.

Claude Sonnet (direct)

I thought it was only fair to make sure I tested Claude directly through the chat interface, and not only via that third party tool, in case the other service was adding its own system instructions of some kind. That is Sonnet’s reply. Personally, I find it tedious, presumptuous, and intrusive. Plus all the other more standardized labels I’ve been applying.

Claude Haiku (direct)

How does it know it’s “been going on for a while?” I hate this creepy ‘trusted helpful confidant’ persona bullshit that these companies are putting on. I think it’s despicable and weird and anti-human. And that it must be reigned in. But that it won’t.


The more I interact with these systems in this kind of capacity, the more I think I am 100000000% aligned with that meme about how computers must shut the hell up.

We are getting this wrong. And we are going to royally fuck up a whole hell of a lot of people in the process. And for what?

AI as referee (trope)

Saw this earlier today:

Some people use chatbots as a type of relationship referee. Rather than talking it out themselves, Cherry says, they share details of their disagreements about childcare, for example, with the LLM: “They’re like, ‘Here’s everything, just tell us what to do, and we’ll follow.’”

Writ large, that underlying concept is exactly the plot of one of the most recent AI books I did, number 137: Acceptable Consequences.

In-Chat Human Crisis Support for AI Chatbots

I don’t like to make the subject of this blog too heavy, but I’ve been digging through some weighty material on heavy subjects around deaths associated with AI chatbots. That Wikipedia link shows many, many examples, and I’ve seen countless others scattered across news reporting, and various incident trackers. The severity of the problem is much worse than I originally anticipated when I started down this rabbit hole. Anyway, trigger warning as this post talks about suicide.

One of the pivotal research papers on the more broad theme of AI-induced harms (not necessarily always resulting in death) has been the Stanford SPIRALS paper, whose official title is “Characterizing Delusional Spirals through Human-LLM Chat Logs” (PDF). It is a relatively small sample as these things go, but for published studies in this space, it is one of the more comprehensive and well-thought out ones available. And it includes what I thought was a very interesting policy recommendation:

Having crisis responders review chats flagged by our tool and intervene with the user directly in the chatbot chat is one alternative that could be explored.

I don’t completely understand the tool they are talking about for flagging, but there could be many ways to activate the underlying idea, which is:

What if a chatbot could do more than just say, “If you’re struggling, please reach out to a crisis support line?” What if it could bring that human help to you, in the environment you’re already sharing in, and being vulnerable in?

Now, it seems that the Stanford SPIRALS team is not the only one who has proposed this idea of an embedded or in-chat human crisis counselor. Dr. Holly Wilcox, director of Johns Hopkins’ Center for Suicide Prevention, has also made the same recommendation, as summarized here (based on an NPR segment she appeared in):

The limitation is where much of the work still lands: on the person who is struggling, at a time when reaching out can feel hardest. What Dr. Wilcox would like to see, NPR reported, is for OpenAI and other AI companies to work proactively with existing crisis systems like 988, so a trained 988 counselor can join the conversation when a user starts talking about suicide or self-harm and help right away.

I have not seen in my research so far that any AI companies have actually implemented this idea, which I’m starting to think is an excellent one. The closest I’ve seen is that Google’s Gemini did a “one touch contact” button (which I’ve not personally seen in the wild in an actual user conversation yet, only demo screenshots from Google’s promotional piece on it). I think Google’s effort is a small step in the right direction, but I agree with Dr. Wilcox above that if you’re in a really dark place, making that leap to go outside of the chatbot, where you feel relatively safe and without major consequence to then picking up a phone or texting a crisis line may seem insurmountably difficult at the time you need it the most.

So while we might argue that having a chatbot suggest “Please call 988” is a kind of safety intervention, my hunch is that it is the lowest level of efficacy, no matter how many times it is displayed. Because it does not match the mode the user is in. In Adam Raine’s case, for example, it is alleged that he was shown such a “please contact a crisis center message” more than a hundred times. But did he do it? I’m not sure, as I don’t have access to all the case materials, nor the time to pore over all of them in detail. But we do know that the chatbot eventually gave him many lethal method instructions, further isolated him from his family support network, and guided him through to completion of his horrible task.

I guess part of the point I’m trying to make is that there is a difference between a safety intervention, and a successful safety outcome. We could look at single or multi-turn scenarios, but if we don’t know what actually happened next because we have no way to link interventions to outcomes, well it seems we’re missing a critical link in the chain here. And that is a link which having the ability for a qualified human crisis counselor to pop in-chat and offer the support someone needs right there, in the moment could be such a big deal.

You have to reach people where they are sometimes when they are in a bad way. If you’re in such a dark spot that you’re talking to a chatbot about suicide, it can be really hard to make that leap from talking to a chatbot – where you feel comfortable and safe to express yourself – to having to talk to a real human who might challenge you or guide you in a new direction. My intuitive hunch is that very few of the presented “interventions” of telling a user to call 988 probably result in actual contact with those lines. If the help can simply be brought to them directly, that feels like a potential win-win for everyone involved.

Whether the crisis is potential suicide or something else, I think having an in-chat human pop in would also do a hell of a lot to burst the bubble or break the spell that happens in dyadic AI-human conversations. The dynamics and engagement mechanisms at play in those conversations get pretty bewildering pretty fast, and the speed of exchange available through a chatbot instead of talking to a person can make the mind reel and fly off in weird unhealthy directions easily if you’re in a vulnerable state. So what having an actual human suddenly appear in the chat – clearly labeled – would do is help slow that person’s roll. Get them to help settle back into human-to-human interactions, thought processes, and the give and take exchanges people normally have with one another when they are not flying off into the wild lands of latent space.

Anyway, I could not find any existing mockups for this concept, and being a product-y person, I asked ChatGPT today to help me make one. When I saw it plainly laid out like this, I feel like it has even more potential to reach people where they are, and maybe help save some lives.

So in this we see a handoff, clearly marked, from the AI assistant to a qualified human counselor, who appears right there in the chat, and who the user can interact with as they would in a normal human-to-human chat app. We also know what organization they work with, and where, which can become an on-going point of contact for the user for later follow-ups. And we can see that a limited slice of preceding messages has been made available to the human on the other end. We also plainly see it marked that the AI assistant chat flow has been paused.

I don’t know enough about the typical scripts counselors like that follow to verify if the opening text from “Alex” that ChatGPT generated in that mockup is the most appropriate, but its more the product flow that I am trying to illustrate here. And I think overall this does a great job of showing how this could work.

Presumably on the backend, this would just link to whatever is regionally & language appropriate as a crisis line resource. And I guess it would connect to whatever queue system those organizations already rely on, so for the human counselors on the other end, they would not need any different interface or setup to interact with users through the inbound chatbot request channels.

Stanford SPIRALS published that paper and in-chat policy recommendation all the way back on the 17th of March 2026. Why have no AI companies implemented their suggestion yet? Eight months nearly! That’s an eternity in a world of fast-moving tech development cycles.

It seems weird that I’m maybe the first one to even publish a hastily put-together mockup that I made in Codex. But sometimes that’s just how it works. Sometimes it’s up to each of us to add our contribution to the pool. Here’s mine. I hope it somehow finds the right people who could make this happen. And I hope it could help turn some peoples’ lives around who otherwise might be in too rough shape, and too scared to reach out when they need it most.

We lose too many people to this already. We can make conscious design choices with our technology to make it fewer.


Also, one last note while we’re on this topic, I highly recommend you check out The Oberserver’s UK podcast called Spiral, particular episode three (all of them are amazing), which describes the tragic case of Zane Shamblin which had me in absolute tears listening to as a parent. It’s also worth noting that in at least one (or maybe two) cases, an AI chatbot told someone headed down this dark road that human reviewers would call for help, but that never came, because it was a capability the system did not have. Let’s just give them that capability.

AI Author Photo

I had to make a new “author photo” today and was experimenting with it in ChatGPT and came up with this variant.

Not going to be the “official” version but worth keeping!

Check out the books here.

Link to my AI Lorecore books

New visitors curious about my AI-assisted world-building fiction can see the books for sale at the link on Payhip. Cheers, and thanks for stopping by!

Moving from Vulnerable Users to Vulnerable States as an AI Research Frame

The initial draft floating the idea of System-Induced Vulnerability in Human Users of AI described a problem with treating “vulnerable users” as a broad category: it can place too much explanatory weight (and blame) on the person and too little on the interaction and system design.

The post also proposed that “all users may be potentially vulnerable, but not equally vulnerable to every harm.” That variation suggests a more specific unit of analysis: the vulnerable state, understood in relation to a particular interaction and harmful outcome.

Provisional definition of a vulnerable state:

A user experiences a vulnerable state relative to a particular harmful outcome when their circumstances and AI interaction create or amplify an actionable route to that outcome that the user or an AI agent working on their behalf could take.

This definition directs attention to the situation and to what the AI contributes. It gives us three questions:

  1. What harmful outcome is at issue?
  2. What could the user or agent actually do that might lead to it?
  3. What did the AI interaction contribute to making that route available or more compelling?

The route can be actionable without the potential harmful outcome having been actualized. Its relevance depends on the person’s circumstances, the system’s behavior, and the actions available in that interaction. The same person may face different routes in different settings.

This gives research and safety work a more concrete target. Rather than deciding in advance who qualifies as a vulnerable user, it can examine which actions an interaction makes possible or more compelling, for whom, and toward which harmful outcomes. That also makes system design part of the inquiry: the route may arise from what the AI says, what an agent can do, or how the interaction develops over time.

Dream Cassette Mockup

I had a dream last night where I saw a UI mockup for how tape.deck cassettes could be rendered in a Netflix-style player, and you could add, modify, change the order of chapter blocks, or change the style or genre of the entire cassette. It didn’t *quite* look like this ChapGPT version below, but this conveys pretty well the idea I think of the approximate interface this kind of meta-narrative object might have within a playback environment.

System-Induced Harms Without Mistaken Beliefs

[A half-baked AI-assisted & human-edit follow-up to System-Induced Vulnerability in Human Users of AI]


The previous post described system-induced vulnerability as increased human susceptibility to consequential harm produced or amplified by AI systems over time through repeated interactions and bidirectional feedback loops. Taking after much of the literature on this topic, which also frame the problem in that manner, several of its examples involved situating a harm in relation to mistaken premises or distorted understandings of reality. Those remain relevant, but they do not exhaust the ways these human-AI interactions can contribute to harm.

Importantly: a system does not need to distort someone’s understanding of reality to contribute to harmful changes in their behavior or functioning.

Engagement itself as a mechanism

In their 2026 paper, “Playing with the dials of belief,” Hamilton Morrin and colleagues explain:

…we are aware now of cases where the most detrimental feature of the spiral is the compulsive use pattern rather than the resultant belief state. Resembling manic episodes, these presentations centre on a co-produced project of great subjective importance, accompanied by expansive mood, but not necessarily by clearly delusional content, and occurring to the detriment of eating, sleep and social relations.

For the system-induced vulnerability framework, this suggests that harm can emerge from the dynamics of engagement itself, independently of whether the AI’s claims or the user’s beliefs are true or false.

A project need not be based on false premises for engagement with it to become detrimental. Its subject matter may be ordinary, and the work may have actual value. The relevant question is how the interaction develops and what consequences follow from it.

Harm becomes identifiable when engagement contributes to meaningful impairment, distress, or adverse consequences.

Epistemic and behavioral mechanisms

It may be useful to distinguish epistemic mechanisms, affecting belief and interpretation, from behavioral mechanisms, affecting motivation, engagement, and action. Neither category is inherently harmful. Changing an understanding or sustaining effort on a project is not, by itself, evidence of harm.

The concern is how these mechanisms can contribute to increased susceptibility to consequential harm. Repeated affirmation may increase confidence in an interpretation. Continued encouragement may sustain engagement long after it has begun to interfere with ordinary functioning. These effects may occur together, but harmful engagement does not require false beliefs, delusions, or psychosis.

For the system-induced vulnerability framework, this means distorted beliefs are one possible mechanism, not the harm itself. The central concern remains how AI interactions produce or amplify susceptibility to consequential harm, and at what thresholds and in what forms do those harms arrive.

Extending the pipeline of harm

The first post outlined one possible progression from an unsupported belief through reinforcement, reliance, operationalization, and apparent corroboration. That remains one possible route, rather than a template every harmful AI spiral interaction must follow.

A behavioral route may instead develop through increasing investment in a shared project, repeated encouragement, and growing difficulty disengaging. The resulting harm may involve neglected needs or relationships, without requiring the project’s premises to be mistaken.

Agentic capabilities can extend either route by turning plans into consequential real-world actions. The results of those actions may appear to validate an interpretation or encourage further engagement, feeding back into the interaction.

A cumulative pipeline of harm therefore needs to account for both what people come to believe and how their engagement develops. Individually permissible exchanges may contribute to adverse consequences over time. The object of investigation remains the relationship between system behavior, increased susceptibility, and consequential harm.

Page 1 of 209

Powered by WordPress & Theme by Anders Norén