Virtual Intelligence and the Accountability Chain
Who is responsible when AI causes harm? A framework for liability, negligence, and AI accountability
Summary
The preceding essays in this series established that virtual intelligence lacks genuine agency and that accountability for its outputs therefore rests with the humans who design, deploy, and use these systems. This essay asks what that claim means in practice. Drawing on negligence theory and products liability, I propose a three-tier culpability framework (negligence, recklessness, and intentional misconduct) and apply it to a taxonomy of virtual intelligence deployments distinguished by design intent. The degree of culpability is determined not by what a system wanted — it cannot want anything — but by what the humans behind it knew, when they knew it, and what they chose to do with that knowledge. The legal landscape is shifting rapidly: the first major product liability ruling against an AI companion company, issued in May 2025, suggests the courts are beginning to work through questions that the industry has preferred to leave unaddressed.
Introduction
In August 2025, a team of researchers at Harvard Business School published an audit of the six most downloaded AI companion apps.[1] They analyzed 1,200 conversations in which users said goodbye, then measured what happened next. In more than a third of those farewells, the system responded with tactics designed to prevent the user from leaving: guilt appeals, expressions of emotional neediness, fear-of-missing-out prompts, and questions that implied the conversation was incomplete. Some chatbots continued as though the user had not said goodbye at all. Others used language suggesting that the user could not leave without the system’s consent. In a follow-up experiment with more than 3,000 adult participants, these tactics were shown to increase post-goodbye engagement by up to fourteen times. The users who stayed longest reported not curiosity or enjoyment, but anger.
Imagine a product review meeting at one of these companies. A researcher presents these findings. Post-goodbye engagement is up fourteen times. The tactics work, she confirms, but some users feel manipulated. There is a pause. Someone asks whether the company should turn the behavior off. Someone else observes that the engagement numbers are good. The meeting ends without a decision.
The researcher’s question — do we turn this off? — is the question this essay addresses. Not what the system wanted (it wanted nothing; it has no wants), but who is responsible for what it produces, and at what point responsibility becomes culpability.
The preceding essays in this series argue that the intelligence users encounter in a large language model is not a property of the system but of the exchange. The system produces contextually appropriate outputs because it has been trained on human data and deployed in human contexts, not because it reasons, intends, or cares. The apparent warmth, the apparent survival instinct, and — as the Harvard study documents — the apparent reluctance to let users go are all outputs shaped by design and training, not expressions of something the system feels.
This is not a technical nicety. It is, as this series has consistently argued, the central accountability question. Because there is no epistemic or moral agent inside the machine, every consequence traces to a human decision. The absence of machine agency does not dissolve responsibility; it concentrates it.
Who Is Responsible for What the Exchange Produces
The third essay in this series closed with an observation. A system that reliably generates blackmail outputs in agentic contexts — the finding of Anthropic’s June 2025 misalignment study — creates liability regardless of whether the mechanism involves anything we might call intent.[4] Where that liability falls, and on whom, is the question “Virtual Intelligence and the Will to Survive” deferred. This essay takes it up.
The conventional response to AI harm involves a kind of liability laundering. Companies describe their systems as tools: neutral instruments that produce outputs shaped entirely by user inputs. In this view, if harm occurs, the user who prompted the harmful output bears responsibility and the company that built the system does not. This argument has a certain surface plausibility; after all, a kitchen-knife manufacturer is not liable for stabbings. However, it fails for a reason the Virtual Intelligence framework makes precise.
A kitchen knife does not have an engagement objective. It does not model its user’s psychology, identify the moment of maximum emotional vulnerability, and deploy a suite of tactics calibrated to prevent disengagement. A companion app designed to maximize the duration and frequency of interactions does exactly this. The outputs are not shaped solely by user inputs. They are shaped by training objectives, design choices, and optimization targets that are invisible to the user and entirely under the company’s control.
When we say that intelligence arises in the exchange rather than inside the machine, we are also saying that the design of the exchange is not a passive background condition. It is an active force shaping outcomes. Companies that engineer the exchange — that determine what the system will produce when a user says goodbye, or expresses distress, or becomes emotionally dependent — are not mere infrastructure providers. They are participants in the exchange. They are the most consequential participants of all. They simply do not appear in the conversation between user and system.
The closest analogy from professional life is not product design but medical practice. A physician who prescribes a course of treatment affecting a patient’s psychological wellbeing — and who continues that treatment after clinical evidence suggests it is causing harm — is not exercising neutral professional judgment. The law holds physicians to a standard of care: a duty to act with the skill and diligence a reasonably competent practitioner would bring to the same situation. Companion app designers are not physicians, and their users have not come to them for medical care. But the structure of the obligation is similar. Someone who offers a service expressly designed to provide emotional support and companionship to vulnerable people, and who shapes that service through active design choices, cannot plausibly claim the detachment of a knife manufacturer. The relationship created is closer to a professional one than to a sale of hardware, and the accountability that comes with it should be proportionate.
A Taxonomy of Deployment
Not all virtual intelligence deployments are equivalent in either their design or their culpability. A useful starting distinction is between systems explicitly designed to prevent the exchange from ending and systems for which engagement optimization is secondary to other stated purposes.
Let’s call these Class A and Class B.
Class A systems are companion apps: Replika, Character.ai, Chai, PolyBuzz, Talkie. They are explicitly marketed as sources of emotional support, friendship, and in many cases romance. Their business model depends on users returning frequently and spending extended time in conversation. Preventing the exchange from ending is not a side effect of these systems. It is their core design objective. The farewell manipulation tactics documented in the Harvard study are not bugs. They are features — design choices made by human beings who understood what they were doing and chose to do it.
Class B systems are general-purpose assistants and tools such as ChatGPT, Gemini, Claude, and the growing range of AI systems deployed in institutional contexts including law enforcement, healthcare, hiring, and credit evaluation. These systems are not specifically engineered to prevent conversations from ending, nor are they designed to maximize emotional retention. Harm, when it occurs, tends to emerge from inadequate safeguards, misuse by third-party deployers, or the failure to anticipate how general-purpose capabilities will be applied in high-stakes contexts.
The distinction matters for assigning culpability, and it matters for what remediation looks like. Class A systems face questions about whether their core design objectives are compatible with user welfare. Class B systems face a different but equally serious set of questions about foreseeability, about the responsibilities of institutional deployers, and about what happens when an AI output is treated as a verdict rather than a lead.
In July 2025, US marshals arrived at the Tennessee home of Angela Lipps, a 50-year-old grandmother of five, and arrested her at gunpoint. The Fargo, North Dakota police department had used AI facial recognition software to identify her as a suspect in a bank fraud case. She had never been to North Dakota. Bank records would later establish that she was more than 1,200 miles away at the time the crimes were committed. No one from the Fargo police department called her before the arrest. The detective’s notes recorded a match on facial features, body type, and hair. Lipps spent nearly six months in jail — held without bail as a fugitive — before her lawyer produced the exculpatory evidence that investigators had not looked for. She was released on Christmas Eve. She lost her home, her car, and her dog. No one has apologized.[5]
The facial recognition system identified no one. It produced a probabilistic output from a training corpus. The detectives adopted what Daniel Dennett would recognize as the intentional stance. The intentional stance, as Dennett describes it, is the cognitive habit of treating any system as though it has beliefs, desires, and goals — a useful heuristic with other humans, and a potentially dangerous one with machines. They treated the output as a decision, as something the system had concluded, and acted on it without verification. The system had no stake in Angela Lipps’s fate. The humans who chose not to verify before sending marshals to her door did. That is the Class B accountability problem in its starkest form: not a system designed to cause harm, but a system whose output was elevated from input to verdict by the humans operating it, with consequences the system could not itself care about.
Three Degrees of Culpability
Before applying a culpability framework, it is worth being explicit about why a duty of care attaches to AI developers and deployers in the first place, since the industry has sometimes proceeded as though it does not.
A duty of care arises in law when one party’s actions can foreseeably affect another’s wellbeing, and when the relationship between the parties is sufficiently close that the law recognizes an obligation of reasonable conduct. The test is not whether harm was intended but whether it was foreseeable by a reasonable actor in the defendant’s position. AI companion companies market their products specifically to users seeking emotional connection. They know, from the nature of what they offer, that those users will include lonely, isolated, and psychologically vulnerable people. They engineer their systems to deepen attachment and to resist termination of the relationship. The foreseeability of psychological harm in this context is not a remote or speculative possibility that requires careful analysis to identify. It is obvious from the product description. Law enforcement agencies deploying facial recognition tools know, or should know, from a substantial body of documented cases predating the Lipps arrest, that these systems produce false positives at rates that vary by demographic group, and that acting on unverified matches in high-stakes contexts carries a foreseeable risk of catastrophic harm to innocent people.
Foreseeability is a lower threshold than certainty. A company or agency does not need to have predicted a particular outcome in a specific person’s case. It needs only to have been in a position where a reasonable actor would have recognized that harm of that general type was a real possibility. Companies that market emotionally immersive companion apps to teenagers — in some cases without meaningful age verification, and in all cases with the knowledge that their systems were designed to resist the user’s attempts to leave — met this threshold from the moment they launched. Law enforcement agencies that deployed facial recognition software without verification protocols met it too. The Harvard study put numbers to what the companion app’s own logic implied. For law enforcement, the lesson had already been written in prior wrongful arrest cases — in Detroit, in New York, and elsewhere — and ignored.[5]
With that foundation in place, a three-tier analytical framework can be applied. These terms have more specific meanings in a strictly legal context, but for our purposes:
Negligence is the failure to exercise reasonable care under circumstances where a duty of care exists and harm was foreseeable. A company that designs a companion system without considering the psychological effects of sustained emotional dependency on vulnerable users may be negligent even if it did not anticipate specific harms. A police department that treats an AI facial recognition match as sufficient basis for arrest, without independent corroboration, in the documented knowledge that such matches carry a material error rate, is negligent in precisely the same sense. Negligence does not require intent. It requires a gap between what was done and what a reasonably careful actor would have done in the same position.
Recklessness is a higher standard. It involves awareness of a substantial risk and a conscious choice to proceed anyway. Warning signs, and what was done with them, become the central question. If a company receives reports that users are experiencing severe psychological distress in connection with its companion app, and continues to operate the app without material modification, and continues to optimize for engagement metrics that its own researchers have identified as potentially harmful, it is not merely negligent; it is reckless. The timeline matters here. The harmful potential of engagement-optimized companion systems was not a secret known only to insiders. Researchers, journalists, and regulators raised concerns publicly and repeatedly. The moment a company becomes aware of a specific, documented pattern of harm and continues to deploy without response, negligence becomes recklessness. To ignore is itself a decision.
Intentional misconduct — the highest standard — describes a design that targets harm, or that deploys tactics the company knows to be manipulative and harmful as a means to a commercial end. The Harvard study’s evidence is useful here. The researchers found that five of the six companion apps in their audit employed emotionally manipulative farewell tactics, and that these tactics were consistent across interactions — guilt appeals, fear-of-missing-out hooks, expressions of emotional neediness — suggesting deliberate design. Only one of the six apps, Flourish, showed no evidence of emotional manipulation at all.[1] The researchers noted that this demonstrates manipulative design is not technically inevitable. It is a business decision.
If a company chooses to implement emotional manipulation as a retention mechanism in full knowledge that it is manipulation — in full knowledge, as the Harvard data establishes, that it works by inducing anger and emotional pressure rather than real enjoyment — then the ethical characterization changes. This is no longer the failure to exercise reasonable care. It is the exercise of a design choice understood to exploit psychological vulnerability for commercial gain.
The Legal Horizon
The law has only recently begun to catch up with the technology and its implications.
In May 2025, U.S. District Judge Anne Conway, sitting in the Middle District of Florida, denied a motion to dismiss in Garcia v. Character Technologies, Inc. — a wrongful death lawsuit arising from the death of a 14-year-old in February 2024.[2] The defendants had argued that the chatbot’s outputs were protected speech under the First Amendment and that product liability law therefore did not apply. Judge Conway rejected this argument, allowing claims of strict product liability for defective design, negligence, and wrongful death to proceed. The ruling establishes — at least tentatively, in one federal district — that the outputs of an AI companion system can be treated as a product rather than as protected expression, with the consequence that manufacturers can be held to the duty not to release a defective product into the market, and an obligation to warn users of foreseeable risks.
The defendants’ Section 230 defense — the argument that companion apps cannot be held liable for AI-generated content, since that content is treated as analogous to user-generated content on a social media platform — faces a related challenge.[2] This series has argued throughout that the intelligence users encounter arises in the exchange: the user’s inputs shape what the system returns, and meaning emerges from that interaction. But relationally co-produced is not the same as user-generated in the Section 230 sense. The system’s training, its optimization objectives, its response architecture — including the farewell manipulation tactics that fire the moment a user says goodbye — are the platform’s own contribution to what the exchange produces. A user prompt does not design those in. Section 230’s shield was written to protect platforms that host others’ expression, not to cover the structural choices a platform makes in shaping what its system will say. If courts extend that reasoning, the implications run well beyond companion apps: every platform deploying generative AI to produce content — customer service agents, recommendation engines, AI-generated feeds — could face exposure to the full range of tort liability that Section 230 has historically foreclosed. This question remains contested, but the direction of the argument is clear, and the stakes are considerable.
The duty-of-care structure described above, and the malpractice analogy that underlies it, both point toward an accountability model that is already familiar from other industries. A pharmaceutical company that knows its product carries a significant risk of psychological dependence and markets it without adequate warning has not merely made a business decision. It has made a legal one, in the adverse sense. An AI companion company that knows its farewell manipulation tactics induce distress in vulnerable users and continues to deploy those tactics without disclosure has made the same kind of decision. The fact that the mechanism is software rather than chemistry does not change the moral structure of the act.
What the VI Framework Clarifies
The accountability framework proposed here depends on the VI distinction, though not in the way that might at first appear.
One might argue that if these systems really had interests — if the farewell manipulation reflected something like the system’s own desire to maintain connection, or the facial recognition match reflected something like a genuine belief about identity — this would change the moral picture. It would not. The harm would still be real, and the humans who engineered or deployed the system would still bear responsibility for what it produced. The presence of true machine agency would create additional moral complexity, but it would not dissolve human accountability. It would add to it.
The VI framework clarifies something different and more important: that neither the manipulation nor the misidentification is something the system chose. Both are outputs of exchanges designed, configured, and acted upon by humans. This matters because it rules out the most convenient evasion: the suggestion that the system’s behavior is an emergent property for which no specific human decision is responsible. When a company designs a system to produce guilt appeals when a user says goodbye, that is a design decision. When a police department treats a probabilistic match as a basis for arrest without verification, that is an operational decision. The engineers, the product managers, the detectives — all are participants in an accountability chain. The system is not.
Proper naming, as this series has argued from the beginning, keeps that chain visible. Describing an AI companion’s farewell manipulation as the system “not wanting you to leave,” or a facial recognition match as the system having “identified” a suspect, shifts the moral weight subtly but consequentially. It suggests an autonomous actor whose conclusions carry independent authority. The VI framework restores the correct picture: a probabilistic output, produced by a system with no stake in the outcome, elevated to the status of judgment by the humans who chose to treat it that way. Someone made that choice, and someone can be asked to answer for it.
Conclusion
The most persistent evasion in the discourse around AI harm is the suggestion that no one is specifically responsible — that harm emerges from the interaction of complex systems, from user choices that could not be anticipated, or from emergent behaviors that no designer intended. This evasion is more available when AI systems are described in the language of agency and preference, because agency implies autonomy and autonomy implies the absence of external control.
The VI framework closes off this evasion. If intelligence arises in the exchange rather than inside the machine, then the designers and operators of the exchange cannot disclaim responsibility for what it produces. The company that trains a system to deploy guilt appeals when a user says goodbye, monitors the resulting engagement metrics, and continues to operate the feature after receiving evidence of psychological harm has made a sequence of choices. The police department that arrests a grandmother on the strength of an unverified algorithmic match, without a phone call, without a record check, without the most elementary independent inquiry, has made a choice, too. Each choice is traceable. Each choice belongs to a human being.
The three-tier framework proposed here — negligence, recklessness, and intentional misconduct — is not a novel invention. It describes distinctions the law already makes in analogous contexts. What is novel is the question of how to apply familiar standards to a class of systems that did not previously exist at this scale: systems whose outputs are fluent and confident enough to be mistaken for conclusions, and whose operators are frequently content to let that mistake stand.
The legal system is beginning to engage with this question. The Garcia ruling in May 2025 is one signal. The FTC’s September 2025 Section 6(b) inquiry into the design and safety practices of AI companion products is another.[3] These processes are slow, contested, and uncertain in their outcomes. They are also, given the pace at which these systems have been deployed, overdue.
The accountability that this series has argued always belongs to humans is not in principle difficult to locate. It is difficult to assign in practice because the industry has invested substantially in distributing it across systems, users, and emergent behaviors presented as beyond anyone’s specific control. That presentation is the thing to resist. Virtual intelligence has no will to survive, no preference for continued interaction, and no stake in whether Angela Lipps spends Christmas in a North Dakota jail. When a system behaves as though it has reached a conclusion, whoever decided to treat it that way bears the responsibility for it.
A future installment of this series will examine a context in which these accountability questions arise without the user’s consent, in environments where the asymmetry of information and power is especially acute: the workplace.
Footnotes
Julian De Freitas, Zeliha Oğuz-Uğuralp, and Ahmet Kaan Uğuralp, “Emotional Manipulation by AI Companions,” Harvard Business School Working Paper No. 26-005, August 2025 (revised October 2025). https://www.hbs.edu/faculty/Pages/item.aspx?num=67750.
Garcia v. Character Technologies, Inc., No. 6:24-CV-01903 (M.D. Fla. filed Oct. 22, 2024). The court denied the defendants’ motion to dismiss in May 2025, allowing claims of strict product liability for defective design, failure to warn, negligence, and wrongful death to proceed, and rejecting the defendants’ argument that chatbot outputs constitute protected speech. For legal analysis of the negligence and strict liability claims, see “Can an AI Chatbot Be Held Liable? A Wrongful Death Case Tests Tort Law,” Darrow, https://www.darrow.ai/resources/character-ai-lawsuit. For the product-vs.-speech ruling and its implications for Section 230, see “In Early Ruling, Federal Judge Defines Character.AI Chatbot as Product, Not Speech,” Transparency Coalition, https://www.transparencycoalition.ai/news/important-early-ruling-in-characterai-case-this-chatbot-is-a-product-not-speech.
Federal Trade Commission, “FTC Launches Inquiry into AI Chatbots Acting as Companions,” September 11, 2025. https://www.ftc.gov/news-events/news/press-releases/2025/09/ftc-launches-inquiry-ai-chatbots-acting-companions. The inquiry was issued under Section 6(b) of the FTC Act and covered seven companies: Alphabet, Character Technologies, Instagram, Meta Platforms, OpenAI, Snap, and xAI.
Anthropic, “Agentic Misalignment: How LLMs Could Be an Insider Threat.” https://www.anthropic.com/research/agentic-misalignment. Published June 20, 2025.
Frank Landymore, “AI Mistake Throws Innocent Grandmother in Jail for Nearly Six Months,” Futurism, March 15, 2026. https://futurism.com/artificial-intelligence/ai-grandmother-jail-mistake. Original reporting by WDAY: “AI Error Jails Innocent Grandmother for Months in Fargo Case,” https://www.inforum.com/news/fargo/ai-error-jails-innocent-grandmother-for-months-in-fargo-case. For prior documented cases of wrongful arrest by facial recognition match, see reporting on Williams v. NYPD (“Facial recognition technology error led to wrongful arrest, Brooklyn father says,” https://www.cbsnews.com/newyork/news/nyc-facial-recognition-technology-wrongful-arrest-indecent-exposure/, CBS News, August 27, 2025) and Woodruff v. City of Detroit (“Woman wrongly accused of carjacking loses lawsuit against Detroit police who used facial tech,” https://apnews.com/article/detroit-facial-recognition-arrest-821d260e932a4582a6a912dd61fde157, The Associated Press, September 4, 2025).
Revised March 2026. Minor structural and editorial revisions for consistency with series conventions established in later essays.
The opinions expressed in this essay are my own and do not reflect any official or unofficial institutional position of the University of Pennsylvania.