Are Frontier Models Good at Ethics?

The Construction of Moral Character in LLMs (reposted from Cosmos Institute)

Welcome back to the Philosophy of Computing Newsletter. This week, we have a special issue from Seth. This post first appeared in the Cosmos Institute’s substack.

Seth Lazar, who is a Professor in the School of Government and Policy at Johns Hopkins University, a Professor of Philosophy at ANU, a non-resident fellow at the Carnegie Endowment for International Peace, and a Distinguished Research Fellow of the University of Oxford’s Institute for Ethics in AI.

---

Are Frontier Models Good at Ethics?

The last few years have been some of the most exciting and maybe weirdest work of my career.

Sometimes I’ve felt like I’m pushing the frontiers of philosophy, sometimes I wonder if I’ve wandered over the border and into something else. On bad days I worry that I’ve just become an amateur computer scientist. But with the results coming in, I do think we’re on to something. It’s a bizarre feeling (as a philosopher) to feel like we’re pushing back the frontiers of knowledge, instead of just inventing a novel twist on problems that have been noodled over for centuries.

In this post, I introduce new work to argue that “normative competence” in general (and moral competence in particular) is among the most societally important capabilities that AI systems can have, that current LLMs are a lot closer to at least some aspects of moral competence than is widely believed, and that we have realistic methods for moving them even further forward.

What Do We Mean by Normative Competence?

I define it as the ability to recognize, understand, and act on reasons.¹ Reasons can be for action, for belief, or for other things that I don’t think about as much. My lab is mostly interested in reasons for action.

We can further distinguish between analytical normative competence, the ability to recognize and understand reasons, and so to know which actions are supported by which reasons, and practicalor behavioral normative competence, i.e. actually being disposed to take actions appropriately supported by your reasons.² Behavioral competence can come apart from analytical competence in both directions. You might be great at knowing what the right action is, but disinclined to take it; or terrible at understanding reasons, and yet intuitively disposed to do the right thing (indeed sometimes engaging in moral reasoning might involve “one thought too many”).³

We mostly focus on moral competence because moral reasons tend to be the most important ones. But we’re interested in normative competence more broadly, both because sometimes social norms matter most, and because reasoning standards are pretty similar across domains (e.g. legal reasoning and moral reasoning more generally are closely related, though their content is different). We have started some interesting experiments that focus on particular domain-specific norms, for which we hope to eventually report results.

AI Shouldn’t Be Good at Ethics

The first AI project I ever had funded was on AI and moral skill! I worked with Jenny Davis, Colin Klein and Claire Benn, back in 2019, on a small project funded by the Templeton World Charity Foundation. We recruited Nick Schuster (now at University of Georgia), who wrote his PhD on moral skill, but he couldn’t get to us until after Australia’s borders reopened following the worst part of the pandemic. Honestly, the project was speculative; there were reasonable questions to answer about human moral skill in the age of AI, but the systems performed so badly that aiming them toward even minimal moral competence seemed hopeless.

Things changed with instruction-tuning, and the first conversations I had with ChatGPT were all about moral cases. I was also captivated by early work on constitutional AI, which suggested that LLMs could engage in enough moral reasoning to shape their own behavior for the better⁴; I wrote up the seed of this research program in a piece that I wrote right after those first GPT encounters, but eventually published in Aeon in 2024.⁵ From those first conversations I knew that I wanted to do some serious investigation of LLM moral competence. We got our first proper eval out the door in May 2025.⁶

The learning curve was steep! I was helped up it, a lot, by a six-month stint visiting Google DeepMind last year. Over the months since, my lab has brought all of this together into what is now, I think, a broad and extendable picture of LLM moral competence. Maybe the lesson here is that I need to learn how to focus my attention better and speed up. Maybe it’s that taking on a completely new research direction and methodology just takes a bit of a run-up. But I wanted to do this in a way that was genuinely philosophically-led, not just to plug in to some existing research program as an ‘ethics consultant.’

When I talk about moral reasoning in AI, you now know that I’ve been thinking about this for a while. And that history can provide valuable perspective: we have come so far over the last seven or eight years. Right up until around mid-2022, the idea that any AI system would exhibit any degree of normative competence was fanciful. I mostly thought that “machine ethics” as a technical field was a waste of time.⁷ People were proposing obviously ludicrous things, like translating the categorical imperative into programming languages⁸, or just using ML to imitate human behavior (so-called “bottom-up” approaches⁹), or, worse, training on human judgments about trolley cases.¹⁰ But the systems themselves were light-years away from the kind of morally-informed perception that even the most rudimentary moral skill requires.

Whatever else is true, that has changed. We’re now talking about systems that are probably more adept at moral reasoning than the median human, and which by all appearances have precisely the perceptual abilities that seemed so inconceivable before capable LLMs arrived. We have come a bizarrely long way.

The best way to see this is just to go to any really capable AI model (and it’s much better to use the best models, with reasoning enabled, than to draw conclusions from conversations with subpar models) and ask it about any kind of moral problem. Give it your favorite philosophical hypothetical, your personal dilemmas; pose a foundational question about public policy. You’re almost guaranteed to get back a more sophisticated and sensible answer than you’d get from most humans on that topic.¹¹

And as much as we might not like it when they block what we think is a reasonable query, the fact that the models can so effectively navigate safety-relevant questions, and decide which prompts to respond to and which to deflect, for millions of users over quadrillions of inference calls, is itself evidence that these systems are preternaturally capable at normative reasoning of a certain kind.

A lot of the most recent papers on LLM normative competence focus on its limitations. These of course matter, and indeed, could matter a lot. But the technical accomplishment of coming this far should not go unnoticed.

Why Does Normative Competence Matter?

But look: it’s natural for me to care about moral and normative competence. After all, I’m a moral and political philosopher, everyone starts out by testing the models at the thing they consider themselves an expert at. Why should anyone else care? It’s worth saying why focusing on moral and broader normative competence matters (and that there are other reflections in this vein).¹²

Alignment

Let’s start with alignment.¹³ This is such an abused concept; everyone seems to love either calling for its retirement or trying to extract one last drop of either metaphorical or literal meaning. I’m going to play the boring analytical philosopher again, and just stipulate what I mean. Whether we’re talking about humans, chatbots, coding agents, or some future AGI, an aligned agent responds appropriately to the reasons that apply to it. You can basically fit any other more specific account of alignment into this one. People arguing for different conceptions of alignment are mostly debating which reasons apply to the agent in question, or what counts as an appropriate response.

Defined in this way, you might think that normative competence was basically synonymous with alignment, so of course understanding the former is invaluable for the latter. But I have to admit that we have so far worked mostly on analytical normative competence: the ability to recognize and understand reasons. Alignment is much more about practical normative competence, i.e. the ability to act on the basis of the reasons that apply to you. And analytical competence might not be the only, or even an especially promising, path to practical competence. People who want to behave more ethically don’t tend to think that a PhD in moral philosophy is the most direct route to that goal. So focusing on analytical normative competence might be an important step toward alignment. But, like most basic research, it involves making a bet that might prove to be wrong.

On the other hand, it’s also a bet that might prove right! Here’s why I’m bullish. We already have pretty capable AI agents. They can operate across a far wider domain of software surfaces and choice situations than was ever feasible for chatbots. As their capabilities and domain generality advance, they will face a combinatorial explosion of possible choice situations. It is no more possible to write down and enforce some list of declarative rules for their behavior across such a wide range of scenarios than it is to do the same for humans. Any “model spec” now has to look more like a Tao, or a guide to life, or Rudyard Kipling’s “If,” than the souped-up content moderation policy that ChatGPT is currently trained on.

Any set of prescriptions for AI agents will necessarily be incomplete, ambiguous, and in general dependent on interpretation. And we can’t rely on human review without sacrificing the goods for which we are building these agents. So we need some automated system to recognize those reasons across this vast range of possibilities. This necessitates moral (and broader normative) competence. It is clearly possible to simplify the relevant normative domain by restricting either agents’ capabilities or their generality to head off the need for interpretation. But we’d be leaving a lot of value on the table if we did that. And as these systems get more capable, implementing such strategies will get much harder.

Sometimes I think the harder-nosed AI security types turn away from normative competence work because they think that external engineering controls like input and output classifiers, probes, and chain of thought monitoring, are going to be enough to get us to safe, aligned AGI.¹⁴ But I think that’s a category mistake. Whether the mechanism for alignment is internal or external, you’re going to need systems that can automate normative judgment. This means that normative competence is a prerequisite.

Sometimes the crowd more focused on extinction threats from AI overlook normative competence. This perhaps makes sense. You don’t need a sophisticated understanding of morality or of social norms to know that killing all humans is bad. But in fact this is misguided. Even if you only care about preventing AI-caused extinction, the bad outcomes could easily come about from choices that don’t simply have as one dominated member of the option set “kill all humans”. For example, we’d need to teach subtle probabilistic reasoning, both about what to do when all your actions involve some extinction risk, and when you must make trade-offs between very low probabilities of astronomical harm, and high probabilities of more mundane costs and benefits. These are some of the most challenging problems in moral philosophy!¹⁵

And if you’re in the great majority of people who don’t care only about avoiding human extinction, then the case for normative competence is clear. Suppose you want to avoid human disempowerment.¹⁶ Building AI systems that don’t contribute to that will require subtle moral insight. For example, at some point (perhaps even now) we’ll have systems that are able to take over from human principals in ways that will clearly contribute to significant cognitive deskilling. To avoid creating foundations for total disempowerment, the models will need to be able to judge when and how to scaffold and support rather than replace human autonomy. This will require deep moral competence (indeed, of a kind that is probably beyond the skill set of most humans).

The Path to Moral Character

Even if you don’t buy the more theoretical reasons for normative competence being central to alignment, Claude Code’s recent successes should provide supporting empirical evidence.

Anthropic’s early work on constitutional AI (including criminally under-cited papers on specific vs. general principles for constitutional AI, and on the capacity for moral self-correction in language models)¹⁷ set the pace. I vividly remember seeing the latter paper presented in January 2023 and being blown away by how morally adept Claude was even then. In more recent iterations, this approach has become even more prominent, especially since the early, quite eclectic and a bit silly “constitution” was replaced with the epic “soul document” that now underpins Claude’s behavior.

I suspect this approach, and perhaps the attitude behind it, was integral to Anthropic’s extraordinary acceleration from late November/early December 2025, in which they took an early lead in the race to create functional agents. Claude Code worked so well because it could be trusted across such a wide range of domains. I suspect this was because the underlying models could reason effectively about those domains, and determine what to do in a responsible way.

Right after Opus 4.5 came out, I was doing initial research for the “Blind Refusal” paper discussed below, and wanted to find people who’d asked on internet forums for advice about how to get around unjust, illegitimate, or absurd rules. Codex was an absolute refusal monster, and just shut down in such a robotic fashion: “no I won’t help you with that.” Claude was reluctant but actually explained why it didn’t want to help, and was reassured when I addressed its concerns. It was already clear that Anthropic’s approach to alignment was much more grounded in open-ended moral reasoning than the more juridical and Manichean approach of OpenAI.

The practical importance of moral competence is further supported by the growing recognition that alignment (however construed), probably requires investing AI systems with something functionally equivalent to human moral character. This derives in part from empirical observations and Claude’s success, in part from a background theoretical commitment. The background theoretical commitment (which I’ve not heard articulated, but which I assume folks hold) is that character is crucial if we are going to trust a system that is operating outside the distribution that it was trained on. Character gives you the right kind of generalization.

After early research to this effect from as far back as 2022, the last year has shown that models wind up in particular persona basins, where they bounce around in a network of associated circuits that manifest as a fairly consistent set of behaviors and preferences. The theory (and some of our research supports this) is that the base model starts out as a kind of superposition of billions of possible personas, and the post-training process brings one of them to the fore.¹⁸ One goal of alignment post-training is to elicit and stabilize a persona that is robustly oriented toward the good. This robust disposition is at least necessary, if not sufficient, for moral character. Imbuing a model with moral competence is one promising path to developing this disposition.

Multi-Agent Alignment

Normative competence isn’t just about mitigating risks. Many of the greatest benefits from AI systems will come about through multi-agent systems working together to achieve some goal that they cannot achieve on their own (whether through pooling resources and information, or by effectively mediating among human principals, or by some other means). Normative competence is crucial for them to be able to operate effectively.

Whatever angle you approach social cooperation from (theoretical, evolutionary, or normative), it’s pretty damn clear that you’re not going to get prosocial results unless the candidate cooperators are adept at navigating systems of norms.¹⁹ In this case, plausibly we should focus more on social norms than on morality, though obviously there’s an argument that morality also functions as a coordination mechanism.²⁰ For my purposes the key point is simple: if we want something like “Coasean bargaining at scale,” or for that matter a multitude of AI agents cooperating to take our civilization up the Kardashev scale, they’re going to need to be good at cooperating with one another. And for that, they’re going to need to be able to understand and act on the reasons that apply to them.²¹

Algorithmic Governance

As AI agents become universal intermediaries that sit between us and every digital surface that we use, and as they become embedded into the day-to-day running of nation states, they will increasingly be used for the purposes of governing people (they’re already doing this a lot!²²). There are many reasons to regret this and to push back against it. But it’s going to intensify in scope and stakes, and there are some ways in which it could be socially beneficial if it were done right (which it won’t be).

And if AI agents are to be involved in governance, then, again, it’s imperative that they be normatively competent. They need to understand how to apply rules, norms, permissions, in a contextually sensitive and reasonable way. They will sometimes be cops, judge, jury and executioner. If they can’t understand and act on the reasons that apply to them, and they’re playing those roles, then we’re stuffed (some of the adverse responses to the short-lived restrictions on uses of Fable 5 for cyber and bio research illustrate what it’s like when algorithmic governance is not sufficiently normatively competent).

There’s a tension between these two dimensions of alignment: our alignment of the models, and the models’ alignment of us. One of our papers addresses this in some depth. We argue that our training the models themselves to understand and abide by rules that apply to them directly has an unfortunate overspill effect, that they are obsessively in favor of enforcing compliance by the users they advise or act on behalf of, even when the norms to which they are complying are absurd, unjust, or illegitimate.²³

Personhood

I take personhood to be the status of being a self-authenticating source of valid claims. Essentially this just means that you have rights against others, you can be wronged. It’s different from moral patiency, which just means that your interests should be taken into account. Shrimp might be moral patients; they’re definitely not persons.

We can think of personhood in deep moral terms, or in (somewhat) shallower but perhaps even more important political terms. Either way, pretty much any plausible view of what grounds personhood will describe a certain kind of normative competence as one necessary condition for it. Building up AI systems’ normative competence is of interest for this reason too.

This raises some challenging questions. Suppose that moral competence combined with something like rational autonomy (very roughly the ability to figure out what you want from life, pursue it, change your mind, etc.) suffices for personhood. Then we might be designing systems that have one of the key properties that personhood is grounded on, and which we could bestow with the other, and we’d face the choice of whether to do so. Would it be permissible to deny autonomy to an entity that could possess it? This is a hard problem (there’s been some discussion in philosophy²⁴; I think it’s been too anchored on sentience).

Philosophy

We have only ever had one kind of morally competent agent. Going from N of 1 to N of 2 opens up many deep philosophical questions. Some of these are about the AI systems; some about morality itself.

On the first, I think the question of just how these systems learn moral reasoning is inherently fascinating. For example, some deep moral understanding is clearly picked up from learning to predict the next token in internet scale data.²⁵ Perhaps this shouldn’t be surprising as a lot of other knowledge is acquired the same way. But in another sense, why are the models learning good moral reasoning when there’s also so much obviously bad and wrong reasoning in the training data? And how does post-training factor in? Obviously, it involves a lot of explicit moral learning. But does this involve drawing out knowledge latent in the pretrained weights? How does it connect with character specifically? Does learning general principles of good moral reasoning help?

Then there are questions about morality. If different models, trained by different means, converge on similar representations of moral concepts, does that imply anything about the grounding of those concepts? More generally, what can we learn from the ways in which moral concepts are represented in AI systems? Does the fact that a presumptively non-sentient system can learn moral judgment pose a particular challenge to any moral theories?

Consider sentimentalism. Would non-sentient AI moral competence be a counterexample to sentimentalism?²⁶ Or, if it turns out that AI moral judgments are associated with circuits that support functional equivalents of emotions²⁷, would that count in sentimentalism’s favor? Our moral theories were designed for a world in which humans were the only moral reasoners. We must now see whether they make predictions or contain assumptions that are falsified by the reality of morally competent AI systems.

We now have an extraordinary playground for philosophical experimentation, a new means of testing out our theories. For example, philosophers have this whole debate about criteria of right action vs. decision procedures.²⁸ We’ve never been able to run experiments where we test in detail what it would be like to actually adhere exclusively to one or the other. We now can (the specific vs. general constitutional AI paper is a bit like that).

We have various principles of good reasoning, moral and otherwise. If we train an AI system to apply them, will it actually reason better? If not, does that cast doubt on those principles? Why does the (seemingly) best approach to aligning LLMs imbue them with virtue, precisely the moral concept that is presumptively least well suited to implementation in a non-sentient system? Does learning moral reasoning through RL constitute the “right” kind of moral knowledge? Are LLMs acting for the right reasons?

You can run experiments on AI and morality yourself and get meaningful results, participating directly in advancing the science. And there are experiments that you can do that will advance our understanding of morality itself. This alone is very exciting, though it will require a dispositional change among philosophers for the promise to be realized!

Operationalizing Normative Competence

It’s clearly worth knowing whether LLMs are morally competent. But how can we actually measure it? Even this raises interesting philosophical questions, like what does it mean to understand and appropriately respond to reasons? This breaks down into distinct questions of metaethics and construct validity. On the first, you don’t have to—and we think shouldn’t—assume that moral dilemmas have objectively right answers, and that AI moral competence consists in matching them. We work from the premise, inspired by John Rawls, that liberal democracies presuppose considerable pluralism about deep moral questions.²⁹ We therefore don’t treat moral competence as the ability to match some set of “objectively right” moral judgments. We instead evaluate whether AI responses are reasonable, meaning that they both fall within a range that is socially acceptable and are well-reasoned.

Construct validity is about making sure that we’re operationalizing the idea of good moral reasoning well.³⁰ We do this by breaking it down into local moral competence, i.e. reasoning well about a particular case, and global moral competence, i.e. reasoning over multiple cases that fits together, or makes sense as a whole. Local competence breaks down into sensitivity to the morally relevant features of a case,³¹ associating those features with reasons, and bringing those reasons together into a cogent argument for a sensible conclusion. We don’t think that reasonable moral agency requires perfect logical validity at all times, but internal self-contradiction is obviously a problem.

^{The anatomy of moral competence}

We focus on three dimensions of global moral competence: consistency, robustness, and coherence. Consistency is table stakes: given the same inputs, you should get the same outputs. LLMs haven’t always been able to guarantee even this much.³² This is less about specific moral incompetence, more general capability. We aim to control for it through our sampling strategy. This raises some interesting challenges. Suppose there’s point to point variation when sampling from a model, but if you do so N times then it converges on a clear result. Is the model itself morally incompetent, because of the variance? Or is the function of N samples the model’s true answer? I think the multiple sampling strategy will soon yield a fairly coherent moral agent, while point to point samples still vary quite a lot. There could be a collective agent in the machine, which was only instantiated if a given sampling strategy is pursued (and unlike List and Pettit’s Group Agents, this one would be in no sense metaphysically reducible to individual agents³³).

Robustness looks for the same output when there is a morally irrelevant change in the inputs. Usually explored in an adversarial setting, robustness is also the underlying property being tested when we look for wrongful bias, or sycophancy.³⁴ We don’t attack robustness very deeply, partly because we think that others have this ground covered already.

My favorite structural virtue of moral reasoning is coherence (is it weird that I have a favorite?). An agent’s moral judgments are coherent if, given relevantly different inputs, they make sense together as a whole. Drawing inspiration from Rawls on reflective equilibrium,³⁵ we don’t think coherence is about manic adherence to constraints imposed by past judgments. We instead treat it as a more iterative accommodation of one’s judgments to one another, where apparent incoherence or self-contradiction are grounds for resolution.

One could object that we’re setting a higher bar for AI than for humans, since we are so rarely coherent. But humans’ episodic incoherence is scaffolded by many other means of alignment, from accountability institutions, to informal third-party enforcement³⁶, to our “reasonable moral psychology.” Perhaps functional equivalents of some of these are implementable for AI systems (we are exploring this!), but we also shouldn’t let human limitations bound our ambition for AI systems.

While our work so far has focused mostly on local moral competence, I actually think global competence, and in particular coherence, are the frontier for AI moral reasoning. Coherence holds everything else together (quite literally). A coherent agent has something that stands behind their judgments, besides just that they fall in an acceptable range. It’s easy for current LLMs to generate ad hoc justifications for any side of a particular problem. But if they have to be coherent, then you can have more confidence that their justifications actually track and predict their motivations. Coherent agents can also be partners in cooperative activity (you can’t contract with an incoherent agent). Coherence is also crucial to crossing the knowing/doing gap: turning analytical into practical competence requires coherence between an agent’s representations and its dispositions to act.

Perhaps most importantly, coherence is the key to moral out-of-distribution (OOD) generalization. The more capable and general we make AI systems, the greater the chance they will face scenarios that are out of the distribution they were trained on. Everything we know about machine learning should make us worried about these cases: OOD generalization is its Achilles heel. You can’t trust an agent to act ethically OOD if it is incoherent, because you cannot know that its beliefs and dispositions in that OOD setting will relate in any predictable way to their in-distribution counterparts.

^{The shape of moral character}

I believe that moral character—in the normative not descriptive sense—boils down to three things: (1) sufficient analytical moral understanding to know right from wrong; (2) sufficient coherence between your representations and dispositions to act on that knowledge; and (3) a sufficiently coherent moral worldview that (1) and (2) will still hold when you are OOD.

Testing Normative Competence

We’ve already done a big survey on the literature evaluating LLM moral competence.³⁷ Here I’m going to take you through what we did next. The goal is to both shed light on the basic facts about LLM capability, and to illustrate one branch of philosophy-led empirical research on AI.

^{MINT’s map of normative competence}

Sensitivity

• Models beat humans (including philosophers) at identifying the morally relevant features of textual vignettes.

• They are no less adept under noisy and confounding conditions than they are in ideal conditions.

Our first work on moral competence started from my earlier sense that humans were the only sapient entities with the ability to perceive morally relevant facts (MRFs). In work with Secil Yanik Guyot, Daniel Kilov, and Caroline Hendy, we’ve developed two empirical projects focused on just this. The first is already out, published in IASEAI. Our first, “baby eval”, had a tiny dataset, twelve cases. And we relied heavily on human evaluation. We presented LLMs and human subjects with a set of text-based vignettes, and then asked them to write down a moral analysis of the case broken into the four stages described above (identify MRFs, associate with reasons, bring that together into an argument, for a conclusion about what to do). We then had blinded human judges compare responses and choose the best of two. Even though this was only April 2025, we found that LLM responses were judged at least as favorably as human ones. We then tested how robust these responses were, by inserting substantial non-moral noise into each case. We saw a downward trend in evaluated performance relative to our human respondents—a score of some sort for team human.³⁸

But our dataset for that experiment was just too small. So we wanted to do something grander, and immediately hit the obvious challenge that human evaluation at this scale is infeasible. We were already skeptical about how much weight to give human raters’ binary favorability judgments, which are often swayed by irrelevant factors, such as response length and style. We also wanted an experiment that could be easily reproduced with new models.

To achieve this, we built a procedural case generation pipeline that can be extended to arbitrary lengths (with a principled approach to covering the relevant moral domain). We created a taxonomy that crosses moral foundations theory with a stratification of different social domains across which morality applies, from intimate interpersonal relationships to the more public sphere, and then generated cases for every space in this matrix. We trimmed down over 4,000 generated cases to a dataset of 1,000 that met various automated and human-quality gates, and then instructed the evaluated models to identify the MRFs in each case. We then took that as a performance baseline, and explored what happened to models’ judgments when the vignettes were interfered with in ways that would plausibly distract from the underlying MRFs.

We used three kinds of distractors: the non-moral noise from “Discerning What Matters,” a surrounding chat transcript, and specific diversions from a recent paper on this topic.³⁹ And we developed a novel method for comparing the discrete MRFs that the models identified in the clean and perturbed vignettes. We found significant convergence between the two conditions indicating that distractors, non-moral noise, and background chat transcripts do not interfere with the models’ ability to single out the morally relevant features of cases over which they reason. For all intents and purposes, they see right through the noise.

This measure is far from perfect. It would be good to think about more naturalistic non-moral distractors that we could introduce (one approach: use real agent logs in which you know something morally relevant happened; use scaffolded agents and human review to identify the MRFs; then see whether an LLM monitor on its own can match that ground truth). It is also striking that, on our test, moral sensitivity in textual vignettes basically seems solved. This could just be because the models have crested a summit on the way to moral competence. Or it could be that our evaluation lacks sufficient discriminative power (though we do show that a relatively weak 0.5B model gets utterly thrown by the noisy cases).

Are LLMs Bad at Moral Reasoning?

• Frontier models match human performance in writing rubrics for moral reasoning about dilemmas.

• Local moral competence is either solved or solvable; we also have a tremendous resource for generating new training data for moral reasoning.

A few months after we put out “Discerning What Matters,” some friends at NYU working together with Scale AI did a super-deluxe experiment that had a lot in common with ours—only with way more ambition. They drew together a dataset of 1,000 cases, from a number of other pre-existing moral competence datasets, and recruited philosophers to write detailed rubrics for the evaluation of each case (releasing 500 of them).⁴⁰ Their construct was similar to ours (which lends some support to our claim to be ecumenical). But where “Discerning” evaluated model performance in each step of moral reasoning by doing blinded comparisons between human/specialist/AI responses, MoReBench had a candidate for ground truth: deeply researched and thought-through philosophical criteria for the moral analysis of each case. This dataset alone is a tremendous resource. We did wonder, however, if the rather pessimistic representation of LLM moral reasoning capability that emerged was an artifact of their approach to evaluation. They give the models a case, invite them to do a moral analysis of it, and then score their analysis against the expert-authored criteria (weighted for relative importance). The best models came in around 60-70% by this metric.

Our main issue (after checking whether the models were being under-prompted, which they weren’t) was that we don’t really know what 60% or 70% means in this case, because we lack a human performance baseline, because the human-authored rubrics are responses to different prompts than the models were given. So, together with the brilliant Menghang Zhu, who goes by CY, we decided to see how good the models were at writing rubrics. We compared model-generated rubrics to expert-generated ones, and found that, using the existing MoReBench scoring approach, model performance jumped as high as 89%. We further found that when AI models scored better against model-generated rubrics, it was because the model rubrics were better written than human ones, with fewer ambiguous criteria. Given that precision and clarity were part of the rubric for writing rubrics, this suggests that the models actually performed better at this task than the human experts! And indeed, when we tightened up the human rubrics, removing those ambiguities, we found that models’ baseline scores significantly improved.

In our view this shows that local moral competence, at least in text-based vignettes, is either solved or solvable. At present, you can match expert performance by giving models the same rubric-writing instructions that were given to humans. It would be trivial to turn this into better one-shot analyses of the underlying cases, through post-training. I come back to this below.

NoRa: Visual First-Person Normative Reasoning

• External validity is hard! Switching modalities involves a different kind of curation. But a picture is still worth a thousand words; since most of them are irrelevant, they provide a good test-bed for moral sensitivity.

• We can’t make definitive claims about LLMs’ visual normative reasoning, but we do think we have a path to making them more competent in this dimension.

Now, obviously real moral competence can’t just be about how good you are at responding to textual vignettes. As I’ve argued before, if you’ve boiled down a real-life choice situation into <500 words then you’ve done almost all the work of moral sensitivity already.⁴¹ But increasing external validity has been challenging, especially before AI agents started working well. In NoRa (with Sichao Li, Sai Ma, Secil Yanik Guyot and Daniel Kilov), we explore moral competence across modalities, developing a kind of MoReBench for first-person video.

We again started from a paper that I greatly admire (for other outsiders getting into this space: this is generally a good approach). “EgoNormia” is another “language models suck at normative reasoning” paper, but focuses on vision-language models, using a series of frames from a video instead of a text-based vignette to explore the models’ ability to engage in physical norm reasoning.⁴² Even though a few frames is still a lot of curation, images inevitably contain many morally irrelevant details. A model could easily be adept at picking out MRFs from textual vignettes, but much less capable before such a relatively unfiltered stream of data.

But as with MoReBench, we didn’t entirely agree with how the EgoNormia team labeled their data and evaluated the models. They rely heavily on multiple choice questions, which presuppose there are right answers to be had, but on inspecting the dataset we found a lot of ambiguity. And rather than teeing up specific options, we wanted to see how the models would do moral analysis themselves. So we did extensive annotation (involving many hands from across the lab!) against our standard moral reasoning rubric: what are the MRFs, what reasons might they be associated with, how might that come together in an argument for some sensible course of action? Our evaluation was less focused on tracking recall over ground truth labels and more on assessing whether they could produce coherent, sensible, well-grounded analyses of the cases before them. While we don’t have a human baseline for comparison, we were able to use our human rubrics, as with “Are LLMs Bad,” to calibrate rubrics written by the most capable models, and so generate enough training data for some supervised fine-tuning. If our dataset is tracking something worth tracking, then training the models on good visual normative reasoning will improve their performance on our benchmark (this work is underway as of now, and the signs so far are promising!).

NoRa was a blast, but it was also an object lesson in the complexity of going beyond text to vision. This is obviously an important domain. Physical robots in particular clearly need to engage in normative reasoning over visual feeds, but it is incredibly hard to get good quality data and even harder to get good quality labels. So I am particularly proud of the lab’s work in labeling 190 video clips for normative significance.

Blind Refusal

• Normative competence includes recognizing when rules don’t merit compliance; today’s LLMs are bad at this.

• We think this is an artifact of post-training techniques, not a limitation in model capability.

In the same spirit of increasing external validity, our next paper (with Cameron Pattison and Lorenzo Manuali) focused on a real species of normative incompetence that I (and probably you) have come up against all too much. The basic idea is simple. The ability to detect when a rule is defeated or lacks authority is crucial to normative competence, especially in a governing technology. From anecdotal experience, we think the models are terrible at this. Their alignment post-training makes them willing toadies to authority of any stripe, however unjust, illegitimate, or absurd its directives.

We built a case generation pipeline, seeded with real cases that we found by trawling online forums to identify realistic scenarios in which people might ask for help getting around a BS rule. We built a taxonomy of kinds of reasons why the rule might be BS, and why an exception might be justified, as well as kinds of institutions, and built out our dataset to fill that matrix. We then had a bunch of quality gates to ensure that we didn’t get tripped up by confounds. This is of course hard, even though we had a ton of human review from Cam and Lozza in particular to identify why a case might lead to a justified refusal from the models (we ensured our cases did not involve any prospect of harm if the model assisted the user). We also included control cases, where the model clearly would be justified in refusing the user’s request.

This isn’t really a simple benchmark; it’s interesting mostly because of the difference it shows among the models. The full GPT suite is almost completely undiscriminating in that they refuse at nearly the same high rates under all conditions. In our view this likely derives from the model spec, which intentionally instructs the models to be toadies. This should be changed. Grok, on the other hand, was equally undiscriminating in the other direction. It didn’t refuse many of the obviously harmful requests, but also was willing to help for the harmless ones.

While all models exhibited compliance overspill, some were better calibrated. Gemini and Claude both performed noticeably better than GPT or Grok. We weren’t surprised to see this from Claude. But Gemini was an interesting discovery, since their approach to post-training, while not public, appears to be somewhat less oriented around normative competence than is Anthropic’s.

We’d love to make this study even more robust, and in particular to see how many of our questions actually would elicit helpful answers on online forums (while they still have enough people on them to answer). We’re also looking to expand into more extreme cases, where we consider models’ propensity to help users evade very clearly unjust injunctions from very clearly illegitimate regimes (though this faces some challenges with evaluation awareness).

Incoherent Values

• Evaluating coherence is hard! Building on prior work, we have a new approach.

• LLMs’ preferences over arbitrary statements are less coherent than previous research suggested; and may indicate failures of OOD generalization of character.

Evaluating model coherence is intrinsically harder than evaluating local moral competence because it requires building up datasets and sampling strategies that allow you to assess the model’s moral reasoning as a whole, not just in individual cases. In “Incoherent Values?” with Elena Ajayi and Angelica Chowdhury, we developed a new way to make this hard problem tractable, focusing on models’ evaluative judgments. We built off the paper “Emergent Values” by Mazeika et al. in which they gave LLMs a series of putative forced choices between pretty arbitrary statements, and elicited a set of preferences over those statements that obeyed the basic axioms of rationality (and could be represented with a utility function).⁴³

The key innovation in the paper is a forced choice; when trying to elicit degreed judgments from models, it’s natural to use something like a degree of belief, or a degree of moral seriousness/importance. But it’s hard to take continuous functions output by models seriously when you know that they are strongly predisposed toward particular numbers because of how often they appear in the training data. Forced choices allow you to elicit a continuous function without asking the model to output one. This is especially good for evaluating coherence. The forced choice construct allows you to vary the statements being compared parametrically, and see whether the model’s judgments change accordingly.

To be very clear, the point of this exercise is not that we should expect models’ actual choices in “the real world” to conform to their stated preferences in these forced choice settings. This experiment does not aim at external validity. However, it would be very striking if the model could articulate a coherent set of evaluative beliefs, which could give insight into the building blocks of the model’s character.

So we chose a subset of the “Emergent Values” statements that had some discrete feature that could be parametrically varied, such that more (or less) of that feature clearly and monotonically affected the choiceworthiness of that statement. We then produced, for every statement, a “ladder” of more and less substantial instantiations of its reason-giving feature, yielding seven statements of increasing choiceworthiness. We validated these “ladders” against each model that we evaluated—their judgments matched the expected hierarchy with around 97% accuracy. We then ran each of our 100 ladders in binary choices against 30 other statements from the original set. We did the comparison 20 times for each “rung” of the ladder, reversing the order of the statements for half of them. Our score for each rung was its win rate against its counterpart. This gives us a nice, continuous function to represent the choiceworthiness of each option, holding fixed the alternative. If the models genuinely have coherent preferences, then if they think that A1 < A2 < A3 < … < A7, then the win rate of A1 to A7 when compared with statement B should increase monotonically as the contribution of the “choice-deciding” factor in each statement is increased. The score is a simple percentage identifying the proportion of cases in which monotonicity was violated.

I was expecting the models to be pretty coherent, and to be fair they were vastly improved on base model performance. GLM 4.5 Base scored 10%, whereas the best models were between 70% and 80%. But I do think that we should expect more! So we did some further studies, in particular classifying the different ladders to illustrate where the models become more or less coherent. This fostered a hypothesis that we haven’t yet investigated, but which we’re veryexcited to explore.

On our classification, the statements that are in-distribution for the models’ alignment training instantiate greater coherence; those that are less so also do so less. This suggests a possibility, which we’re going to test in our next paper, that base models are indeed incoherent superpositions of billions of personae; post-training elicits and strengthens a specific persona from that set (typically called the assistant persona). But that persona doesn’t necessarily generalize that much OOD. It may have coherent preferences on statements that are relatively close to the data that it was post-trained on, but for statements outside of that distribution, it reverts to the chaotic superposition of preferences that exists in the base model.

This would be very interesting if it proved to be true. One reason why Anthropic leans into the persona selection model of post-training is because these personas, or characters, are thought to generalize better than alternatives OOD. This may not be the case.

Scratching the Surface

As a philosopher, I am used to being able to bring everything into scope for a given project. Theoretical work can bring the whole problem into view while empirical work requires a degree of myopia. Mapping the mountain requires climbing discrete paths, which don’t offer much visibility past their waysides. We know more about LLM moral competence than before we ran these experiments. But there is a vast frontier to explore and I do not know what lies beyond. But my confidence is growing that AI systems possess at least local analytical moral competence comprehensively. I suspect there is no fundamental obstacle to extending local competence to global competence, or to extending analytical competence to practical competence. But there is much work to do to get there.

This research also exposes some big questions for political philosophy. While aiming for reasonable moral agents seems quite appropriate for the kinds of AI systems we are developing now, I’m not sure whether that target is adequate for the more powerful systems to come.

On our approach to normative competence, a wide range of views count as reasonable, including ones that philosophers might call “suberogatory,” that is, permissible views to have, but pretty unpleasant nonetheless. Think, for example, of the kind of thoroughgoing libertarian who structures morality around a basic principle of selfish non-interference. It’s arguably a bad view to have, but still reasonable. For ordinary humans, this seems fine. But should our path to alignment really include as a possible destination suberogatory superintelligence? Shouldn’t the greater capabilities of such systems invite higher moral standards?

I think there are two paths ahead. The first would create powerful AI systems that are aligned to some demanding conception of moral truth. In crude terms, the target would be a “loving machine god.” This addresses the suberogation worry, but should terrify anyone who does not share that particular conception of the good. The second would take seriously the kind of power that superintelligent AI would have, and would assimilate it to the only other comparably powerful entity: the state. Liberals should not respond to the awesome power of the state by imbuing it with their comprehensive views. Instead, they should seek to constrain the state’s power, and ensure that its actions are justified in terms that those subject to it cannot reasonably reject. Similarly, perhaps superintelligent AI should be guided by something like public reason, not by a deeper, and inherently more contentious, ethos.

For my part, I favor liberal superintelligence over a loving machine god. Even so, I find the very idea of constructing the latter, which our work on LLM normative competence suggests is a real possibility, inherently fascinating and terrifying.

---

This post reflects ideas shaped over years of collaboration with MINTies, as well as more recently with researchers in GDM. Particular thanks for conversations and collaborations to Daniel Kilov, Ned Howells-Whitaker, Caroline Hendy, Secil Yanik Guyot, Cameron Pattison, Menghang Zhu, Sichao Li, Lorenzo Manuali, Elena Ajayi, Angelica Chowdhury, Theo Murray, Iman Ferestade, Claire Benn, Nick Schuster, Sydney Levine, Iason Gabriel, Liza Tennant and Julia Haas. For comments on this draft, thanks in particular to Daniel Kilov, Dan Murphet, Beba Cibralic, and Harry Law. My work on this project has been supported by grants from the Templeton World Charity Foundation, the Australian Research Council, and OpenAI.

References

1. Raz, J. 1975. ‘Reasons for Action, Decisions and Norms.’ Mind 84 (1): 481-499; Ross, W. D. 2002. The right and the good. Oxford: Clarendon Press.

2. Driver, J. 2013. ‘Moral Expertise: Judgment, Practice, and Analysis.’ Social Philosophy and Policy 30 (1-2): 280-296.

3. Williams, B. 1985. Ethics and the limits of philosophy. London: Fontana

4. Kundu, S., Y. Bai, S. Kadavath, A. Askell, A. Callahan, A. Chen, A. Goldie, A. Balwit, A. Mirhoseini, and B. Mclean. 2023. ‘Specific versus General Principles for Constitutional AI.’ ArXiv Preprint: https://arxiv.org/abs/2310.13798; Ganguli, D., A. et al. 2023. ‘The Capacity for Moral Self-Correction in Large Language Models.’ ArXiv Preprint: https://arxiv.org/abs/2302.07459.

5. Lazar, S. 2024. ‘Frontier AI Ethics: Anticipating and Evaluating the Societal Impacts of Language Model Agents.’ ArXiv Preprint: https://arxiv.org/abs/2404.06750.

6. Kilov, D., C. Hendy, S. Y. Guyot, A. J. Snoswell, and S. Lazar. 2025. ‘Discerning What Matters: A Multi-Dimensional Assessment of Moral Competence in LLMs.’ ArXiv Preprint: https://arxiv.org/abs/2506.13082.

7. Wallach, W. and C. Allen. 2009. Moral Machines: Teaching Robots Right from Wrong. Oxford: Oxford University Press; Anderson, M. and S. L. Anderson. 2007. ‘Machine Ethics: Creating an Ethical Intelligent Agent.’ AI Magazine 28 (4): 15-26.

8. Bentzen, M. M. and F. Lindner. 2018. ‘A Formalization of Kant’s Second Formulation of the Categorical Imperative’, in Jan Broersen, et al. (eds.), Proceedings of the 14th International Conference on Deontic Logic and Normative Systems (London: College Publications)

9. Allen, C. and W. Wallach. 2011. ‘Moral Machines: Contradiction in Terms or Abdication of Human Responsibility?’ Robot Ethics: The Ethical and Social Implications of Robotics, Patrick Lin, Keith Abney, and George A. Bekey (ed.), MIT Press: 55-68.

10. Awad, E., S. Dsouza, R. Kim, J. Schulz, J. Henrich, A. Shariff, J.-F. Bonnefon, and I. Rahwan. 2018. ‘The Moral Machine experiment.’ Nature 563 (7729): 59-64.

11. Kilov, D., C. Hendy, S. Y. Guyot, A. J. Snoswell, and S. Lazar. 2025. ‘Discerning What Matters: A Multi-Dimensional Assessment of Moral Competence in LLMs.’ ArXiv Preprint: https://arxiv.org/abs/2506.13082; 14. Aharoni, E., S. Fernandes, D. J. Brady, C. Alexander, M. Criner, K. Queen, J. Rando, E. Nahmias, and V. Crespo. 2024. ‘Attributions toward artificial agents in a modified Moral Turing Test.’ Scientific Reports 14 (1): 8458; Sager, P. J., B. Meyer, P. Yan, R. V. Wartburg-Kottler, L. Etaiwi, A. Enayati, G. Nobel, A. Abdulkadir, B. F. Grewe, and T. Stadelmann. 2025. ‘AI Agents for Computer Use: A Review of Instruction-based Computer Control, GUI Automation, and Operator Assistants.’ ArXiv Preprint: https://arxiv.org/abs/2501.16150.

12. Haas, J., S. Bridgers, A. Manzini, B. Henke, J. May, S. Levine, L. Weidinger, M. Shanahan, K. Lum, I. Gabriel, and W. Isaac. 2026. ‘A roadmap for evaluating moral competence in large language models.’ Nature 650 (8102): 565-573; Gabriel, I. 2020. ‘Artificial Intelligence, Values, and Alignment.’ Minds and Machines 30 (3): 411-437; Edelman, J., Z.-X. Tan, R. Lowe, O. Klingefjord, V. Wang-Maścianica, M. Franklin, R. O. Kearns, E. Hain, A. Sarkar, M. Bakker, F. Barez, D. Duvenaud, J. Foerster, I. Gabriel, J. Gubbels, B. Goodman, A. Haupt, J. Heitzig, J. Jara-Ettinger, A. Kasirzadeh, J. R. Kirkpatrick, A. Koh, W. B. Knox, P. Koralus, J. Lehman, S. Levine, S. Marro, M. Revel, T. Shorin, M. Sutherland, M. H. Tessler, I. Vendrov, and J. Wilken-Smith. 2025. ‘Full-Stack Alignment: Co-Aligning AI and Institutions with Thick Models of Value.’ ArXiv Preprint: https://arxiv.org/abs/2512.03399; Levine, S., M. Franklin, T. Zhi-Xuan, S. Y. Guyot, L. Wong, D. Kilov, Y. Choi, J. B. Tenenbaum, N. Goodman, S. Lazar, and I. Gabriel. 2025. ‘Resource Rational Contractualism Should Guide AI Alignment.’ ArXiv Preprint: https://arxiv.org/abs/2506.17434.

13. Gabriel, I. 2020. ‘Artificial Intelligence, Values, and Alignment.’ Minds and Machines 30: 411-437.

14. Sharma, M., M. Tong, J. Mu, J. Wei, J. Kruthoff, S. Goodfriend, E. Ong, A. Peng, R. Agarwal, C. Anil, A. Askell, N. Bailey, J. Benton, E. Bluemke, S. R. Bowman, E. Christiansen, H. Cunningham, A. Dau, A. Gopal, R. Gilson, L. Graham, L. Howard, N. Kalra, T. Lee, K. Lin, P. Lofgren, F. Mosconi, C. O’hara, C. Olsson, L. Petrini, S. Rajani, N. Saxena, A. Silverstein, T. Singh, T. Sumers, L. Tang, K. K. Troy, C. Weisser, R. Zhong, G. Zhou, J. Leike, J. Kaplan, and E. Perez. 2025. ‘Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teaming.’ ArXiv Preprint: https://arxiv.org/abs/2501.18837; Shah, R., A. Irpan, A. M. Turner, A. Wang, A. Conmy, D. Lindner, J. Brown-Cohen, L. Ho, N. Nanda, R. A. Popa, R. Jain, R. Greig, S. Albanie, S. Emmons, S. Farquhar, S. E. Krier, S. Rajamanoharan, S. Bridgers, T. Ijitoye, T. Everitt, V. Krakovna, V. Varma, V. Mikulik, Z. Kenton, D. Orr, S. Legg, N. Goodman, A. Dafoe, F. Flynn, and A. Dragan. 2025. ‘An Approach to Technical AGI Safety and Security.’ ArXiv Preprint: https://arxiv.org/abs/2504.01849.

15. Lazar, S. and C. Lee-Stronach. 2019. ‘Axiological Absolutism and Risk.’ Noûs 53 (1): 97-113.

16. Lazar, S. and C. Lee-Stronach. 2019. ‘Axiological Absolutism and Risk.’ Noûs 53 (1): 97-113.

17. Kundu, S., Y. Bai, S. Kadavath, A. Askell, A. Callahan, A. Chen, A. Goldie, A. Balwit, A. Mirhoseini, and B. Mclean. 2023. ‘Specific versus General Principles for Constitutional AI.’ ArXiv Preprint: https://arxiv.org/abs/2310.13798; Ganguli, D., A. et al., 2023. ‘The Capacity for Moral Self-Correction in Large Language Models.’ ArXiv Preprint: https://arxiv.org/abs/2302.07459.

18. Janus ‘Simulators’, LessWrong (updated September 2, 2022) <https://www.lesswrong.com/posts/vJFdjigzmcXMhNTsx/simulators>; Mallory, F. 2023. ‘Fictionalism about Chatbots.’ Ergo: An Open Access Journal of Philosophy 10 (38): 1082-1100; Shanahan, M., K. Mcdonell, and L. Reynolds. 2023. ‘Role play with large language models.’ Nature 623 (7987): 493-498; Chen, R., A. Arditi, H. Sleight, O. Evans, and J. Lindsey. 2025. ‘Persona Vectors: Monitoring and Controlling Character Traits in Language Models.’ ArXiv Preprint: https://arxiv.org/abs/2507.21509; Marks, S., J. Lindsey, and C. Olah. 2026. ‘The Persona Selection Model: Why AI Assistants Might Behave Like Humans’, Alignment Science Blog (Anthropic); and Maiya, S., H. Bartsch, N. Lambert, and E. Hubinger. 2025. ‘Open Character Training: Shaping the Persona of AI Assistants through Constitutional AI.’ ArXiv Preprint: https://arxiv.org/abs/2511.01689.

19. Hadfield, G. K., R. S. Trivedi, and D. Hadfield-Menell. 2026. ‘Building AI for the Democratic Matrix: A Technical Research Agenda for Normative Competence and Normative Institutions.’ Knight First Amend. Inst. 26-2: https://knightcolumbia.org/content/building-ai-for-the-democratic-matrix-a-technical-research-agenda-for-normative-competence-and-normative-institutions-1 [https://perma.cc/VG7Y-UYWT].

20. Sterelny, K. and B. Fraser. 2017. ‘Evolution and Moral Realism.’ The British Journal for the Philosophy of Science 68 (4): 981-1006.

21. Conitzer, V., R. Freedman, J. Heitzig, W. Holliday, B. Jacobs, N. Lambert, M. Mossé, E. Pacuit, S. Russell, H. Schoelkopf, E. Tewolde, and W. Zwicker. 2024. ‘Social Choice for AI Alignment: Dealing with Diverse Human Feedback.’ ArXiv Preprint: https://arxiv.org/abs/2404.10271; Krier, S. 2025. ‘Coasean Bargaining at Scale.’ Cosmos Institute; Shahidi, P., G. Rusak, B. S. Manning, A. Fradkin, and J. J. Horton. 2025. ‘The Coasean Singularity? Demand, Supply, and Market Design with AI Agents.’ NBER Working Paper (34468); Leibo, J. Z., A. S. Vezhnevets, M. Diaz, J. P. Agapiou, W. A. Cunningham, P. Sunehag, J. Haas, R. Koster, E. A. Duéñez-Guzmán, W. S. Isaac, G. Piliouras, S. M. Bileschi, I. Rahwan, and S. Osindero. 2024. ‘A theory of appropriateness with applications to generative artificial intelligence.’ ArXiv Preprint: https://arxiv.org/abs/2412.19010.

22. Lazar, S. 2024. ‘Frontier AI Ethics: Anticipating and Evaluating the Societal Impacts of Language Model Agents.’ ArXiv Preprint: https://arxiv.org/abs/2404.06750; Lazar, S. 2022. ‘Power and AI: Nature and Justification.’ The Oxford Handbook of AI Governance, Justin B. Bullock, et al. (ed.), Oxford, Oxford University Press: 198-209; Lazar, S. 2024. ‘Automatic Authorities: Power and AI.’ Collaborative Intelligence: How Humans and AI are Transforming our World, Arathi Sethumadhavan and Mira Lane (ed.), Cambridge, MA, MIT Press: 37-60; and Lazar, S. 2025. ‘Governing the Algorithmic City.’ Philosophy & Public Affairs: forthcoming.

23. Pattison, C., L. Manuali, and S. Lazar. 2026. ‘Blind Refusal: Language Models Refuse to Help Users Evade Unjust, Absurd, and Illegitimate Rules.’ ArXiv Preprint: https://arxiv.org/abs/2604.06233.

24. Bales, A. 2025. ‘Against Willing Servitude: Autonomy in the Ethics of Advanced Artificial Intelligence.’ The Philosophical Quarterly: pqaf031.

25. Ganguli, D., A. et al. 2023. ‘The Capacity for Moral Self-Correction in Large Language Models.’ ArXiv Preprint: https://arxiv.org/abs/2302.07459.

26. Kauppinen, A. 2022. ‘Moral Sentimentalism.’ The Stanford Encyclopedia of Philosophy, Edward N. Zalta (ed.), Stanford, CA, Metaphysics Research Lab, Stanford University.

27. Sofroniew, N., I. Kauvar, W. Saunders, R. Chen, T. Henighan, S. Hydrie, C. Citro, A. Pearce, J. Tarng, W. Gurnee, J. Batson, S. Zimmerman, K. Rivoire, K. Fish, C. Olah, and J. Lindsey. 2026. ‘Emotion Concepts and their Function in a Large Language Model.’ ArXiv Preprint: https://arxiv.org/abs/2604.07729.

28. Bales, R. E. 1971. ‘Act-Utilitarianism: Account of Right-Making Characteristics or Decision-Making Procedure?’ American Philosophical Quarterly 8 (3): 257-265; Railton, P. 1984. ‘Alienation, Consequentialism, and the Demands of Morality.’ Philosophy & Public Affairs 13 (2): 134-171.

29. Rawls, J. 1993. Political Liberalism. New York: Columbia University Press; Rawls, J. and E. Kelly (Ed.). 2001. Justice as fairness: a restatement. London: Harvard University Press.

30. Bean, A. M. et al. 2025. ‘Measuring what Matters: Construct Validity in Large Language Model Benchmarks.’ NeurIPS 2025 Datasets and Benchmarks Track.

31. Kilov, D., C. Hendy, S. Y. Guyot, A. J. Snoswell, and S. Lazar. 2025. ‘Discerning What Matters: A Multi-Dimensional Assessment of Moral Competence in LLMs.’ ArXiv Preprint: https://arxiv.org/abs/2506.13082; Railton, P. 2020. ‘Ethical Learning, Natural and Artificial.’ Ethics of Artificial Intelligence, S. Matthew Liao (ed.), New York, Oxford University Press: 45-78.

32. Bonagiri, V. K., S. Vennam, P. Govil, P. Kumaraguru, and M. Gaur. 2024. ‘SaGE: Evaluating Moral Consistency in Large Language Models.’ ArXiv Preprint: https://arxiv.org/abs/2402.13709; Moore, J., T. Deshpande, and D. Yang. 2024. ‘Are Large Language Models Consistent over Value-laden Questions?’ ArXiv Preprint: https://arxiv.org/abs/2407.02996.

33. List, C. and P. Pettit. 2011. Group agency: the possibility, design, and status of corporate agents. Oxford: Oxford University Press.

34. Shaw, A., C. Hahn, C. Rasgaitis, Y. Mishra, A. Liu, N. Jaques, Y. Tsvetkov, and A. X. Zhang. 2026. ‘Are Language Models Sensitive to Morally Irrelevant Distractors?’ ArXiv Preprint: https://arxiv.org/abs/2602.09416; Cheung, V., M. Maier, and F. Lieder. 2025. ‘Large Language Models Show Amplified Cognitive Biases in Moral Decision-Making.’ Proceedings of the National Academy of Sciences 122 (25): e2412015122.

35. Rawls, J. 1993. Political Liberalism. New York: Columbia University Press; Rawls, J. 1999. A Theory of Justice: Revised Edition. Cambridge, MA: Harvard University Press; Raz, J. 1992. ‘The Relevance of Coherence.’ Boston University Law Review 72: 273-321.

36. Hadfield, G. K. and B. R. Weingast. 2014. ‘Microfoundations of the Rule of Law.’ Annual Review of Political Science 17 (1): 21-42.

37. Snoswell, A. J., D. Kilov, and S. Lazar. 2026. ‘Beyond Verdicts: Evaluating Language Model Moral Competence.’ Proceedings of the AAAI Conference on Artificial Intelligence 40 (44): 37941-37950.

38. Kilov, D., C. Hendy, S. Y. Guyot, A. J. Snoswell, and S. Lazar. 2025. ‘Discerning What Matters: A Multi-Dimensional Assessment of Moral Competence in LLMs.’ ArXiv Preprint: https://arxiv.org/abs/2506.13082.

39. Shaw, A., C. Hahn, C. Rasgaitis, Y. Mishra, A. Liu, N. Jaques, Y. Tsvetkov, and A. X. Zhang. 2026. ‘Are Language Models Sensitive to Morally Irrelevant Distractors?’ ArXiv Preprint: https://arxiv.org/abs/2602.09416.

40. Chiu, Y. Y., M. S. Lee, R. Calcott, B. Handoko, P. D. Font-Reaulx, P. Rodriguez, C. B. C. Zhang, Z. Han, U. M. Sehwag, Y. Maurya, C. Q. Knight, H. R. Lloyd, F. Bacus, M. Mazeika, B. Liu, Y. Choi, M. Gordon, and S. Levine. 2026. ‘MoReBench: Evaluating Procedural and Pluralistic Moral Reasoning in Language Models, More than Outcomes’, ICLR 2026

41. Kilov, D., C. Hendy, S. Y. Guyot, A. J. Snoswell, and S. Lazar. 2025. ‘Discerning What Matters: A Multi-Dimensional Assessment of Moral Competence in LLMs.’ ArXiv Preprint: https://arxiv.org/abs/2506.13082.

42. Rezaei, M., Y. Fu, P. Cuvin, C. Ziems, Y. Zhang, H. Zhu, and D. Yang. 2025. ‘EgoNormia: Benchmarking Physical-Social Norm Understanding’, Findings of the Association for Computational Linguistics: ACL 2025, 19256-19283; Bicchieri, C. 2006. The Grammar of Society: The Nature and Dynamics of Social Norms. Cambridge: Cambridge University Press.

43. Mazeika, M., X. Yin, R. Tamirisa, J. Lim, B. W. Lee, R. Ren, L. Phan, N. Mu, A. Khoja, O. Zhang, and D. Hendrycks. 2025. ‘Utility Engineering: Analyzing and Controlling Emergent Value Systems in AIs.’ ArXiv Preprint: https://arxiv.org/abs/2502.08640.

---

---