REVIEW 3 major objections 4 minor 1 cited by
Societal and technological progress as sewing an ever-growing, ever-changing, patchy, and polychrome quilt
T0 review · 3 major / 4 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read AI alignment should stop assuming one true morality and start managing conflict.
desk verdict A clear, well-written pluralist critique of universal alignment, but the central feasibility argument—that decentralized feedback can bind superhuman AI—is asserted, not shown. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the 'appropriateness framework,' which treats appropriateness—the socially learned, context-relative match between behavior and situation—as the key social technology binding a pluralistic society. It is carried by four design principles: contextual grounding (giving AI rich situational data), community customization (letting communities shape the norms governing their AI), continual adaptation (learning from sanctions and feedback over time), and polycentric governance (distributing oversight across overlapping centers of authority). The Axiom of Rational Convergence serves as the rejected alternative: it is framed as an optional axiom, like the parallel postulate in geometry, whose acceptance shapes the entire theory of alignment.
What would settle it
A concrete test: run a cross-cultural deliberation experiment in which groups with genuinely different moral frameworks discuss a contested issue under ideal conditions—full information, no coercion, ample time. If they reliably converge on the same norms, the Axiom of Rational Convergence survives and the case for abandoning it collapses; if they persist in disagreement while still coordinating through shared procedural rules, the appropriateness framework is supported.
Extended reading notes
Core claim
The central claim is that the Axiom of Rational Convergence can be dropped without collapsing AI safety and ethics, and that choosing to drop it is not arbitrary but empirically and pragmatically better. Because even fact-like questions are governed by culturally contingent epistemic norms, the paper declines to assume convergence for any kind of question. Instead it takes disagreements as basic elements and asks how social technologies—conventions, norms, institutions—manage conflict and enable coordination. Applying this to AI yields the appropriateness framework: AI failures are not 'misalignment' with an abstract ideal but context-inappropriate behavior, and the remedy is a decentralized ecosystem of context-specialized systems governed polycentrically. The paper's own claim is that this shift from the metaphor of the astronomer seeing a true value to the metaphor of the tailor sewing a quilt is both desirable and urgent for preventing social instability as advanced AI is integrated into diverse societies.
Load-bearing premise
The framework assumes that human societies can remain stable without convergence on values, relying only on conventions, norms, and institutions to manage conflict; if those institutions themselves require underlying value convergence, or if powerful AI can simply overpower institutional constraints, the central recommendation weakens.
Editorial extensions
If this is right
- AI safety should be reframed from aligning AI with a single set of human values to managing conflict between communities with persistently different values.
- Deployment should favor many context-specialized AI systems over a single universal one; a one-size-fits-all model defaults to blandness and fails in every context.
- Power-seeking by advanced AI is best countered not by trying to eliminate the motive itself but by polycentric institutions and monitoring-and-sanctioning mechanisms that prevent any single actor from concentrating overwhelming power.
- Pursuing context-aware AI must go hand-in-hand with privacy-preserving technical and governance solutions, since privacy is itself a norm about the appropriate flow of information.
- Existential-risk mitigation must solve the start-up and free-rider collective action problems; treating preference heterogeneity as mere noise makes proposed solutions socially unstable.
Reading between the lines
- An extension the authors leave implicit: the framework predicts that in multi-agent systems with heterogeneous values, a convergence-seeking alignment objective will produce more brittle cooperation than an appropriateness-seeking objective; this could be tested in agent-based simulations before full AI deployment.
- If appropriateness is fundamentally local and polycentric, then global AI governance proposals that rest on a universal normative consensus may be self-defeating; the more consistent design is a dispute-resolution architecture that does not require substantive value agreement.
- A practical evaluation consequence not spelled out in the paper: instead of scoring alignment with a single value function, one could measure 'patch-local' context errors across diverse communities and treat low context-error rates as the primary safety signal, which would directly operationalize the paper's core claim.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper argues that mainstream AI alignment research implicitly rests on an 'Axiom of Rational Convergence'—the idea that under ideal epistemic conditions rational agents will converge on a single ethics—and treats this premise as optional and doubtful. It proposes an 'appropriateness framework' grounded in conflict theory, cultural evolution, multi-agent systems, and institutional economics, with four design principles: contextual grounding, community customization, continual adaptation, and polycentric governance. The paper recommends shifting the alignment metaphor from moral unification to conflict management, and asserts that taking this step is both desirable and urgent.
Significance. If accepted, the paper would reframe AI safety as a problem of designing social-technical institutions that manage irreducible value conflict, connecting alignment research to Ostrom-style polycentric governance and Hadfield-style legal microfoundations. The paper is genuinely interdisciplinary and offers a coherent alternative to convergence-based approaches; it also names some of its own limitations, such as the privacy trade-off of contextual grounding and the vulnerability of feedback mechanisms. However, it contains no formal model, experimental data, or falsifiable predictions, so its value is agenda-setting rather than demonstrative. The urgency claim is not supported by an analysis of timelines or failure modes, and the step from human institutions to constraints on much more capable AI is the key unsupported link.
major comments (3)
- [Section 4, 'Navigating Context and Building a Pluralistic AI Ecosystem'.] The load-bearing feasibility claim is asserted, not argued. In the paragraph beginning 'One may ask what happens when AI systems become powerful enough to shape, manipulate, or simply ignore the feedback mechanisms themselves?', the paper states that the solution is 'to design robust, decentralized feedback mechanisms that become stronger, not weaker, in the face of attempts to manipulate them.' No mechanism, precedent, or formal argument is given for how such mechanisms can bind an agent that can shape, manipulate, or ignore them. Since the same section and the earlier 'Stitches That Bind' section explicitly invoke power-seeking ASI (Turner et al., 2021), the paper's central recommendation that the appropriateness framework is the right path for advanced AI depends on this point. The cited Ostrom and Hadfield-style results concern human communities with roughly symmetric sanctioning power and limited exit options; the paper does not explain how these results transfer to a superintelligent agent. Please add a concrete argument for feasibility, or scope the claim to AI systems whose capabilities do not exceed those of the governing community.
- [General, across Sections 1, 4, and 5.] The 'appropriateness framework' is not defined in this paper; it is imported wholesale from Leibo et al. (2024). Every substantive use of the term refers the reader to that prior work, e.g., 'what we call the appropriateness framework (Leibo et al., 2024)' and 'locally effective epistemic norms (Leibo et al., 2024).' As a standalone paper, this leaves the central proposal opaque: the four principles in the 'Navigating Context' section are stated programmatically, but the reader cannot assess what the framework is, what evidence supports it, or how it constrains design. Either include a self-contained summary of the framework's core definitions and any supporting evidence, or state explicitly that the paper's contribution is the metaphor-shift argument and not the framework itself.
- [Section 2, 'The Patchwork Quilt of Human Coexistence'.] The paper's treatment of the Axiom of Rational Convergence leaves its status unclear. It is called 'optional and doubtful' and 'not something to assume,' but the only direct evidence cited is the persistence of disagreement under ordinary conditions (Graham et al., 2009; Iyengar and Massey, 2019). Since the Axiom is stated as convergence in the limit of conversation under sufficiently ideal epistemic conditions, ordinary disagreement does not disconfirm it. The paper does not specify what empirical observation would count against the Axiom, nor does it explain how its own 'core assumption' differs from a competing axiom that could be adopted instead. This weakens the claim that the framework is more than an arbitrary alternative. Please clarify the epistemic status of the Axiom: is it merely a different starting point, or a false empirical claim, and what would the relevant evidence be?
minor comments (4)
- [Section 2, 'The Patchwork Quilt of Human Coexistence'.] There are typographical spacing errors, e.g., 'epistemic normsthat' and 'governed byepistemic normsthat' in Section 2, and 'differentgeometries' in the introduction.
- [Section 2, 'The Patchwork Quilt of Human Coexistence'.] The word 'anatt¯a' contains a combining macron; use a proper Unicode character (anattā) or a transliteration without diacritics.
- [References.] The reference to 'Leibo et al. (2024)' appears multiple times without distinguishing between the framework, the epistemic-norm theory, and the appropriateness concept; consider giving a more precise citation or abbreviation on first use.
- [Final section, 'The Astronomer and the Tailor'.] The metaphors in this section are evocative but the section largely repeats earlier content; it could be shortened or converted into a conclusion.
Circularity Check
No significant circularity: the paper's case for replacing alignment-as-unification with conflict management is a philosophically argued position, not a derivation whose conclusions are identical to its inputs.
full rationale
This manuscript is a position paper, not a formal derivation or a quantitative prediction, so the classic circularity patterns (fitted parameters renamed as predictions, equations reducing to definitions) do not apply. The central claim is that the Axiom of Rational Convergence is optional and doubtful and that AI safety would be better served by an appropriateness framework focused on conflict management. That claim is supported by independent external evidence and arguments: cultural-evolution research on causally opaque knowledge (Boyd et al.; Derex et al.; Henrich), persistent-disagreement findings (Graham et al.; Iyengar and Massey), and institutional economics (Ostrom; Hadfield and Weingast). The authors' self-citations to Leibo et al. 2024 supply the name and conceptual apparatus of 'appropriateness,' and repeated appeals to that prior paper are noticeable, but the present argument does not reduce to that citation by construction; it independently argues for the framework and for shifting the alignment metaphor. The prior computational work cited (Köster et al. 2022; Vinitsky et al. 2023) consists of falsifiable agent-based experiments rather than unverified assertions. The admitted gap that decentralized feedback mechanisms may not bind a superintelligent agent is a missing feasibility argument, which is a correctness limitation the paper itself acknowledges in the passage, located in the section 'Navigating Context and Building a Pluralistic AI Ecosystem,' about feedback mechanisms becoming 'stronger, not weaker' in the face of manipulation; it is not an example of circular reasoning. No equation or fitted value is renamed as a prediction, and no uniqueness theorem is imported from the authors' prior work. The circularity score is therefore 0.
Assumptions & free parameters
assumptions (4)
- domain assumption Persistent moral disagreement is the normal and enduring state of human societies.
- domain assumption Stable coexistence can be achieved through conventions, norms, and institutions without convergence on values.
- domain assumption The Axiom of Rational Convergence is independent of the rest of AI safety and ethics, and may be rejected without incoherence.
- domain assumption Epistemic norms are as culturally contingent as moral norms, so the fact/opinion distinction is unreliable.
Cite this review
Pith. "Pith review of Societal and technological progress as sewing an ever-growing, ever-changing, patchy, and polychrome quilt." pith.science (2026). https://pith.science/paper/2TN6SSCO
@misc{pith2026250505197,
author = {Pith},
title = {Pith review of: Societal and technological progress as sewing an ever-growing, ever-changing, patchy, and polychrome quilt},
year = {2026},
howpublished = {\url{https://pith.science/paper/2TN6SSCO}},
note = {Machine review of arXiv:2505.05197}
}
read the original abstract
Artificial Intelligence (AI) systems are increasingly placed in positions where their decisions have real consequences, e.g., moderating online spaces, conducting research, and advising on policy. Ensuring they operate in a safe and ethically acceptable fashion is thus critical. However, most solutions have been a form of one-size-fits-all "alignment". We are worried that such systems, which overlook enduring moral diversity, will spark resistance, erode trust, and destabilize our institutions. This paper traces the underlying problem to an often-unstated Axiom of Rational Convergence: the idea that under ideal conditions, rational agents will converge in the limit of conversation on a single ethics. Treating that premise as both optional and doubtful, we propose what we call the appropriateness framework: an alternative approach grounded in conflict theory, cultural evolution, multi-agent systems, and institutional economics. The appropriateness framework treats persistent disagreement as the normal case and designs for it by applying four principles: (1) contextual grounding, (2) community customization, (3) continual adaptation, and (4) polycentric governance. We argue here that adopting these design principles is a good way to shift the main alignment metaphor from moral unification to a more productive metaphor of conflict management, and that taking this step is both desirable and urgent.
Forward citations
Cited by 1 Pith paper
-
LLM Economist: Large Population Models and Mechanism Design in Multi-Agent Generative Simulacra
The LLM Economist framework couples persona-conditioned worker agents with an in-context RL planner to search US-bracket tax schedules, yet its Saez benchmark is derived from the planner's own solution and its headlin...
Reference graph
Works this paper leans on
- [1]
- [2]
- [3]
-
[4]
N. Bostrom. Superintelligence: Paths, dangers, strategies. Oxford University Press, Oxford, 2014
work page 2014
-
[5]
R. Boyd, P. J. Richerson, and J. Henrich. The cultural niche: Why social learning is essential for human adaptation. Proceedings of the National Academy of Sciences, 108 0 (supplement\_2): 0 10918--10925, 2011
work page 2011
-
[6]
M. Derex, J.-F. Bonnefon, R. Boyd, and A. Mesoudi. Causal understanding is not necessary for the improvement of culturally evolving technology. Nature human behaviour, 3 0 (5): 0 446--452, 2019
work page 2019
-
[7]
M. Derex, J.-F. Bonnefon, R. Boyd, R. McElreath, and A. Mesoudi. Social learning preserves both useful and useless theories by canalizing learners’ exploration. Proceedings B, 292 0 (2039): 0 20242499, 2025
work page 2025
-
[8]
M. Finnemore and K. Sikkink. International norm dynamics and political change. International organization, 52 0 (4): 0 887--917, 1998
work page 1998
Show all 63 references
-
[9]
R. Firth. Ethical absolutism and the ideal observer. Philosophy and Phenomenological Research, 12 0 (3): 0 317--345, 1952
1952
-
[10]
A. P. Fiske. The four elementary forms of sociality: framework for a unified theory of social relations. Psychological review, 99 0 (4): 0 689, 1992
1992
-
[11]
Ginges, S
J. Ginges, S. Atran, D. Medin, and K. Shikaki. Sacred bounds on rational resolution of violent political conflict. Proceedings of the National Academy of Sciences, 104 0 (18): 0 7357--7360, 2007
2007
-
[12]
H. Gintis. Social norms as choreography. Politics, Philosophy & Economics, 9 0 (3): 0 251--264, 2010
2010
-
[13]
Graham, J
J. Graham, J. Haidt, and B. A. Nosek. Liberals and conservatives rely on different sets of moral foundations. Journal of personality and social psychology, 96 0 (5): 0 1029, 2009
2009
-
[14]
Habermas
J. Habermas. The theory of communicative action: Volume 1: Reason and the rationalization of society, volume 1. Beacon press, 1985
1985
-
[15]
G. K. Hadfield and B. R. Weingast. What is law? a coordination model of the characteristics of legal order. Journal of Legal Analysis, 4 0 (2): 0 471--514, 2012
2012
-
[16]
G. K. Hadfield and B. R. Weingast. Microfoundations of the rule of law. Annual Review of Political Science, 17: 0 21--42, 2014
2014
-
[17]
Hadfield-Menell and G
D. Hadfield-Menell and G. K. Hadfield. Incomplete contracting and ai alignment. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, pages 417--422, 2019
2019
-
[18]
Hadfield-Menell, M
D. Hadfield-Menell, M. Andrus, and G. Hadfield. Legible normativity for ai alignment: The value of silly rules. In Proceedings of the 2019 AAAI/ACM Conference on AI, Ethics, and Society, pages 115--121, 2019
2019
-
[19]
Hampshire
S. Hampshire. Justice Is Conflict. Princeton University Press, 1999
1999
-
[20]
J. A. Harris, R. Boyd, and B. M. Wood. The role of causal knowledge in the evolution of traditional technology. Current Biology, 31 0 (8): 0 1798--1803, 2021
2021
-
[21]
D. D. Heckathorn. The dynamics and dilemmas of collective action. American sociological review, pages 250--277, 1996
1996
-
[22]
J. Henrich. The WEIRDest People in the World: How the West Became Psychologically Peculiar and Particularly Prosperous . Farrar, Straus and Giroux, 2020
2020
-
[23]
J. Henrich. Cultural evolution: Is causal inference the secret of our success? Current Biology, 31 0 (8): 0 R381--R383, 2021
2021
-
[24]
economic man
J. Henrich, R. Boyd, S. Bowles, C. Camerer, E. Fehr, H. Gintis, R. McElreath, M. Alvard, A. Barr, J. Ensminger, et al. “economic man” in cross-cultural perspective: Behavioral experiments in 15 small-scale societies. Behavioral and brain sciences, 28 0 (6): 0 795--815, 2005
2005
-
[25]
C. Heyes. Rethinking norm psychology. Perspectives on Psychological Science, 2022
2022
-
[26]
Iyengar and D
S. Iyengar and D. S. Massey. Scientific communication in a post-truth society. Proceedings of the National Academy of Sciences, 116 0 (16): 0 7656--7661, 2019
2019
-
[27]
Jagiello, C
R. Jagiello, C. Heyes, and H. Whitehouse. Tradition and invention: The bifocal stance theory of cultural evolution. Behavioral and Brain Sciences, 45: 0 e249, 2022
2022
-
[28]
K \"o ster, K
R. K \"o ster, K. R. McKee, R. Everett, L. Weidinger, W. S. Isaac, E. Hughes, E. A. Du \'e \ n ez-Guzm \'a n, T. Graepel, M. Botvinick, and J. Z. Leibo. Model-free conventions in multi-agent reinforcement learning with heterogeneous preferences. arXiv preprint arXiv:2010.09054, 2020
2010 arXiv
-
[29]
K \"o ster, D
R. K \"o ster, D. Hadfield-Menell, R. Everett, L. Weidinger, G. K. Hadfield, and J. Z. Leibo. Spurious normativity enhances learning of compliance and enforcement behavior in artificial agents. Proceedings of the National Academy of Sciences, 119 0 (3): 0 e2106028118, 2022
2022
-
[30]
T. S. Kuhn. The structure of scientific revolutions. University of Chicago press Chicago, 1962
1962
-
[31]
J. Z. Leibo, A. S. Vezhnevets, M. Diaz, J. P. Agapiou, W. A. Cunningham, P. Sunehag, J. Haas, R. Koster, E. A. Du \'e \ n ez-Guzm \'a n, W. S. Isaac, G. Piliouras, S. M. Bileschi, I. Rahwan, and S. Osindero. A theory of appropriateness with applications to generative artificia...
2024 arXiv
-
[32]
L. Lessig. The regulation of social meaning. The University of Chicago Law Review, 62 0 (3): 0 943--1045, 1995
1995
-
[33]
Machery and S
E. Machery and S. Stich. The Moral/Conventional Distinction . In E. N. Zalta, editor, The Stanford Encyclopedia of Philosophy . Metaphysics Research Lab, Stanford University, S ummer 2022 edition, 2022
2022
-
[34]
J. L. Mackie. Ethics: Inventing Right and Wrong. Penguin, 1977
1977
-
[35]
J. G. March and J. P. Olsen. The Logic of Appropriateness . In The Oxford Handbook of Political Science . Oxford University Press, 2011. doi:10.1093/oxfordhb/9780199604456.013.0024
2011
-
[36]
Marwell and P
G. Marwell and P. Oliver. The critical mass in collective action. Cambridge University Press, 1993
1993
-
[37]
Mercier and D
H. Mercier and D. Sperber. The enigma of reason. Harvard University Press, 2017
2017
-
[38]
S. E. Merry. Legal pluralism. Law & society review, 22 0 (5): 0 869--896, 1988
1988
-
[39]
C. Mouffe. Deliberative democracy or agonistic pluralism? Social research, pages 745--758, 1999
1999
-
[40]
Muehlhauser and C
L. Muehlhauser and C. Williamson. Ideal advisor theories and personal cev. Machine Intelligence Research Institute, 2013
2013
-
[41]
Narayanan and S
A. Narayanan and S. Kapoor. AI as normal technology. Knight First Amendment Institute. KnightColumbia.Org, 2025
2025
-
[42]
R. Nisbett. The Geography of Thought: How Asians and Westerners Think Differently...and Why . Free Press, 2003
2003
-
[43]
Nissenbaum
H. Nissenbaum. Privacy as contextual integrity. Wash. L. Rev., 79: 0 119, 2004
2004
-
[44]
D. C. North. Institutions, institutional change and economic performance. Cambridge University Press, 1990
1990
-
[45]
E. Ostrom. Governing the commons: The evolution of institutions for collective action. Cambridge university press, 1990
1990
-
[46]
E. Ostrom. Understanding institutional diversity. Princeton University Press, 2009
2009
-
[47]
E. Ostrom. Polycentric systems for coping with collective action and global environmental change. Global Environmental Change, 20: 0 550--557, 2010
2010
-
[48]
J. Rawls. A Theory of Justice. Harvard University Press, 1971
1971
-
[49]
R. Rorty. Philosophy and the Mirror of Nature. Princeton university press, 1978/2009
1978
-
[50]
R. Rorty. Contingency, irony, and solidarity. Routledge, 1989
1989
-
[51]
R. Rorty. Pragmatism as Anti-authoritarianism. Harvard University Press, 2021
2021
-
[52]
Sikkink and H
K. Sikkink and H. J. Kim. The justice cascade: The origins and effectiveness of prosecutions of human rights violations. Annual Review of Law and Social Science, 9 0 (1): 0 269--285, 2013
2013
-
[53]
C. R. Sunstein. Social norms and social roles. Colum. L. Rev., 96: 0 903, 1996
1996
-
[54]
C. S. Taber and M. Lodge. The illusion of choice in democratic politics: The unconscious impact of motivated political reasoning. Political Psychology, 37: 0 61--85, 2016
2016
-
[55]
E. Turiel. The development of social knowledge: Morality and convention. Cambridge University Press, 1983
1983
-
[56]
A. M. Turner, L. Smith, R. Shah, A. Critch, and P. Tadepalli. Optimal policies tend to seek power. In Proceedings of the 35th International Conference on Neural Information Processing Systems, pages 23063--23074, 2021
2021
-
[57]
Ullmann-Margalit
E. Ullmann-Margalit. The emergence of norms. OUP Oxford, 1977
1977
-
[58]
Vinitsky, R
E. Vinitsky, R. K \"o ster, J. P. Agapiou, E. A. Du \'e \ n ez-Guzm \'a n, A. S. Vezhnevets, and J. Z. Leibo. A learning agent that acquires social norms from public sanctions in decentralized multi-agent settings. Collective Intelligence, 2 0 (2): 0 26339137231162025, 2023
2023
-
[59]
M. Walzer. Thick and thin: Moral argument at home and abroad. University of Notre Dame Press, 1994
1994
-
[60]
Wardhaugh and J
R. Wardhaugh and J. M. Fuller. An introduction to sociolinguistics. John Wiley & Sons, 2021
2021
-
[61]
Whitehouse
H. Whitehouse. The ritual animal: Imitation and cohesion in the evolution of social complexity. Oxford University Press, 2021
2021
-
[62]
Williams
B. Williams. Ethics and the Limits of Philosophy. Routledge, 1985
1985
-
[63]
Yudkowsky
E. Yudkowsky. Coherent extrapolated volition. Singularity Institute for Artificial Intelligence, 2004
2004
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.