PhD Research
My PhD, Measuring the Machine: Evaluating Generative AI as Pluralist Sociotechnical Systems, examined how generative AI systems are evaluated, measured, and made governable.
The thesis argues that AI evaluation is not neutral measurement. Evaluation helps shape what AI systems appear to be, what organisations optimise, and whose values become visible in practice. Benchmarks, prompts, datasets, metrics, and evaluation protocols do not simply observe model behaviour. They participate in producing the behaviour they measure.
Across five chapters and a coda, the thesis develops a pluralist, descriptive, and sociotechnical approach to generative AI evaluation. It brings together measurement theory, enactivism, moral value pluralism, sociotechnical systems thinking, and participatory realism to show how evaluation can reveal, rather than erase, cultural and value diversity.
Key contributions include MaSH Loops, a framework for understanding recursive Machine-Society-Human feedback, and the World Values Benchmark, a method for comparing generative AI behaviour against plural human value distributions.
The central question of the thesis is:
How can AI evaluation become a transparent, accountable and pluralist part of governance, rather than a hidden mechanism for reproducing narrow assumptions?
Read the full thesis on arXiv: https://arxiv.org/abs/2604.20545
PhD Highlights
- Enactivism and AI evaluation: Meaning and behaviour emerge through situated interaction, not from models alone.
- MaSH Loops (Machine, Society, Human): A framework for tracing how machines, societies, and humans recursively shape one another through data, design, use, and governance.
- The Model is not the Territory: A method for showing how AI systems simplify, represent, and reorganise the world they claim to model.
- Descriptive Evaluation and the World Values Benchmark: A pluralist method for comparing generative AI behaviour with human value distributions rather than reducing values to a single score.
- Participatory Realism and Measurement: An account of evaluation as an active measurement practice that helps produce the AI behaviour it claims to observe.
- Evaluation as Governance: The thesis coda, arguing that benchmarks, prompts, metrics, audits, and monitoring systems are part of how AI systems become accountable in practice.
1 • Enactivism and AI evaluation
Meaning and behaviour emerge through situated interaction, not from models alone.
Enactivism offers a way to understand AI systems without treating meaning as something simply stored inside a model. Meaning emerges through interaction: between model behaviour, prompt design, user interpretation, interface conditions, institutional context, and downstream use.
For evaluation, this matters because outputs are not simple windows into model capability. They are produced through configured encounters. What a system appears to know, value, or refuse depends on how it is prompted, situated, interpreted, and measured.
This framing moves evaluation away from the search for fixed internal properties and toward the study of enacted behaviour across real interactions..
This illustration expresses the enactivist view that knowledge and meaning are not passively received or stored, but actively brought forth through embodied engagement with the world. Mind and environment continually shape one another through dynamic loops of perception and action.
This approach builds on earlier ways of thinking—like functionalism, which looks at how systems work, and constructivism, which sees knowledge as socially shaped—but adds the dimension of lived participation. It reminds us that meaning arises not just from what machines do or represent, but from how people and systems act together. From this view, evaluation is not only about what a model contains, but about what it brings into being through its interaction with us.
“Functionalism privileges efficiency and performance; constructivism uncovers context and bias; enactivism asks how systems participate in meaning.”
— PhD thesis: Ch. 1 — Epistemological Rumbles in Responsible AI
2 • MaSH Loops (Machine – Society – Human)
Mapping feedback and co-construction across sociotechnical systems.
MaSH Loops map how machines, societies, and humans recursively shape one another. The framework treats generative AI as part of a sociotechnical system where design choices, training data, evaluation methods, user practices, institutional norms, and public narratives feed back into each other.
The point is not simply that AI affects society. It is that social values are already inside AI systems through data, design, benchmarks, deployment settings, and institutional priorities. Once deployed, systems then reshape human behaviour, organisational practice, and future data.
MaSH Loops make these recursive pathways visible, so evaluation can ask where values enter, how they move, and what forms of accountability are possible.
Meaning and value arise in the spaces where humans, machines, and societies interact and co-create.
The framework builds on the spirit of cybernetics and constructivism, while extending both through enactivism’s focus on participation. Functionalism reminds us that systems have structure; constructivism shows that structure is socially shaped; MaSH Loops brings them together through interaction, mapping how meaning circulates through the recursive ties of design, deployment, and interpretation.
“MaSH Loops—Machine, Society, Human—trace how models, people, and institutions recursively co-construct meaning and values.”
— PhD thesis: Ch. 2 — The Ghost in the Machine Has an American Accent
3 • Models, maps, and sociotechnical systems
Pedagogies for seeing how models make worlds.
My work uses sociotechnical mapping to examine how AI systems simplify, represent, and reorganise the world. Like maps, models highlight some relations and suppress others. Those choices shape what becomes visible, measurable, actionable, and governable.
Like maps, AI models highlight some features and omit others, shaping how we see and understand the world.
Sociotechnical mapping makes these choices explicit by tracing relations between people, data, models, institutions, prompts, interfaces, metrics, and consequences. It helps identify where assumptions enter, whose perspectives are missing, and how technical systems become embedded in organisational practice.
This method is especially useful for AI governance because many risks emerge through configuration and use, not through the model alone.
Adapted from sociotechnical systems theory, this framework illustrates how the evaluation of language models emerges from interactions between technical components and social contexts. Benchmark schemas, prompts, datasets, and metrics are shaped by underlying values and assumptions within broader social systems.
Through real-world case studies, this approach turns complex theory into lived insight. Students learn to see modelling as an interpretive act and to use mapping as both an analytical and ethical practice for designing and evaluating AI systems.
“The map is not the territory—but our maps decide which parts of the territory matter.”
— PhD thesis: Ch. 3 — The Model is Not the Market
4 • Descriptive evaluation and the World Values Benchmark
Developing new methods for measuring what AI enacts
The World Values Benchmark develops a descriptive approach to evaluating generative AI systems against plural human value distributions. Rather than asking whether a model gives the “right” answer, it asks which human value patterns its outputs most resemble under controlled prompting conditions.
The WVB links human data from the World Values Survey with AI model outputs to show how systems reflect global value patterns and how evaluation choices shape what AI appears to value.
The method links model outputs to World Values Survey data using balanced prompt design, paraphrase variation, calibrated anchors, Bayesian correction, and distributional comparison. This makes value representation more visible without collapsing pluralism into a single normative score.
The purpose is not to prescribe what AI should value. It is to show what evaluative choices make visible, what they conceal, and how model behaviour shifts across prompts, cultures, and measurement conditions.
“Evaluation should be descriptive, pluralist, and enactivist—it should reveal assumptions rather than conceal them.”
— PhD thesis: Ch. 4 — The World Values Benchmark
5 • Participatory realism and measurement
Understanding how observation and evaluation co-create meaning.
This area extends my work from systems and methods to the question of observation itself. Participatory realism builds on enactivism’s insight that knowing happens through interaction and adds a further idea: measurement is participatory. In both quantum physics and generative AI, observation does not simply reveal a pre-existing state—it helps bring one into being.
Just as observing a photon changes its pattern in the quantum experiment, prompting an AI helps shape the patterns it produces. In both cases, outcomes are not simply discovered—they are brought into being through interaction.
Generative models can be thought of as vast fields of potential meaning. A prompt acts like a measurement, turning possibility into a specific result. Each output reflects not only the model’s design and data but also the human questions, cultural assumptions, and interpretive context that shape the exchange.
Seen this way, evaluation becomes a kind of measurement: a meeting point between human intention and machine probability. Just as physics shows that the observer cannot stand outside the system, Responsible AI must recognise that our evaluations help shape what AI becomes.
Adapted from Bell’s theorem, this diagram contrasts two views of meaning in generative AI. The top path assumes fixed values that can be retrieved. The lower path reflects the enactivist view: meaning arises only through interaction, as each prompt collapses a field of possibilities into a single outcome..
Just as quantum measurement resolves potential into actuality, evaluation in generative AI selects from a range of possible meanings. There are no hidden variables determining an outcome in advance; each prompt is an experiment that helps define the system it probes. In this sense, responsible evaluation is less about revealing what a model is than about observing what emerges when human intention and machine probability meet.
“Evaluation, like observation in quantum mechanics, is a participatory act that helps bring outcomes into being.”
— PhD thesis: Ch. 5 — Semantic Auroras
6 • Evaluation as Governance
Designing measures that shape accountability
The Coda of my thesis argues that evaluation is not a peripheral task but a central mechanism through which AI systems are shaped and governed. Every benchmark, dataset, and metric carries normative assumptions that influence what becomes visible, comparable, and optimisable. Evaluation, therefore, functions as an instrument of governance; it determines how capability, alignment, and responsibility are defined in practice.
This diagram shows how evaluation operates as a governance system. Each stage—from designing benchmarks to reflecting on whose values are measured—forms a feedback loop linking machines, societies, and humans. The four orders of cybernetics capture escalating layers of accountability: behaviour, thinking, shared perception, and self-observation.
My ongoing research extends this insight toward the design of evaluative infrastructures: frameworks that integrate descriptive, pluralist, and enactivist approaches into policy and institutional processes. By treating measurement as part of governance design, we can make explicit whose values are being reinforced, where accountability resides, and how evaluation criteria evolve alongside the systems they assess.
“What we choose to measure determines what AI becomes in practice.”
— PhD thesis: Coda — Measuring What We Enact
“Evaluation is not a side activity—it is how we come to know ourselves in relation to the machines we make.”
— PhD thesis: Coda — Measuring What We Enact
From PhD research to current work
This thesis provides the foundation for my current work on generative and agentic AI evaluation. The core concern remains the same: evaluation is one of the ways AI systems become knowable, governable, and accountable.
My current research extends this argument from model outputs to deployed configurations: systems shaped by prompts, tools, memory, workflows, users, institutions, and downstream consequences.