Dr. Rebecca L. Johnson
PhD in AI Evaluation, Ethics, and Governance
MA (Research), B.Sc., B.A.
The University of Sydney, Australia.
I am an AI evaluation and governance expert focused on how generative and agentic AI systems behave in real-world contexts. My work examines how outputs are produced across prompts, tools, workflows and institutional settings, and how evaluation shapes what organisations measure, optimise and treat as acceptable behaviour.
I evaluate how AI systems actually behave in deployment, not just what benchmarks say.
Selected recognition
Listed among the 100 Brilliant Women in AI Ethics (Lighthouse3, 2020)
Former Google Research, Ethical AI
Lead Guest Editor, AI and Ethics (Springer) – AI Agents collection
Scholarships: MIT, Stanford, University of Sydney
Click here to connect with me.
Short Media Bio
Dr Rebecca Johnson is an AI evaluation and governance expert focused on how generative and agentic AI systems behave in real-world contexts. She holds a PhD from the University of Sydney and previously worked in Google Research’s Ethical AI team. Her work focuses on how AI systems behave in practice, how responsibility and accountability can be evaluated and made visible, and how benchmarks shape what organisations measure and optimise. She has developed new approaches to AI evaluation and advises organisations on how to evaluate, explain and remain accountable for AI systems.
Longer Media Bio
Dr Rebecca Johnson is an AI evaluation and governance expert focused on how generative and agentic AI systems behave in real-world contexts. She holds a PhD from the University of Sydney, along with a Bachelor of Science, a Bachelor of Communications, and a Master’s by Research. She previously worked in Google Research’s Ethical AI team and was listed among the 100 Brilliant Women in AI Ethics by Lighthouse3 in 2020. She has worked directly with frontier generative AI systems in research and evaluation contexts since 2021 and has designed new evaluation methods addressing benchmark instability, prompt sensitivity and real-world system behaviour.
Her work examines how AI systems behave in practice, how responsibility and accountability can be evaluated and made visible, and how benchmarks shape what organisations measure and optimise. She has developed new approaches to AI evaluation and critically examined the limitations of widely used benchmarks. During her doctorate, she received scholarships from MIT and Stanford to participate in leading AI-focused programs, and was awarded the University of Sydney Postgraduate Research Prize for Leadership.
Rebecca is Lead Guest Editor for Springer’s AI and Ethics special collection on AI agents. She has convened large-scale conferences and policy think tanks on generative AI and its governance, founded the global network PhD Students in AI Ethics, and contributed to expert review for the Australian eSafety Commissioner on generative AI regulation. Since 2016, she has taught Master’s and undergraduate courses across Science, Arts and Business at the University of Sydney. She advises organisations and institutions on how to evaluate, explain and remain accountable for AI systems in practice, and is a regular media commentator on AI and society.
Research Overview
Generative AI evaluation is not neutral measurement. It shapes what AI systems appear to be, what organisations optimise, and whose values become visible in practice.
I have been researching generative AI since 2019 and technology ethics since 2015. My work examines AI systems as sociotechnical systems whose behaviour emerges through interaction between models, prompts, users, tools and institutions. Rather than treating outputs as fixed indicators of capability, I focus on how they are produced, interpreted and amplified across real contexts.
Drawing on measurement theory, moral philosophy, cybernetics and participatory realism, this work bridges theory and practice to develop new approaches to AI evaluation and to critically examine the limitations of widely used benchmarks. The aim is to make evaluation a more transparent and accountable part of AI governance, providing practical tools for organisations and policymakers to improve decision-making, risk management and the deployment of generative AI.
Selected quotes from my PhD thesis
- “What we measure, we amplify.”
- “Evaluation is governance.”
- “Meaning in generative AI is not retrieved. It is enacted.”
- “The model is not the governance object. The deployed configuration is.”
- “AI does not remove judgement. It relocates it.”
- “Outputs are residues. To govern AI well, we need to understand the path that produced them.”
- “Models don’t merely describe reality; they participate in shaping it.”
- “Pluralist evaluation is not an optional add-on but a minimum condition for deploying generative AI responsibly in a value-diverse world.”
Research interests
Research interests
Bec’s research sits at the intersection of AI evaluation, governance, philosophy of technology, and sociotechnical systems. Her work combines conceptual analysis with methodological design to develop evaluation practices that are rigorous, transparent, and useful for organisations deploying generative and agentic AI.
AI evaluation and benchmark design
Developing methods to assess how AI systems behave in real-world contexts, and critically examining the limitations of widely used benchmarks.
Agentic AI governance
Studying how systems that use tools, memory, retrieval and workflows change risk, accountability and oversight.
Measurement theory and evaluation validity
Using measurement theory to ask what AI evaluations actually measure, what they miss, and how they shape organisational optimisation.
Moral value pluralism
Designing evaluation approaches that recognise cultural and value diversity rather than collapsing “alignment” into a single normative standard.
Sociotechnical systems and MaSH Loops
Mapping how models, institutions and humans recursively shape one another in deployment.
Prompted interaction and participatory realism
Examining how meaning and behaviour emerge through interaction, rather than being simply retrieved from the model.
Responsible AI governance
Building evaluation practices that make risks, values, assumptions and accountability structures visible in practice.
Scholarships & Awards
Research Training Program (RTP) Stipend Scholarship (3.5 years): A primary scholarship that provides a stipend for the duration of the PhD program. “This is a Commonwealth Government scholarship awarded to assist students of exceptional research potential with their studies.” Letter of offer from The University of Sydney.
Massachusetts Institute of Technology (MIT) Scholarship 2019: to attend EmTech and EmTech Next at the MIT Media Lab.
Massachusetts Institute of Technology (MIT) Scholarship 2020:to attend EmTech and EmTech Next. The conference was held virtually due to the pandemic; nevertheless, the insights from many business technology leaders were very enlightening.
Postgraduate Research Prize for Leadership 2022: Faculty of Science, The University of Sydney. Awarded to HDR students who take on additional responsibilities through leadership roles.
Stanford University Scholarship 2023: I received a generous scholarship from The Embedded Ethics Centre and the Human-AI Center (HAI) to attend the Embedded Ethics conference.
Paulette Isabel Jones Career Award 2025: Awarded to HDR students to support research and thesis examination.
Teaching
I have worked as a Unit Co-ordinator, Lecturer, and Tutor since 2016. I have taught and coordinated subjects at Master’s, Honours, and Undergraduate levels. Subjects include: Ethics in Science, Emerging Technology, Introduction to Philosophy of Science, Organisational Communications, Leadership in Business, and Media and Communications. I have written and developed entire unit outlines as well as numerous lesson plans and assessments. I have also taught new tutors how to teach both in person and on-line. I hold two certifications for higher education teaching courses. Feedback reports from my students consistently place my teaching at 4.6-4.9 out of 5.
Previous education
Master of Arts by Research, The University of Sydney, 2016-2018
Thesis: Direct Voice: Including the student voice in academic governance in a large, research-intensive university. The work used cybernetic theory to design and test new communication and organisational learning structures between research students and university governance. The project was supported by the Deputy Vice-Chancellor of Education and the Director of Graduate Research. Awarded without corrections and high distinctions from both examiners.
- “[T]his thesis makes an important contribution to the field of higher education studies and is a result of excellent research.” Examiner, Harvard University.
- “The thesis is well-written, and the arguments made in a nuanced way come to a variety of interesting conclusions.” Examiner, The University of South Australia.
Professional Development in Higher Education teaching
- Professional teaching development training in the Business School, a five-week program, 2018.
- Six-week Professional Teaching Development Program run by the Faculty of Arts, 2016.
- Two-day intensive run by the Deputy Vice Chancellor – Education portfolio, 2016.
Bachelor of Arts (Communications) The University of Calgary, 2014-2015
Graduated ‘With Distinction’ and a place on the Dean’s list. G.P.A. of 3.8 (out of 4). I completed two capstone theses during this degree; one looked at building a conceptual framework of complex systems theory combined with constitutive communication theory, and one explored the sociological impacts of how our online actions construct ‘digital terraforming’.
Bachelor of Science (Earth Science) Deakin University, 1997 – 2001
My primary areas of interest were planetary geology and science communications. Invited to the Golden Key Society for Academic Excellence. Distinction average. Spent three summers volunteering with Monash University’s paleontological dig site in Inverloch. Commenced an Honours degree at Melbourne University, working on geomorphological flow processes on Mars under a CO2 model; work conducted in collaboration with NASA.
University Engagement
HDR Connect – May 2019
I conceived, developed, and executed the largest higher degree by research (HDR) student-led event of this kind that the University of Sydney had seen in memorable history. By bringing 400 HDR students together with 100 industry partners, I created a network across all faculties and with representatives from many industry sectors. After securing a total budget of AUD$38,000 (~USD$25,000) from the Deputy Vice Chancellor and other funding sources, I brought together a team of 40 research students to create this event. I also connected with many external stakeholders from a variety of industries, including national government science bodies, state government departments, international consulting firms, and many other industry partners. The executive leadership, including the Deputy Vice-Chancellor Education, the Director of Graduate Research, the Deputy Vice-Chancellor Research, and the Academic Board, strongly supported this initiative. I received a personal letter of thanks from the Vice-Chancellor of the University on completion of the project and warm praise from many Senior Executive leaders. Two of the industry participants subsequently offered new scholarships to the University as a result of this event. Photographs of the day.
HDR Liaison Committee
After my Master’s by Research and many years as a student Councillor and advocate, I observed that the diversity of graduate researcher voices was not always adequately represented in governance due to a reliance on traditional majority vote structures. I decided to act on this problem and used the knowledge I developed in my previous work to build a novel format of student engagement. After months of planning and discussion with the Deputy Vice-Chancellor of Education and the Director of Graduate Research, I founded the HDR Liaison Committee with 40 research students. This body forms a conduit for a two-way dialogue between the graduate researcher cohort and the senior executives charged with developing policy and strategy for the HDR environment and experience. The largest social event I organised during my time as Chair of this body was a BBQ attended by +100 PhD students; here are the photos.
Elected Student Representative
- I served as an active Student Councillor for the University of Sydney’s postgraduate student representative association for five terms.
- The University Executive – Research and Education committee (five years).
- The Academic Board (three years).
- The University Executive – Research Committee (one year).
- The Academic Qualities Committee (two years).
- Supervisory Policy Review Working Party Committee.
- Sydney Operating Model reference group, student representative
- Learning Analytics Advisory Board.
- HDR+ grant review panel.
- Research student representative to the Faculty Board for Arts and Social Sciences (FASS) 2017
