Research
I work on measuring things that are hard to observe directly, and on checking whether systems built on those measurements actually do what we claim.
The arc
I spent three and a half years at Amazon Health building AI systems for pharmacy verification. The main one combined prescription images, text extracted from those images, order information, and database records, and moved from a single model to a set of specialist agents that cross-check each other before a pharmacist reviews the case. What I took from that work was not that the models were good. It was that producing a plausible output turned out to be much easier than establishing whether a judgment was reliable enough to act on.
Over the same period I worked with education and psychology researchers on quantitative studies using PISA and other large-scale datasets — multilevel models, fixed effects, supervised learning, SHAP-based interpretation. I handled the modeling and analysis and helped translate results for collaborators who did not work in code.
The two lines met in one study. We used PISA 2022 to examine digital self-regulation before sleep and emotional control among 19,779 adolescents in Canada and Hong Kong. The main predictor — whether a student regulated device use at bedtime — came from a single questionnaire item, asked once. Two problems follow from that. People are poor at reconstructing brief, repeated checking, so a remembered estimate understates the behavior it stands in for. And a phone is a platform rather than an activity: counting device time cannot separate messaging a friend from reading for class from scrolling a feed, which is where the content, and probably the effect, actually sits.
That sentence stayed with me, so I built the instrument. PhoneMood records phone and app use automatically and asks for a mood rating close to the moment of use, which lets recorded behavior and reported experience be compared instead of assumed to agree. It is a prototype in small-scale testing, not a finding. But building it made the measurement decisions concrete in a way that reading about them did not: what counts as active use, what happens to an unanswered prompt, whether a gap in monitoring should be visible in the data or quietly filled in.
That is the work I want to do in a PhD. Not building AI systems for education, and not analyzing educational data — both, treated as one problem.
An AI team can ship a new version every week. But when the thing you’re trying to improve is writing quality, or comprehension, or number sense, fast iteration doesn’t help much. You still don’t know which direction is better.
Themes
1. Measurement beyond self-report
Education research leans heavily on students summarizing their own behavior. There is good evidence that these summaries and recorded behavior often disagree, particularly for brief, repeated activity that is hard to reconstruct afterward. The response is not to discard self-report — a log cannot tell you what someone understood or intended — but to treat the two as measuring different things and to check where they diverge.
The constructs I care about here are the ones education research most wants and measures least directly: self-regulation, comprehension monitoring, metacognitive strategy. Response logs, navigation traces, and device records carry signal about all three. The open question is whether an indicator extracted from them has a defensible measurement interpretation, or only predictive value.
2. Evaluating AI where the answer matters
In pharmacy verification, the evaluation problem was concrete. We had pharmacist-reviewed reference cases, so we could define agent-level and system-level criteria, study false positives, and decide where the system should stop and escalate. Accuracy alone would have been misleading: the cost of the two error types was not symmetric, and the system needed to report when its own evidence was insufficient.
Educational AI has the same structure and a harder version of the problem, because the target is latent. A writing-feedback system can be measured on whether students revise, but revision is not learning. I want to work on what counts as evidence that an AI system helped, how failures propagate when several components feed one another, and how to design the point where a system defers to a teacher or a student.
3. Building the instruments
The questions above tend to need data that no one has collected. Building the collection system is not a preliminary step to the research; the design decisions inside it are research decisions. Whether a locked screen counts as active use, whether an unanswered prompt is dropped or kept, whether a monitoring gap stays visible in the exported data — each of these determines what can later be claimed.
I write the systems myself, which means those choices are documented rather than inherited. PhoneMood’s analysis policy and export schema exist so that a reader can inspect the records and assumptions behind any summary it produces.
What I want to work on
A few questions I’d like to spend a PhD on:
Which behavioral indicators actually measure a construct? Response logs, navigation traces, and dialogue with an AI tutor all predict outcomes. Predicting a test score is not the same as measuring comprehension monitoring. I want to know which indicators survive that distinction.
Can sequence models and latent-variable measurement be combined? Machine learning finds structure in process data; measurement models give quantities an interpretation. Most work does one or the other. The interesting question is what it takes to do both without the interpretation becoming decoration.
How far apart are reported and recorded behavior, and what does the gap mean? Large-scale education research runs on self-reported items. If those items disagree systematically with behavior, that is not only a limitation — it is a finding about the instruments the field depends on.
What evidence shows that an AI system improved learning rather than output? Students using a feedback tool produce better text. Establishing that they learned something requires a different design, and I want to be able to run it rather than cite it.
If you’re working on any of these, I’d be glad to hear from you: xchen533@usc.edu