AI-Assisted Research Design and Data Quality
- Question
- Can an AI assistant find the missing decisions, unclear definitions and contradictions in a real-world data request before anyone builds the dataset?
- Status
- Proposed research: Proposal in development
- Approach
- A set of real-world data requests reviewed three ways (expert reviewers, a structured template, an AI copilot) and scored against a reference list of problems
- Updated
I am developing a University of Florida seed-funding proposal to test whether an AI copilot can review researchers' data requests as well as expert reviewers and structured templates do. The idea is at an early stage and there are no results yet.
RequestGLP-1 receptor agonists and glycemic control
Identify adults with type 2 diabetes2 who started a GLP-1 receptor agonist1 between 2019 and 2023. Exclude anyone with prior insulin use3. Baseline HbA1c is the most recent value on record5; the outcome is HbA1c six months later6. Also include the adolescent endocrinology cohort4 so the sample is large enough.
- Missing decisionNew users only? No washout period or index-date rule is given, so people already on the drug can enter the cohort.Cohort entry
- Ambiguous definitionOne diagnosis code or two, over what lookback, and do medications or HbA1c count? Each choice selects different patients.Phenotype
- Undefined windowPrior to what, and how far back? Without a window the exclusion depends on how much history each patient happens to have.Temporal window
- Internal contradictionConflicts with the adult criterion above. One of the two has to change, and the request should say which.Inclusion criteria
- Potential errorIf "most recent" is evaluated at extraction time rather than at drug start, baseline can fall after treatment begins, and the result changes with each data refresh.Reproducibility
- ObservabilityLabs appear only when ordered and resulted inside this system. The request needs a window around six months and a plan for patients with no value.Outcome measurement
Synthetic example
Reviewing a data request before any data are pulled
The request and the review notes were written for this page. They do not come from a real study and are not the output of an AI system. Hover or focus a phrase to see its note.
Figure description
A short request asks for adults with type 2 diabetes who started a GLP-1 receptor agonist, excluding prior insulin users, with baseline and six-month HbA1c, and also asks to include an adolescent cohort. Six notes flag: no new-user or index-date rule; an undefined diabetes phenotype; no lookback window for prior insulin; a contradiction between the adult criterion and the adolescent cohort; a baseline value that could fall after treatment starts; and an outcome lab that is observed only when tested.
Work in this direction
Proposal in development
Seed-funding proposal
Proposed research
Designing a study that gives the same data requests to expert reviewers, a structured template and an AI copilot, then scores what each finds, what it misses and what it flags in error.
Exploratory
A shared vocabulary for request problems
Proposed research
Defining the categories a review should cover, from cohort entry and temporal windows to data-dictionary mismatches and observability, so that different reviews can be scored the same way.
Exploratory
A longer-term question
Proposed research
If the pilot is informative, a larger study could ask whether reviewed requests lead to fewer rebuilt datasets and more reproducible analyses.
Why this question
Most real-world data studies begin with a written request: who belongs in the cohort, which events define entry and exclusion, over what windows, and which variables to extract. An analyst then turns that request into code. When the request leaves a decision open, the analyst makes it, often without the investigator knowing. When two criteria conflict, one of them quietly wins. These choices change the cohort and sometimes the answer, and they are hard to see afterwards.
Careful review catches many of these problems before any data are pulled, but expert review is slow and depends on who is available. Structured templates help, yet a template cannot read a request for meaning. Large language models can read a request, follow its logic and ask questions, which makes them a candidate for a first-pass reviewer. Whether they find the problems that matter, and how often they raise problems that are not there, is an open question that needs a fair test.
What a review looks for
- Cohort definition
- Is the index event stated? Are new users separated from people already on treatment?
- Inclusion and exclusion
- Is each phenotype defined precisely enough to code, and do any two criteria contradict each other?
- Temporal windows
- Are lookback, washout and follow-up periods stated, and anchored to the right date?
- Data dictionaries
- Do the requested variables exist in the source data, with the meaning the request assumes?
- Observability
- Can the outcome be seen in these data at all, and what does a missing value mean?
- Quality assurance and reproducibility
- Will the same request produce the same dataset next month, and can someone else check it?
Planned evaluation
The proposal being developed would assemble a set of data requests, have expert reviewers and a structured template review each one, and run an AI copilot on the same requests. Each review would be scored against a reference list of problems for what it finds, what it misses and what it flags in error, and the comparison would report where each approach is weakest as well as where it is strongest. Nothing has been run yet; this page will report results when they exist.
Earlier work this builds on
This question grew out of earlier work on how study definitions become data. Studies of trial eligibility criteria showed how much a cohort depends on the exact wording of each criterion, and work on computable representations of eligibility criteria for Alzheimer’s disease trials examined how to express those criteria so a machine can apply them. Projects on ontology-based data integration and on validating real-world cancer endpoints dealt with the gap between what a variable is called and what it records.