TOXKAL
Reliable toxicity capture from routine documentation, across hospital boundaries.
In radiotherapy, how well a treatment is tolerated decides how far the dose may go. In German hospitals the entries this requires sit mostly as free text in progress and follow-up notes: which grade, at what point in time, brought on by which therapy. Dose-response models, and any comparison of care quality between hospitals, need them structured, graded and placed in time.
Language models can recover those entries from free text. The real obstacle lies elsewhere. A method that works well at one hospital is not automatically reliable at the next: different templates, different abbreviations, different documentation depth. Until now that could only be established after laborious manual re-annotation at the new site.
What we are investigating
TOXKAL develops methods that state, before deployment at an unfamiliar hospital and without manual work there, which extracted entries can be trusted and which must be put in front of a person. That includes a test which detects when the estimate itself no longer holds. A quietly wrong promise of reliability is more dangerous than none at all.
How we work
Several locally operated language models are adapted inside the hospital to its own documentation and set against one another. Their disagreement, together with measures of documentation practice, produces the reliability estimate. It is checked against physician-set reference ratings, following a procedure fixed in advance: the prediction is sealed before the reference data of the test site is opened.
Data protection is part of the architecture
Patient data does not leave the hospital. The analysis runs inside the hospital's own infrastructure, and only vetted aggregate measures flow back. For development and testing outside the hospital we use synthetic documentation.
What it is meant to produce
A tested method together with a product concept, with the aim of making it available as medical device software. We also intend to publish the methodological building blocks and a synthetic German-language test set, so other groups can run cross-site comparisons without moving patient data.
A negative result would be a result too. Should reliability turn out to stay site-bound, we would publish how much re-annotation each hospital needs at minimum.
Status
The project is in preparation. We are talking to hospitals that would like to take part as a validation site.