About · Research · Publications · CV · News · Contact

The GerMedIQ Corpus

Key Findings

Resource: 4,524 simulated question-response pairs from 39 laypersons answering 116 German medical history questions.
Synthetic responses: 18 LLMs generated five responses each to the same questions.
Structural evaluation: Synthetic responses were generally longer and more complex than human-produced simulated responses in this study.
Semantic evaluation: Gemma 3 produced the responses closest to the human-produced responses under the study's semantic evaluation.
Acceptability evaluation: Human raters and LLM judges preferred Mistral (124B) responses under the study's evaluation setup.

These findings concern simulated medical interviews and the evaluated models and questions. They do not establish performance on real clinical documents or downstream clinical tasks.

Published Abstract

Due to strict privacy regulations, text corpora in non-English clinical contexts are scarce. Consequently, synthetic data generation using Large Language Models (LLMs) emerges as a promising strategy to address this data gap. To evaluate the ability of LLMs in generating synthetic data, we applied them to our novel German Medical Interview Questions Corpus (GerMedIQ), which consists of 4,524 unique, simulated question-response pairs in German. We augmented our corpus by prompting 18 different LLMs to generate responses to the same questions. Structural and semantic evaluations of the generated responses revealed that large-sized language models produced responses comparable to those provided by humans. Additionally, an LLM-as-a-judge study, combined with a human baseline experiment assessing response acceptability, demonstrated that human raters preferred the responses generated by Mistral (124B) over those produced by humans. Nonetheless, our findings indicate that using LLMs for data augmentation in non-English clinical contexts requires caution.

Resources

Citation

Please use the following citation to cite our paper:

@InProceedings{hofenbitzer2025germediq,
  author           = {Hofenbitzer, Justin and Sch{\"o}ning, Sebastian and Belle, Sebastian and Lammert, Jacqueline and Modersohn, Luise and Boeker, Martin and Frassinelli, Diego},
  booktitle        = {Proceedings of the 63rd {Annual} {Meeting} of the {Association} for {Computational} {Linguistics} ({Volume} 4: {Student} {Research} {Workshop})},
  title            = {{GerMedIQ}: {A} {Resource} for {Simulated} and {Synthesized} {Anamnesis} {Interview} {Responses} in {German}},
  year             = {2025},
  address          = {Vienna, Austria},
  editor           = {Zhao, Jin and Wang, Mingyang and Liu, Zhu},
  month            = jul,
  pages            = {1064--1078},
  publisher        = {Association for Computational Linguistics},
  doi              = {10.18653/v1/2025.acl-srw.84},
  isbn             = {9798891762541},
  url              = {https://aclanthology.org/2025.acl-srw.84/},
}

Please use the following citation to cite the GerMedIQ Corpus (the resource):

@Misc{hofenbitzer2025jhofenbitzer,
  author           = {Justin Hofenbitzer and Sch{\"o}ning, Sebastian and Belle, Sebastian and Lammert, Jacqueline and Modersohn, Luise and Boeker, Martin and Frassinelli, Diego},
  title            = {{Jhofenbitzer/GerMedIQ-Corpus: Official Github Repository of the GerMedIQ Corpus}},
  year             = {2025},
  copyright        = {Creative Commons Attribution 4.0 International},
  doi              = {10.5281/zenodo.16460622},
  publisher        = {Zenodo},
}

If you use the GerMedIQ Corpus, please cite both our paper and the resource itself!