BERT-based natural language processing analysis of French CT reports: Application to the measurement of the positivity rate for pulmonary embolism

Émilien Jupin-Delevaux, Aissam Djahnine, François Talbot, Antoine Richard, Sylvain Gouttard, Adeline Mansuy, Philippe Douek, Salim Si-Mohamed, Loïc Boussel

Research in Diagnostic and Interventional Imaging, March 2023, doi: 10.1016/j.redii.2023.100027

pdf url

Abstract

{Rationale and objectives To develop a Natural Language Processing (NLP) method based on Bidirectional Encoder Representations from Transformers (BERT) adapted to French CT reports and to evaluate its performance to calculate the diagnostic yield of CT in patients with clinical suspicion of pulmonary embolism (PE). Materials and methods All the CT reports performed in our institution in 2019 (99,510 reports, training and validation dataset) and 2018 (94,559 reports, testing dataset) were included after anonymization. Two BERT-based NLP sentence classifiers were trained on 27.700, manually labeled, sentences from the training dataset. The first one aimed to classify the reports’ sentences into three classes (“Non chest”, “Healthy chest”, and "Pathological chest" related sentences), the second one to classify the last class into eleven sub classes pathologies including "pulmonary embolism". F1-score was reported on the validation dataset. These NLP classifiers were then applied to requested CT reports for pulmonary embolism from the testing dataset. Sensitivity, specificity, and accuracy for detection of the presence of a pulmonary embolism were reported in comparison to human analysis of the reports. Results The F1-score for the 3-Classes and 11-SubClasses classifiers was 0.984 and 0.985, respectively. 4,042 examinations from the testing dataset were requested for pulmonary embolism of which 641 (15.8%) were positively evaluated by radiologists. The sensitivity, specificity, and accuracy of the NLP network for identifying pulmonary embolism in these reports were 98.2%, 99.3% and 99.1%, respectively. Conclusion BERT-based NLP sentences classifier enables the analysis of large databases of radiological reports to accurately determine the diagnostic yield of CT screening. }

Bibtex

@article{JUPINDELEVAUX2023100027,
  TITLE = {BERT-based natural language processing analysis of French CT reports: Application to the measurement of the positivity rate for pulmonary embolism},
  AUTHOR = {Émilien Jupin-Delevaux and Aissam Djahnine and François Talbot and Antoine Richard and Sylvain Gouttard and Adeline Mansuy and Philippe Douek and Salim Si-Mohamed and Loïc Boussel},
  URL = {https://www.sciencedirect.com/science/article/pii/S2772652523000066},
  JOURNAL = {Research in Diagnostic and Interventional Imaging},
  PUBLISHER = {ScienceDirect},
  YEAR = {2023},
  MONTH = {Mar},
  DOI = {10.1016/j.redii.2023.100027},
  KEYWORDS = {Natural language processing ; CT ; Pulmonary embolism},
  abstract = {{Rationale and objectives
To develop a Natural Language Processing (NLP) method based on Bidirectional Encoder Representations from Transformers (BERT) adapted to French CT reports and to evaluate its performance to calculate the diagnostic yield of CT in patients with clinical suspicion of pulmonary embolism (PE).
Materials and methods
All the CT reports performed in our institution in 2019 (99,510 reports, training and validation dataset) and 2018 (94,559 reports, testing dataset) were included after anonymization. Two BERT-based NLP sentence classifiers were trained on 27.700, manually labeled, sentences from the training dataset. The first one aimed to classify the reports’ sentences into three classes (“Non chest”, “Healthy chest”, and "Pathological chest" related sentences), the second one to classify the last class into eleven sub classes pathologies including "pulmonary embolism". F1-score was reported on the validation dataset. These NLP classifiers were then applied to requested CT reports for pulmonary embolism from the testing dataset. Sensitivity, specificity, and accuracy for detection of the presence of a pulmonary embolism were reported in comparison to human analysis of the reports.
Results
The F1-score for the 3-Classes and 11-SubClasses classifiers was 0.984 and 0.985, respectively. 4,042 examinations from the testing dataset were requested for pulmonary embolism of which 641 (15.8%) were positively evaluated by radiologists. The sensitivity, specificity, and accuracy of the NLP network for identifying pulmonary embolism in these reports were 98.2%, 99.3% and 99.1%, respectively.
Conclusion
BERT-based NLP sentences classifier enables the analysis of large databases of radiological reports to accurately determine the diagnostic yield of CT screening.
}}
}