Dataset for the Research2Clinics @ NeurIPS 2021 Paper: What Do You See in this Patient? Behavioral Testing of Clinical NLP Models

Last update: Sep 20, 2022

Overview

Behavioral Testing of Clinical NLP Models

This repository contains code for testing the behavior of clinical prediction models based on patient letters. For a detailed description of the testing framework see our paper What Do You See in this Patient? Behavioral Testing of Clinical NLP Models.

Usage

Install requirements: pip install -r requirements.txt

Run main.py, e.g. for diagnosis prediction test on gender, age and ethnicity:

python main.py 
    --test_set_path ./path_to_test_set
    --model_path bvanaken/CORe-clinical-diagnosis-prediction
    --task diagnosis
    --shift_keys gender,age,ethnicity
    --save_dir ./results
    --gpu False

Parameter	Description
test_set_path	Path to original test set file
model_path	Path to model or Huggingface model hub checkpoint
task	Current options: diagnosis, mortality
shift_keys	Which patient characteristics to test. Current options: age, gender, ethnicity, weight, intersectional (gender + ethnicity)
save_dir	Directory to save results, default: "./results"
gpu	Whether to use a gpu during inference or not, default: False

Using Non-Transformer models

The framework currently focuses on testing Transformer-based models. However, it is easy to extend it to any other prediction model. To do so, simply create a new class implementing the Predictor interface and add it to the TASK_MAP in main.py.

Cite

@inproceedings{vanAken2021,
  author    = {Betty van Aken and
               Sebastian Herrmann and
               Alexander Löser},
  title     = {What Do You See in this Patient? Behavioral Testing of Clinical NLP Models},
  booktitle = {Bridging the Gap: From Machine Learning Research to Clinical Practice, 
               Research2Clinics Workshop @ NeurIPS 2021},
  year      = {2021}
}

Dataset for the Research2Clinics @ NeurIPS 2021 Paper: What Do You See in this Patient? Behavioral Testing of Clinical NLP Models

Related tags

Overview

Behavioral Testing of Clinical NLP Models

Usage

Using Non-Transformer models

Cite

Owner

Betty van Aken

scikit-learn inspired API for CRFsuite

Deep learning operations reinvented (for pytorch, tensorflow, jax and others)

DI-HPC is an acceleration operator component for general algorithm modules in reinforcement learning algorithms

Flower classification model that classifies flowers in 10 classes made using transfer learning (~85% accuracy).

Bootstrapped Representation Learning on Graphs

PyTorch implementation of DARDet: A Dense Anchor-free Rotated Object Detector in Aerial Images

A different spin on dataclasses.

Pixel Consensus Voting for Panoptic Segmentation (CVPR 2020)

A font family with a great monospaced variant for programmers.

On Size-Oriented Long-Tailed Graph Classification of Graph Neural Networks

WORD: Revisiting Organs Segmentation in the Whole Abdominal Region

Powerful unsupervised domain adaptation method for dense retrieval.

Malware Bypass Research using Reinforcement Learning

Python implementation of 3D facial mesh exaggeration using the techniques described in the paper: Computational Caricaturization of Surfaces.

Code Repository for The Kaggle Book, Published by Packt Publishing

Real-time multi-object tracker using YOLO v5 and deep sort

[ICCV 2021 Oral] Just Ask: Learning to Answer Questions from Millions of Narrated Videos

Official implementation for the paper "Attentive Prototypes for Source-free Unsupervised Domain Adaptive 3D Object Detection"

PGPortfolio: Policy Gradient Portfolio, the source code of "A Deep Reinforcement Learning Framework for the Financial Portfolio Management Problem"(https://arxiv.org/pdf/1706.10059.pdf).

Geometric Sensitivity Decomposition