Predict patient outcomes using self-supervision
Aim
In digital pathology, we have access to a lot of whole slide images. However, these images often lack annotations and therefore are discarded when training machine learning algorithms. The goal of his Ph.D. is to find a way to revalue these data through self-supervision. Self-supervision is a field of machine learning that aims at learning object representation without any labels. This can be achieved by using various tricks such as in-painting or data augmentation. As a result, we are capable to describe tissues from complex whole-slide images and predicting segmentation maps. The segmentation maps can be used to predict clinically relevant metrics such as tumor border configuration as depicted in the example Fig.1. Tumor border configuration gives an indication of the level of infiltration of the tumor which can be used to predict patient overall and disease-free survival.
Figure 1: Estimation of the tumor front. (a): Original H&E whole slide image. (b): Estimation of the primary tumor, muscle, and adipose areas. We make a first estimate of the borderline between healthy tissue and tumor (dashed line) as well as the estimation of the tumor front (yellow line). (c): Estimation of the pushing direction of the primary tumor area. (d): Local numerical estimation of the pushing (green) or infiltrating (orange) pattern along the tumor border.
Methods
Whole slide images are abundant in digital pathology but rarely come with annotations, which limits their use for training. We tackled this with self-supervised learning: instead of relying on manual labels, the model learns to describe tissue directly from the images themselves and adapts across the staining and scanner differences found between cohorts [1]. This yields a tissue segmentation model that can be applied to new whole slide images without any additional annotation. From the resulting segmentation maps, we then computed a set of clinically relevant metrics automatically: the tumor to stroma ratio (TSR), the tumor border configuration (TBC), the tumor to mucin ratio (TMR), and the distribution of stroma along the tumor border [2]. Each metric was first compared against expert pathologist annotations to confirm it was measured reliably, and then tested as a prognostic marker through univariate and multivariate survival analysis across five colorectal cancer cohorts (over 2,000 slides and 1,700 patients with overall and disease-free survival data).
Results
The self-supervised model produced accurate tissue segmentation, outperforming state-of-the-art domain-adaptation methods and approaching the accuracy of a fully supervised model trained with labels [1].
| Tissue segmentation (Kather19 → Kather16) | Weighted F1 |
|---|---|
| Source only (no adaptation) | 75.1% |
| SENTRY (best prior method) | 85.7% |
| SRA (our earlier method) | 86.9% |
| SRMA (ours) | 87.7% |
| Fully supervised (upper bound) | 93.0% |
Building on these segmentation maps, the automated metrics agreed well with pathologists' assessments, and the TSR, the TBC and the presence of stroma at the tumor border were all significant predictors of patient survival across the pooled cohorts (over 2,000 slides, 1,700 patients) [2]. Added to the standard clinical variables in a multivariate model, they improved the prediction of disease-free survival and matched it for overall survival, while being computed automatically, saving pathologists time and making these analyses feasible on large cohorts in a reproducible, unbiased way.
| Multivariate survival (all cohorts, C-index) | Clinical only | + Automated metrics |
|---|---|---|
| Overall survival | 0.690 | 0.689 |
| Disease-free survival | 0.721 | 0.732 |
Members
Christian Abbet
Inti Zlobec
Jean-Philippe Thiran
Funding source
Publications
[1] Abbet C, Studer L, Fischer A, Dawson H, Zlobec I, Bozorgtabar B, Thiran J-P. Self-rule to multi-adapt: Generalized multi-source feature learning using unsupervised domain adaptation for colorectal cancer tissue detection. Medical Image Analysis, vol. 79, 102473, 2022. https://doi.org/10.1016/j.media.2022.102473
[2] Abbet C, Studer L, Zlobec I, Thiran J-P. Toward Automatic Tumor-Stroma Ratio Assessment for Survival Analysis in Colorectal Cancer. Medical Imaging with Deep Learning (MIDL), Short Paper Track, 2022. https://openreview.net/forum?id=PMQZGFtItHJ
Collaboration