Investigating biomarker heterogeneity on colorectal cancer whole slide images by using tissue and cell types detected using Deep Learning
Aim
We are working on quantifying a variety of biomarkers on H&E-stained whole slide images of colorectal cancers by automatically detecting tissue- and cell types using deep learning models. In particular, we are interested in the distribution and heterogeneity of these biomarkers and the relationship of such heterogeneity with patient outcome, response, and other clinicopathological characteristics. Placing 2nd in the 2022 CoNiC Challenge, we, together with collaborators from Charité Berlin, developed a state-of-the-art nuclei segmentation and classification model with a variety of potential applications and are investigating its use cases.
Building on this model, we developed HoVer-NeXt [1], a faster and more accurate pipeline that detects, segments and classifies cell nuclei — including rare but informative types such as eosinophils and dividing (mitotic) cells — directly from routine H&E slides. As a first application, we used it to read the immune microenvironment of colorectal cancer, separating the tumor front from the tumor center to ask where different immune populations sit, how they relate to one another, and whether they carry prognostic value.
Figure 1: Examples of colorectal cancer whole slide images with overlaid nuclei detections. Colors represent associated classes.
Methods
HoVer-NeXt [1] pairs a ConvNeXt-v2 encoder with a streamlined U-Net architecture and an efficient inference engine, making it accurate on rare cell types yet fast enough to run across large cohorts — roughly 5× faster than CellViT and 17× faster than HoVer-Net on whole-slide inference. Because eosinophils and mitoses are rare and easily missed, we extended the public Lizard dataset with a dedicated mitosis class and released new expert-annotated validation sets for both.
We then applied the pipeline to a colorectal cancer cohort of 1,625 patients and 2,529 slides across four independent centres (Bern, Radboud, Toronto and TCGA) [2]. A tissue-segmentation model (SRMA) was used to locate the tumour and automatically estimate the invasive front, so that immune cells could be counted separately in the tumour front and tumour centre. For each tumour cell, surrounding immune cells within a fixed radius were counted and normalised by the number of tumour cells, yielding scores for eosinophils and lymphocytes in each region, as well as for intraepithelial lymphocytes (IELs). These scores were then correlated with clinicopathological features, molecular subtypes and patient outcome, using time to recurrence (TTR) as the survival endpoint.
Results
HoVer-NeXt matched or improved on its predecessors across nuclei detection and classification while running many times faster, with the largest gains on the rarest and most clinically relevant classes:
| Task | Metric | Result |
|---|---|---|
| Binary nuclei detection (GlaS) | F1 | 0.841 |
| Eosinophil classification (GlaS) | Balanced accuracy | 0.785 |
| Eosinophil detection (independent test set) | F1 | 0.69 |
| Mitosis detection (independent test set) | F1 | 0.62 |
| Whole-slide inference speed | vs. HoVer-Net / CellViT | 17× / 5× faster |
Applied to the four-centre colorectal cancer cohort, the pipeline showed that eosinophils in the tumour front (EosF) are an independent prognostic factor, with higher counts linked to more favourable outcomes (HR 0.70, 95% CI 0.55–0.90, p = 0.005), an effect that was especially pronounced in microsatellite-unstable (MSI) tumours (HR 0.32, 95% CI 0.14–0.74, p = 0.008). Crucially, eosinophils and intraepithelial lymphocytes carried statistically independent, additive prognostic information (EosF HR 0.71, p = 0.005; IELs HR 0.59, p = 0.047 in the same model), indicating they reflect two distinct arms of the anti-tumour response.
| Prognostic factor (tumour front) | Hazard ratio | 95% CI | p |
|---|---|---|---|
| Eosinophils (all stages) | 0.70 | 0.55–0.90 | 0.005 |
| Eosinophils (MSI cases) | 0.32 | 0.14–0.74 | 0.008 |
| Eosinophils + IELs | 0.71 / 0.59 | 0.55–0.90 / 0.35–0.99 | 0.005 / 0.047 |
In short, this work delivers a fast, fully automated way to read the immune landscape of colorectal cancer straight from H&E, and shows that eosinophils in the tumour front are an independent, automatically quantifiable prognostic marker, pointing towards an integrated immune score derived from routine stained sections.
Members
Elias Baumann
Inti Zlobec
María Rodríguez Martínez
Funding source
Publications
[1] Baumann, E., Dislich, B., Rumberger, J. L., Nagtegaal, I. D., Rodríguez Martínez, M. & Zlobec, I. HoVer-NeXt: A Fast Nuclei Segmentation and Classification Pipeline for Next Generation Histopathology. Proceedings of Machine Learning Research (MIDL 2024) (2024). https://openreview.net/forum?id=3vmB43oqIO
[2] Baumann, E., Lechner, S., Krebs, P., Kirsch, R., Berger, M. D., Lugli, A., Nagtegaal, I. D., Perren, A. & Zlobec, I. Deep learning-based quantification of eosinophils and lymphocytes shows complementary prognostic effects in colorectal cancer patients. npj Precision Oncology 9, 175 (2025). https://doi.org/10.1038/s41698-025-00955-0