Advertisement

Feasibility of using CT radiomic signatures for predicting CD8-T cell infiltration and PD-L1 expression in renal cell carcinoma

Open AccessPublished:September 02, 2022DOI:https://doi.org/10.1016/j.ejro.2022.100440

      Abstract

      Objectives

      To identify computed tomography (CT)-based radiomic signatures of cluster of differentiation 8 (CD8)-T cell infiltration and programmed cell death ligand 1 (PD-L1) expression levels in patients with clear-cell renal cell carcinoma (ccRCC).

      Methods

      Seventy-eight patients with pathologically confirmed localized ccRCC, preoperative multiphase CT and tumor resection specimens were enrolled in this retrospective study. Regions of interest (ROI) of the ccRCC volume were manually segmented from the CT images and processed using a radiomics panel comprising of 1708 metrics. The extracted metrics were used as inputs to three machine learning classifiers: Random Forest, AdaBoost, and ElasticNet to create radiomic signatures for CD8-T cell infiltration and PD-L1 expression, respectively.

      Results

      Using a cut-off of 80 lymphocytes per high power field, 59 % were classified to CD8 highly infiltrated tumors and 41 % were CD8 non highly infiltrated tumors, respectively. An ElasticNet classifier discriminated between these two groups of CD8-T cells with an AUC of 0.68 (95 % CI, 0.55–0.80). In addition, based on tumor proportion score with a cut-off of > 1 % tumor cells expressing PD-L1, 76 % were PD-L1 positive and 24 % were PD-L1 negative. An Adaboost classifier discriminated between PD-L1 positive and PD-L1 negative tumors with an AUC of 0.8 95 % CI: (0.66, 0.95). 3D radiomics metrics of graylevel co-occurrence matrix (GLCM) and graylevel run-length matrix (GLRLM) metrics drove the performance for CD8-Tcell and PD-L1 classification, respectively.

      Conclusions

      CT-radiomic signatures can differentiate tumors with high CD8-T cell infiltration with moderate accuracy and positive PD-L1 expression with good accuracy in ccRCC.

      Keywords

      1. Introduction

      The American Cancer Society (ACS) estimates 79,000 new renal cancers and 13,920 deaths due to renal cancers in the United States in 2022 [

      Key Statistics About Kidney Cancer. 〈https://www.cancer.org/cancer/kidney-cancer/about/key-statistics.html〉 (Accessed 20 August 2021).

      ]. Renal cell carcinoma (RCC) is the most common of renal cancer and constitutes over 90 % of all renal malignancies [
      • Siegel R.L.
      • Miller K.D.
      • Fuchs H.E.
      • Jemal A.
      Cancer statistics, 2022.
      ]. Clear Cell Renal Cell Carcinoma (ccRCC) is the most common type (70–80 %) of RCC. ccRCC is characterized by inactivation of the VHL gene function by deletion of chromosome 3p (Cancer Genome Atlas Research, 2013) leading to abnormal accumulation of hypoxia-inducible factors (HIF) and activation of the angiogenesis program [
      • Kaelin W.G.
      The von Hippel-Lindau tumor suppressor protein and clear cell renal carcinoma.
      ,
      • Majmundar A.J.
      • Wong W.J.
      • Simon M.C.
      Hypoxia-inducible factors and the response to hypoxic stress.
      ,
      • Semenza G.L.
      HIF-1 mediates metabolic responses to intratumoral hypoxia and oncogenic mutations.
      ]. ccRCC has one of the highest immune infiltration scores in pan-cancer analysis and high expression of immune checkpoints, such as PD-L1 and CTLA-4 [
      • Rooney C.M.
      • Leen A.M.
      • Vera J.F.
      • Heslop H.E.
      T lymphocytes targeting native receptors.
      ,
      • Şenbabaoğlu Y.
      • Gejman R.S.
      • Winer A.G.
      • Liu M.
      • Van Allen E.M.
      • de Velasco G.
      • Miao D.
      • Ostrovnaya I.
      • Drill E.
      • Luna A.
      • et al.
      Tumor immune microenvironment characterization in clear cell renal cell carcinoma identifies prognostic and immunotherapeutically relevant messenger RNA signatures.
      ]. Based on this unique hypervascular and hyperinflammed biology of ccRCC, vascular endothelial growth factor inhibitors (VEGFi) and immune checkpoint inhibitors (ICI) either alone or in combinations have resulted in significant improvement in clinical outcomes [
      • Choueiri T.K.
      • Motzer R.J.
      Systemic therapy for metastatic renal-cell carcinoma.
      ,
      • Motzer R.J.
      • Tannir N.M.
      • McDermott D.F.
      • Arén Frontera O.
      • Melichar B.
      • Choueiri T.K.
      • Plimack E.R.
      • Barthélémy P.
      • Porta C.
      • George S.
      • et al.
      Nivolumab plus Ipilimumab versus sunitinib in advanced renal-cell carcinoma.
      ,
      • Motzer R.J.
      • Robbins P.B.
      • Powles T.
      • Albiges L.
      • Haanen J.B.
      • Larkin J.
      • Mu X.J.
      • Ching K.A.
      • Uemura M.
      • Pal S.K.
      • et al.
      Avelumab plus axitinib versus sunitinib in advanced renal cell carcinoma: biomarker analysis of the phase 3 JAVELIN Renal 101 trial.
      ,
      • Rini B.I.
      • Wilding G.
      • Hudes G.
      • Stadler W.M.
      • Kim S.
      • Tarazi J.
      • Rosbrook B.
      • Trask P.C.
      • Wood L.
      • Dutcher J.P.
      Phase II study of axitinib in sorafenib-refractory metastatic renal cell carcinoma.
      ,
      • Rini B.I.
      • Escudier B.
      • Martini J.-F.
      • Magheli A.
      • Svedman C.
      • Lopatin M.
      • Knezevic D.
      • Goddard A.D.
      • Febbo P.G.
      • Li R.
      • et al.
      Validation of the 16-gene recurrence score in patients with locoregional, high-risk renal cell carcinoma from a phase III trial of adjuvant sunitinib.
      ,
      • Rini B.I.
      • Battle D.
      • Figlin R.A.
      • George D.J.
      • Hammers H.
      • Hutson T.
      • Jonasch E.
      • Joseph R.W.
      • McDermott D.F.
      • Motzer R.J.
      • et al.
      The society for immunotherapy of cancer consensus statement on immunotherapy for the treatment of advanced renal cell carcinoma (RCC).
      ,
      • Rini B.I.
      • Motzer R.J.
      • Powles T.
      • McDermott D.F.
      • Escudier B.
      • Donskov F.
      • Hawkins R.
      • Bracarda S.
      • Bedke J.
      • De Giorgi U.
      • et al.
      Atezolizumab plus Bevacizumab Versus Sunitinib for patients with untreated metastatic renal cell carcinoma and sarcomatoid features: a prespecified subgroup analysis of the IMmotion151 clinical trial.
      ]. ICI in particular have shown promising clinical efficacy with durable complete responses in a subset of patients and therefore are considered a new standard of care. Despite these developments, less than half of patients respond to these treatments, and these treatments are associated with significant toxicities. Thus, a better understanding of the molecular basis of clinical heterogeneity in patients with advanced RCC is needed to guide treatment selection strategies. Information regarding tumor microenvironment is instrumental in assessing the tumor response in relation to ICI [
      • Nishino M.
      • Ramaiya N.H.
      • Hatabu H.
      • Hodi F.S.
      Monitoring immune-checkpoint blockade: response evaluation and biomarker development.
      ]. For example, CD8+ T cell infiltration and PD-L1 expression has been associated with enhanced response to treatment [
      • Motzer R.J.
      • Tannir N.M.
      • McDermott D.F.
      • Arén Frontera O.
      • Melichar B.
      • Choueiri T.K.
      • Plimack E.R.
      • Barthélémy P.
      • Porta C.
      • George S.
      • et al.
      Nivolumab plus Ipilimumab versus sunitinib in advanced renal-cell carcinoma.
      ,
      • Tumeh P.C.
      • Harview C.L.
      • Yearley J.H.
      • Shintaku I.P.
      • Taylor E.J.M.
      • Robert L.
      • Chmielowski B.
      • Spasic M.
      • Henry G.
      • Ciobanu V.
      • et al.
      PD-1 blockade induces responses by inhibiting adaptive immune resistance.
      ]. However, these markers are not routinely tested in clinical practice due to challenges of pathologic specimen requirement (biopsy), tumor heterogeneity and sampling variability [
      • Nishino M.
      • Ramaiya N.H.
      • Hatabu H.
      • Hodi F.S.
      Monitoring immune-checkpoint blockade: response evaluation and biomarker development.
      ].
      Response rates to these immune checkpoint inhibitors like nivolumab range from 17 % to 48 % depending on disease state indicating a need for markers that may help guide treatment [
      • Zahoor H.
      • Duddalwar V.
      • D’Souza A.
      • Merseburger A.S.
      • Quinn D.I.
      What comes after immuno-oncology therapy for kidney cancer?.
      ,
      • Zahoor H.
      • Grivas P.
      The cancer immunogram: a pledge for a comprehensive biomarker approach for personalized immunotherapy in urothelial cancer.
      ]. Presence of CD8 T cells are connected to an improved treatment response in renal cell carcinoma [
      • Motzer R.J.
      • Tannir N.M.
      • McDermott D.F.
      • Arén Frontera O.
      • Melichar B.
      • Choueiri T.K.
      • Plimack E.R.
      • Barthélémy P.
      • Porta C.
      • George S.
      • et al.
      Nivolumab plus Ipilimumab versus sunitinib in advanced renal-cell carcinoma.
      ]. The inherent issues that arise when collecting this information include sampling variability when collecting a biopsy, tumor heterogeneity, reproducing results, and the need for a pathologic specimen [
      • Nishino M.
      • Ramaiya N.H.
      • Hatabu H.
      • Hodi F.S.
      Monitoring immune-checkpoint blockade: response evaluation and biomarker development.
      ]. It is clear a consistent standardized approach is required to gather this information accurately.
      Radiomics, the high-throughput quantification of tumor phenotypes from standard-of-care imaging data, is emerging as an effective tool for comprehensive assessment of tumor behavior [
      • Aerts H.J.W.L.
      • Velazquez E.R.
      • Leijenaar R.T.H.
      • Parmar C.
      • Grossmann P.
      • Carvalho S.
      • Bussink J.
      • Monshouwer R.
      • Haibe-Kains B.
      • Rietveld D.
      • et al.
      Decoding tumour phenotype by noninvasive imaging using a quantitative radiomics approach.
      ,
      • Varghese B.A.
      • Cen S.Y.
      • Hwang D.H.
      • Duddalwar V.A.
      Texture analysis of imaging: what radiologists need to know.
      ]. Radiomics may help in evaluation of a tumor’s microenvironment, its spatial heterogeneity, and longitudinal assessment, using imaging obtained as standard of care during treatment. The current approach to gather information about tumor phenotype is through biopsies, which are inherently invasive and have their own complications and are subject to sampling biases. Radiomics potentially allows tumor phenotype to be classified noninvasively, while assessing the entire tumor volume. We investigate and pilot the use of radiomics to help evaluate tumor microenvironment and tumor heterogeneity to identify radiomic signatures that correlate to CD8-T cell infiltration and PDL-1 expression in clear cell renal cell carcinoma (ccRCC) non-invasively. Identification of imaging correlates of immune markers such as CD-8 and PD-L1 are invaluable in guiding treatment choices, particularly when the immune expression is unknown either because the tissue sample is not of adequate quality for immunohistochemistry, or the patient cannot be subject to invasive tissue extraction procedures. Currently, little evidence exists in the literature that supports a correlation between spatial imaging heterogeneity (texture) and underlying mechanistic processes or biological heterogeneity, particularly in ccRCC. This is difficult and expensive task to undertake considering the amount of resources and expertize required. The Cancer Genome Atlas Kidney Renal Clear Cell Carcinoma (TCGA-KIRC) provide a good platform for exploring the correlations between tissue genotype, radiological phenotype, and genomic analysis in ccRCC. Sun et al., using 4 public datasets of cancer identified a radiogenomic marker for estimating CD8 cell count and predicting clinical outcomes of patients treated with immunotherapy [
      • Sun R.
      • Limkin E.J.
      • Vakalopoulou M.
      • Dercle L.
      • Champiat S.
      • Han S.R.
      • Verlingue L.
      • Brandao D.
      • Lancia A.
      • Ammari S.
      • et al.
      A radiomics approach to assess tumour-infiltrating CD8 cells and response to anti-PD-1 or anti-PD-L1 immunotherapy: an imaging biomarker, retrospective multicohort study.
      ]. While, the validation was performed in 3 different datasets, the heterogeneity in the acquisition of the tissue and/or images and their subsequent processing and analysis limit the widespread applicability of the findings. More recently, using pretreatment CT radiomics models constructed using data from 120 advanced-stage NSCLC patients, Wen et al. reported good performance for predicting PD-L1 and tumor mutational burden with AUCs of 0.73 and 0.76, respectively [
      • Wen Q.
      • Yang Z.
      • Dai H.
      • Feng A.
      • Li Q.
      Radiomics study for predicting the expression of PD-L1 and tumor mutation burden in non-small cell lung cancer based on CT images and clinicopathological features.
      ].
      Using a consistent cross-sectional imaging protocol with multiphase contrast-enhanced computed tomography (CECT) in 78 patients with biopsy proven (ccRCC) at our institution, we extract robust radiomic metrics and evaluate the potential of CT-based radiomic signature for predicting CD8- T cell infiltration and PD-L1 expression in ccRCC. We have further conducted sensitivity analyses using reliable (robust, reproducible, and repeatable) radiomic metrics which we have previously identified using imaging studies conducted on CT-radiomics phantoms. And finally, we elucidate the key radiomic metrics that drive the prediction of the best performing radiomics signature with a goal to understand the association of these metrics with immune response.

      2. Materials and methods

      2.1 Patient demographics

      In this IRB approved, HIPAA compliant study, 78 patients with localized RCC who had undergo nephrectomy at USC with multiphase clinical CT data pre-nephrectomy were chosen. There were 44 male patients and 34 female patients. Average age was 55.9 years old. Most patients were grade 2 and 3 (47 and 26 respectively, Table 1). 72.9 % of cases were in Tumor stage T1, and 21.7 % of cases were in Tumor stage 21.7 % (Table 1).
      Table 1ccRCC Patient Demographics.
      CharacteristicsSample Size
      Gender
      • Male
      44 (56.4 %)
      • Female
      34 (43.5 %)
      AgeMean: 55.9 years
      ISUP Gradea
      • 1
      3 (3.8 %)
      • 2
      47 (60.2 %)
      • 3
      26 (33.3 %)
      • 4
      2 (2.6 %)
      T Stage
      • T1a
      45 (57.6 %)
      • T1b
      12 (15.3 %)
      • T2a
      2 (2.6 %)
      • T2b
      1 (1.2 %)
      • T3a
      17 (21.7 %)
      • T3b
      1 (1.2 %)

      2.2 Inclusion and exclusion criteria

      Selection criteria included patients who had [

      Key Statistics About Kidney Cancer. 〈https://www.cancer.org/cancer/kidney-cancer/about/key-statistics.html〉 (Accessed 20 August 2021).

      ] pathologically confirmed clear cell RCC, [
      • Siegel R.L.
      • Miller K.D.
      • Fuchs H.E.
      • Jemal A.
      Cancer statistics, 2022.
      ] an available tissue specimen [
      • Kaelin W.G.
      The von Hippel-Lindau tumor suppressor protein and clear cell renal carcinoma.
      ] a preoperative multiphase CT performed at USC. Our final patient group consisted of 78 patients with pathologically confirmed ccRCC, and who had preoperative multiphase CT with available tumor tissue from June 2009–2018.

      2.3 CD8- T cell analysis

      The hematoxylin & eosin-stained slides of the case were reviewed by an experienced genitourinary pathologist. Sections of the tumor with the densest lymphocytic infiltrate were selected and stained with CD8 immunohistochemistry (IHC) (Fig. 1.) using Novocastra ready to use monoclonal antibody (clone 4B11) and Leica Bond III automated IHC platform. Antigen was retrieved by using Epitome Retrieval Solution 2 for 20 min. The IHC slides were examined for the area with highest lymphocyte density (Hot Spot). The hot spots were then pictured using Olympus BX53X, 40X objective and Olympus cellSens Standard software. Each image obtained was 335.1 µm and 262.9 µm in length and height, respectively. A 3 × 3 grid was overlaid on the entire image and the lymphocytes were counted using ‘Object counting’ function. A CD8 positive section was determined when there was greater than 80 lymphocytes per high power field [
      • Xiao B.
      • Peng J.
      • Zhang R.
      • Xu J.
      • Wang Y.
      • Fang Y.
      • Lin J.
      • Pan Z.
      • Wu X.
      Density of CD8+ lymphocytes in biopsy samples combined with the circulating lymphocyte ratio predicts pathologic complete response to chemoradiotherapy for rectal cancer.
      ]. Of the 78 patients, using a cut-off of 80 lymphocytes per high power field, 59 % were classified to CD8 highly infiltrated tumors and 41 % were CD8 non highly infiltrated tumors, respectively.
      Fig. 1
      Fig. 1IHC with antiCD8 monoclonal antibody clone 4B11 (B) of a grade 3 ccRCC (A).

      2.4 PD-L1 analysis

      PD-L1 staining was performed on a Leica Bond-III automated stainer using anti-PD-L1 monoclonal antibody clone 28–8 (Abcam) (Fig. 2). A tumor proportion score (tumor staining only) was used, with cut-off of ≥ 1 % tumor cells positive considered PD-L1 positive and others PD-L1 negative. The percentage of positive tumor cells ( % tumor cells positive/ total tumor cells) were calculated. Intensity of tumor staining was measured on a qualitative scale from 1 (weak) to 3 (strong). Only linear (partial or circumferential) membrane staining for PD-L1 was considered positive in tumor cells, whereas any staining (membrane or cytoplasmic) positive in immune cells. The presence of immune cells (IC) staining for PD-L1 (macrophages and lymphocytes) was noted and qualitatively assessed and categorized as none (not identified), rare (only focally present), few (several clusters), mod (frequent clusters), and many (most fields with clusters). Given the thin membranes in clear cell RCC, interpretation of linear membrane staining was challenging. Also, the prominent vasculature around tumor nests can occasionally demonstrate linear staining for PD-L1 along the edge of the tumor cells. It is controversial if this "vascular pattern" should be considered positive on tumor cells. In our study tumor cells positive for vascular pattern were counted as PD-L1 negatives.
      Fig. 2
      Fig. 2IHC with PD-L1 (clone 28–8, Abcam). Representative cases with 1 % (A), 20 % (B), and 80 % (C) staining are shown.

      2.5 Multiphase contrast-enhanced CT settings

      All scans were obtained with the same technique and with the same scanner (Brilliance 64, Philips Healthcare) at the University of Southern California. Imaging was performed with the patient performing a breath-hold and the following parameters: 120 kVp, variable tube current, slice thickness of 0.5 mm with reconstruction interval of 2 mm. An unenhanced CT scan of the abdomen was obtained first and followed by three contrast-enhanced scans obtained in the corticomedullary (30 s), nephrographic (90 s), and excretory (5–7 min) phases. Approximately 100–150 mL of nonionic water-soluble IV iodinated contrast medium (Iopamidol, Isovue 370, Bracco Imaging) dosed to weight was administered with a power injector at a rate of 5 mL/s.

      2.6 Tumor segmentation

      It is well known that an unrecognized renal tumor can be hidden behind a suspected diagnosis of pyelonephritis [
      • Ciccarese F.
      • Brandi N.
      • Corcioni B.
      • Golfieri R.
      • Gaudiano C.
      Complicated pyelonephritis associated with chronic renal stone disease.
      ]. Therefore, tumors and whole kidneys were manually segmented by 3 radiologists (>10years experience in abdominal CT) slice by slice using Synapse 3D software (Fujifilm Medical Systems) (Fig. 3). 3D regions of interest (ROIs) of the primary tumor were delineated from surrounding voxels in the nephrographic phase when available. The nephrographic phase was preferred as it provided the best delineation of the tumor and hence was used as the reference target for subsequent co-registration of other phases. Images were co-registered with the normalized mutual information cost function implemented in the Statistical Parametric Mapping software package (Wellcome Trust Centre for Neuroimaging). Custom Matlab (MathWorks) code was used to extract voxel data corresponding to the ROI. Two-dimensional CT based texture analysis (CTTA) was conducted in the orientation that provided the largest tumor diameter in each phase in the axial, coronal, or sagittal projection. Three-dimensional CTTA was conducted on the whole tumor volume.
      Fig. 3
      Fig. 3The workflow of radiomics. (a) Multiphase CT imaging; (b) Image segmentation was performed on contrast-enhanced CT images in the nephrographic phase. Experienced radiologists contour the tumor areas on all CT slices. The tumor contour on the nephrographic phase is projected on all other phases of co-registered CT volumes(c) Texture features are extracted from within the defined tumor regions, quantifying the distribution of tumor intensity and its spatial and higher order relationships (d) The last step of this process involves radiomic model building using machine learning classifiers.

      2.7 Contrast-enhanced CT texture analysis

      Texture analysis involves the study of the variation of pixel image intensity (Fig. 3). For both training and validation sets, algorithmic 13 parameter linear 3D co-registration was completed using the Statistical Parametric Mapping (SPM) software in MATLAB® [
      • Friston K.J.
      Statistical Parametric Mapping: the Analysis of Functional Brain Images.
      ]. Overall, 2824 radiomics features across 13 texture families were then extracted in MATLAB® using the custom data processing algorithms in the radiomics pipeline developed by our institution (Fig. 3). The techniques have been described previously in [
      • Zahoor H.
      • Grivas P.
      The cancer immunogram: a pledge for a comprehensive biomarker approach for personalized immunotherapy in urothelial cancer.
      ] the literature. To summarize, the 13 texture methods included: Intensity/Histogram analysis (3D), Gray Level Size Zone Matrix (GLSZM; 2D and 3D), Laws Texture Energy (LTE; 2D and 3D), Gray Level Run Length Matrix (GLRLM; 2D and 3D), Fast Fourier Transform (FFT; 2D), Gray Level Dependence Matrix (GLDM; 2D and 3D), Discrete Cosine Transform (DCT; 2D) and Gray Level Co-Occurrence Matrix (GLCM; 2D and 3D).

      2.8 Statistical analysis

      Following feature extraction, descriptive analyses were performed using Wilcoxon sum rank test or independent t-test with corrections for multiple comparisons by Benjamini and Hochberg. Machine-learning prediction models were constructed using ElasticNet, Random Forest (RF) and Real Adaptive Boosting (AdaBoost) [
      • Friedman J.
      • Hastie T.
      • Tibshirani R.
      Additive logistic regression: a statistical view of boosting (With discussion and a rejoinder by the authors).
      ]. For all 3 classifiers, 10-fold cross validation was used to evaluate model performance. The full dataset was equally divided into 10 folds. We re-iterated the learning process 10 times and applied the classifier to each of the testing sample. Thus, each study sample served as an independent testing case once. Receiver Operating Characteristic (ROC) curve was constructed using the predicted probability from 10 testing datasets combined and the area under the curve (AUC) with 95 % confidence interval was used to assess prediction accuracy. Within each iteration, we applied a 5-fold cross validation process to determine the final prediction model before scoring through the 10 % independent testing sample. The 10 % of independent testing data was excluded from learning phase to avoid information leaking. For Random Forest we have used 800 trees with maximal depth of 50, leaf size of 16 and variable to try was square root of variable number. For Adaboost, since it is more efficient, only 25 trees were built with a depth of 3 as recommended by Hastie et al., 2009 [
      • Hastie T.
      • Tibshirani R.
      • Friedman J.
      Random Forests.
      ]. For Random Forest and Adaboost, Gini impurity index was used as the loss function. We did not preselect subset features prior to cross-valuation in preventing information leaking from learning to testing. Applying the feature selection prior to the cross-validation showed a bias of up to 0.15 in AUCROC, 0.29 in AUC-F1 [
      • Demircioğlu A.
      Measuring the bias of incorrect application of feature selection when using cross-validation in radiomics.
      ], and 0.17 in Accuracy. Predicted residual sum of squares (CVPRESS) was used for ElasticNet to select candidate predictor as well as the final model. For imbalanced outcome, prior correction as described by King et.al. [
      • King G.
      • Zeng L.
      Logistic regression in rare events data.
      ] was used. Variable-of-importance (VOI) from Random Forest and Adaboost was selected and ranked using Out-of-bag Gini index (OOBGini), while ElasticNet was the remaining variables in the final model. The optimal cut-point was determined as the point maximizing the product of sensitivity and specificity [
      • Liu X.
      Classification accuracy and cut point selection.
      ]. The confusion matrix was generated based on the optimal cut-point.
      Differences in CT vendors and imaging protocols followed across different centers led to multicenter radiomic studies showing significant variability and non-convergence in results [
      • Da-Ano R.
      • Visvikis D.
      • Hatt M.
      Harmonization strategies for multicenter radiomics investigations.
      ,
      • Da-ano R.
      • Masson I.
      • Lucia F.
      • Doré M.
      • Robin P.
      • Alfieri J.
      • Rousseau C.
      • Mervoyer A.
      • Reinhold C.
      • Castelli J.
      • et al.
      Performance comparison of modified ComBat for harmonization of radiomic features for multicenter studies.
      ]. To alleviate this issue, we created a prediction model using only the robust metrics. The family of robust metrics was shortlisted from the entire radiomics panel based on series of imaging studies conducted using custom-built CT radiomics phantoms scanned on CT scanners from multiple vendors and imaging protocols [
      • Varghese B.A.
      • Hwang D.
      • Cen S.Y.
      • Levy J.
      • Liu D.
      • Lau C.
      • Rivas M.
      • Desai B.
      • Goodenough D.J.
      • Duddalwar V.A.
      Reliability of CT-based texture features: phantom study.
      ,
      • Varghese B.A.
      • Hwang D.
      • Cen S.Y.
      • Lei X.
      • Levy J.
      • Desai B.
      • Goodenough D.J.
      • Duddalwar V.A.
      Identification of robust and reproducible CT-texture metrics using a customized 3D-printed texture phantom.
      ]. Robustness was measured using the interclass correlation two-way-mixed with absolute agreement for single measurement (ICC 3.1) of each of the radiomics metrics across the four CT scanners [
      • Rini B.I.
      • Wilding G.
      • Hudes G.
      • Stadler W.M.
      • Kim S.
      • Tarazi J.
      • Rosbrook B.
      • Trask P.C.
      • Wood L.
      • Dutcher J.P.
      Phase II study of axitinib in sorafenib-refractory metastatic renal cell carcinoma.
      ]. Specifically, 60 % of features with ICC ≥ 0.70 were deemed robust. The families for the robust features were Intensity/Histogram analysis, GLCM 2D and 3D, GLDM 2D and 3D, GLRLM 2D and 3D, and GLSZM 2D and 3D. Creating prediction models based on the robust metrics only allowed us to more confidently select our variables of importance, which were ranked using Out-of-bag Gini index. SAS Enterprise Miner 15.1: High-Performance procedures were used for machine learning. SAS9.4 was used for all other statistical analysis.

      2.9 Interobserver reliability assessment and sensitivity analysis

      Segmentation variability in contouring the ROIs used for radiomics analysis affects both radiomic features and predictive accuracy [
      • Liu R.
      • Elhalawani H.
      • Radwan Mohamed A.S.
      • Elgohari B.
      • Court L.
      • Zhu H.
      • Fuller C.D.
      Stability analysis of CT radiomic features with respect to segmentation variation in oropharyngeal cancer.
      ]. We conducted an interobserver reliability analysis with three radiologists. As previously reported, each radiologist segmented the tumor margins independently for 15 subjects. Intraclass correlation (ICC) 2-way-mixed with absolute agreement was used to evaluate reliability of the radiomics results, as obtained using a RF classifier, despite the differences in segmentation contours.

      3. Results

      In general, radiomic features were consistent across the 3 radiologists with completed the tumor segmentation. The median and Q1, Q3 of ICC across all features were 0.88 (0.67, 0.97).
      Table 2, Table 3 showed the prediction accuracy when using all radiomic features (full model) in predicting CD8-T cell infiltration and PD-L1 expression respectively. The best accuracy for CD8-T is 0.62 95 % CI: (0.48, 0.76), for PD-L1 is 0.72 95 % CI: (0.58, 0.86) by RF.
      Table 2Prediction of CD8-T cell infiltration using full model based on a clinical cutoff of 80 lymphocytes per high power field. Based on this cutoff, of the 78 patients, 59 % were classified to CD8 high tumors and 41 % were CD8 low tumors, respectively.
      ModelAUC 95 % CI: (Range)
      Random Forest0.62 95 % CI: (0.48, 0.76)
      AdaBoost0.54 95 % CI: (0.4, 0.68)
      ElasticNet0.53 95 % CI: (0.39, 0.67)
      Table 3Prediction of PD-L1 expression using full model based on tumor proportion score with a cut-off of > 1 % tumor cells expressing PD-L1. Based on this cutoff, of the 78 patients, 76 % were PD-L1 positive and 24 % were PD-L1 negative, respectively.
      ModelAUC 95 % CI: (Range)
      Random Forest0.72 95 % CI: (0.58, 0.86)
      AdaBoost0.68 95 % CI: (0.53, 0.82)
      ElasticNet0.68 95 % CI: (0.53, 0.82)
      When using robust feature only, the best accuracy for CD8-T is 0.68 95 % CI: (0.55, 0.8) by ElasticNet (Table 4). The associated sensitivity, specificity, positive predictive value and negative predictive value were 0.61 95 % CI (0.46, 0.76), 0.64 95 % CI (0.47, 0.82), 0.71 95 % CI (0.56, 0.86) and 0.53 95 % CI (0.36, 0.7), respectively (Table 5).
      Table 4Prediction of CD-8T-cell infiltration and PD-L1 expression using robust models based on predefined clinical cutoffs. 0.8 (clinical) cutoff refers to the clinical cutoff of 80 lymphocytes per high power field for CD8- Tcell infiltration and 1 (clinical) refers to the clinical cutoff of tumor proportion score with a cut-off of > 1 % tumor cells expressing PD-L1.
      ExpressionModelAUC 95 % CI: (Range)Cut-Off
      CD8-Tcell infiltrationRandom Forest0.6 95 % CI: (0.46, 0.74)0.8 (clinical)
      CD8-Tcell infiltrationAdaBoost0.46 95 % CI: (0.32, 0.6)0.8 (clinical)
      CD8-Tcell infiltrationElasticNet0.68 95 % CI: (0.55, 0.8)0.8 (clinical)
      PD-L1 expressionRandom Forest0.78 95 % CI: (0.65, 0.92)1 (clinical)
      PD-L1 expressionAdaBoost0.8 95 % CI: (0.66, 0.95)1 (clinical)
      PD-L1 expressionElasticNet0.76 95 % CI: (0.63, 0.89)1 (clinical)
      Table 5Confusion matrices for the various radiomics signatures.
      MethodSensitivitySpecificityPPVNPVMarker
      Random Forest0.73 95 % CI (0.62, 0.84)0.74 95 % CI (0.54, 0.93)0.9 95 % CI (0.81, 0.98)0.47 95 % CI (0.29, 0.65)PD-L1
      Ada Boost0.71 95 % CI (0.6, 0.83)0.74 95 % CI (0.54, 0.93)0.89 95 % CI (0.81, 0.98)0.45 95 % CI (0.28, 0.63)PD-L1
      ElasticNet0.66 95 % CI (0.54, 0.78)0.68 95 % CI (0.48, 0.89)0.87 95 % CI (0.77, 0.97)0.39 95 % CI (0.23, 0.56)PD-L1
      Random Forest0.61 95 % CI (0.46, 0.76)0.64 95 % CI (0.47, 0.82)0.71 95 % CI (0.56, 0.86)0.53 95 % CI (0.36, 0.7)CD8
      Ada Boost0.46 95 % CI (0.31, 0.62)0.5 95 % CI (0.31, 0.69)0.58 95 % CI (0.41, 0.74)0.39 95 % CI (0.23, 0.55)CD8
      ElasticNet0.61 95 % CI (0.46, 0.76)0.64 95 % CI (0.47, 0.82)0.71 95 % CI (0.56, 0.86)0.53 95 % CI (0.36, 0.7)CD8
      For predicting PD-L1 the robust feature only model has the best AUC of 0.8 95 % CI: (0.66, 0.95) by Adaboost (Table 4). The associated sensitivity, specificity, positive predictive value and negative predictive value were 0.71 95 % CI (0.6, 0.83), 0.74 95 % CI (0.54, 0.93), 0.89 95 % CI (0.81, 0.98) and 0.45 95 % CI (0.28, 0.63), respectively (Table 5).
      VOI analysis revealed that 3D radiomic metrics of graylevel co-occurrence matrix (GLCM) played an important role in driving the predictive performance of the radiomics signature for CD8-T cell infiltration, particularly the difference average (difAve) or dissimilarity (Fig. 4).
      Fig. 4
      Fig. 4Top 10 VOI-based on ElasticNet model for prediction of CD8-Tcell infiltration. The format adopted to represent the radiomic metric is ‘texture family_image orientation_CECT phase_metric’. 3D analyses do not have an image orientation section. Here, GC2: Greylevel co-occurrence matrix (2D), GD3: Greylevel difference matrix (3D), LR2: Greylevel run-length matrix (2D), INT: Intensity (3D), GC3: Greylevel co-occurrence matrix (3D). A: Axial, S: Sagittal, C: Coronal, A: Corticomedullary, D: Excretory phase. OOBGini: Sum of out-of-bag Gini index times 1000 from 10-fold cross validation.
      VOI analysis also revealed that 3D radiomic metrics of graylevel run-length matrix (GLRLM) played an important role in driving the predictive performance of the radiomics signature for PD-L1 expression, particularly, LRLGE (Fig. 5).
      Fig. 5
      Fig. 5Top 10 VOI-based on AdaBoost model for prediction of PD-L1 expression. The format adopted to represent the radiomic metric is ‘texture family_image orientation_CECT phase_metric’. 3D analyses do not have an image orientation section. Here, GC2: Greylevel co-occurrence matrix (2D), GD3: Greylevel difference matrix (3D), GD2: Greylevel difference matrix (2D), LR3: Greylevel run-length matrix (3D), INT: Intensity (3D), A: Axial, S: Sagittal, C: Coronal, P: Noncontrast phase, A: Corticomedullary phase, D: Excretory phase. OOBGini: Sum of out-of-bag Gini index times 1000 from 10-fold cross validation.

      4. Discussion

      Immunotherapy is a becoming a mainstay in cancer treatment. Multiomic approaches at basic, preclinical, translational, and clinical levels have generated previously unknown relationships between immunogenic markers and other indicators of treatment response and clinical outcomes. However, the literature on radiomic studies in the context of assessing immune response is scant, with most focus on NSCLC. Using pretreatment contrast-enhanced CT imaging data from primary and metastatic lesions from 203 patients with advanced melanoma and non-small-cell lung cancer (NSCLC) undergoing anti-PD1 therapy, Trebeschi et al. reported that lesions with compact borders that displayed a more heterogeneous morphological and spatial texture phenotypes were associated with a better response to immunotherapy [
      • Trebeschi S.
      • Drago S.G.
      • Birkbak N.J.
      • Kurilova I.
      • Cǎlin A.M.
      • Delli Pizzi A.
      • Lalezari F.
      • Lambregts D.M.J.
      • Rohaan M.W.
      • Parmar C.
      • et al.
      Predicting response to cancer immunotherapy using noninvasive radiomic biomarkers.
      ]. Velcheti et al., demonstrated that changes in computer extracted features of vessel tortuosity on CT scans posttreatment could differentiate responders from non-responders in patients with NSCLC post nivolumab treatment with an AUC of 0.79 [
      • Velcheti V.
      • Alilou M.
      • Khunger M.
      • Thawani R.
      • Madabhushi A.
      Changes in computer extracted features of vessel tortuosity on CT scans post-treatment in responders compared to non-responders for non–small cell lung cancer on immunotherapy.
      ]. Similarly, Alilou et al., demonstrated that quantitative vessel tortuosity radiomics on baseline non-contrast lung CT predict response to immunotherapy and are prognostic of overall survival with an AUC of 0.73 using a small cohort of 111 patients [

      M. Alilou P. Vaidya M. Khorrami A. Zagouras P. Patil K. Bera P. Fu V. Velcheti A. Madabhushi. Madabhushi A. Quantitative vessel tortuosity radiomics on baseline non-contrast lung CT predict response to immunotherapy and are prognostic of overall survival, 2019, pp. 10950:109501F. 〈https://doi.org/10.1117/12.2513648〉.

      ]. Khorami et al., reported strong associations of changes in CT radiomic features with lymphocyte distribution and predicted response to immunotherapy in NSCLC with an AUC of 0.88 [
      • Khorrami M.
      • Prasanna P.
      • Gupta A.
      • Patil P.
      • Velu P.D.
      • Thawani R.
      • Corredor G.
      • Alilou M.
      • Bera K.
      • Fu P.
      • et al.
      Changes in CT radiomic features associated with lymphocyte distribution predict overall survival and response to immunotherapy in non-small cell lung cancer.
      ]. In a recent study, using 38 patients, Del Re et al., showed that PD-L1 baseline levels were significantly directly and inversely associated with CT radiomic features of NSCLC patients [
      • Del Re M.
      • Cucchiara F.
      • Rofi E.
      • Fontanelli L.
      • Petrini I.
      • Gri N.
      • Pasquini G.
      • Rizzo M.
      • Gabelloni M.
      • Belluomini L.
      • et al.
      A multiparametric approach to improve the prediction of response to immunotherapy in patients with metastatic NSCLC.
      ]. Considering, that a consensus regarding combining biomarkers to identify patients who will truly benefit from anti-immune agents in NSCLC is lacking, the study presented a multiparametric approach that with further validation may provide a better understanding of the molecular determinants of immunotherapy response.
      With the wider acceptance of immunotherapy across different caners, radiomic signatures predicting CD8-T cell infiltration and PD-L1 expression will be valuable tool to noninvasively and objectively assess treatment response. In this study, we used a whole-tumor volume machine-learning approach to develop radiomic signatures to quantitatively differentiate between patients showing CD8 high and CD8 low profiles and between patients showing PD-L1 positive and PD-L1 negative profiles in ccRCC. We report on CT-based radiomic signatures predict CD8-T cell infiltration with moderate discrimination (AUC = 0.68) and PD-L1 expression with good discrimination (AUC = 0.80) in ccRCC. Multiple studies have reported intra-scanner, inter-scanner, and multicenter variability in radiomics results [
      • Shafiq-ul-Hassan M.
      • Latifi K.
      • Zhang G.
      • Ullah G.
      • Gillies R.
      • Moros E.
      Voxel size and gray level normalization of CT radiomic features in lung cancer.
      ,
      • Mackin D.
      • Fave X.
      • Zhang L.
      • Fried D.
      • Yang J.
      • Taylor B.
      • Rodriguez-Rivera E.
      • Dodge C.
      • Jones A.K.
      • Court L.
      • Measuring C.T.
      scanner variability of radiomics features.
      ,
      • Mackin D.
      • Fave X.
      • Zhang L.
      • Yang J.
      • Jones A.K.
      • Ng C.S.
      • Court L.
      Harmonizing the pixel size in retrospective computed tomography radiomics studies.
      ,
      • Shafiq-Ul-Hassan M.
      • Zhang G.G.
      • Latifi K.
      • Ullah G.
      • Hunt D.C.
      • Balagurunathan Y.
      • Abdalah M.A.
      • Schabath M.B.
      • Goldgof D.G.
      • Mackin D.
      • et al.
      Intrinsic dependencies of CT radiomic features on voxel size and number of gray levels.
      ]. To increase reproducibility, we performed sensitivity analyses on the dependency of our result based on the following factors: [

      Key Statistics About Kidney Cancer. 〈https://www.cancer.org/cancer/kidney-cancer/about/key-statistics.html〉 (Accessed 20 August 2021).

      ] manual segmentation by multiple users, [
      • Siegel R.L.
      • Miller K.D.
      • Fuchs H.E.
      • Jemal A.
      Cancer statistics, 2022.
      ] choice of machine learning classifiers to create the radiomics signatures, and [
      • Kaelin W.G.
      The von Hippel-Lindau tumor suppressor protein and clear cell renal carcinoma.
      ] choice of radiomics panel (i.e., full vs. robust model). These are important steps in establishing the generalizability of radiomics-based decision algorithms for prospective applications in future validation studies.
      While histopathological evaluation is the current standard for evaluating immune markers, the approach is prone to sampling bias due to the discretely quantized nature of the tissue biopsy process. In comparison, radiomics can holistically evaluate whole-tumor volume heterogeneity at a single point in time, noninvasively and objectively. However, despite the improvements brought about by using radiomics, its application in the clinical workflow is hampered due to poor generalizability and reproducibility of results, particularly in multicenter studies using various scanners, scanning protocols and radiomic workflows. While in our study we use a single institution, single scanner protocol, to increase the generalizability of our results, we access the repeatability of our results on a robust model. The robust model only comprises of only CT-radiomic metrics that we previously reported on based on imaging studies conducted on accessing the reliability of radiomic metrics using a custom-built radiomics phantom. These metrics do not require harmonization. Results from our studies show consistent and, in some cases, slightly better performance of the robust models in comparison to the full model. This bolsters our expectation that during cross-validation, the robust features are much more likely to be retained as important predictors. The features with large difference between institutes and scanners would drop out since the unstable signals mostly cancel each other. Therefore, while conservative, our results are robust.
      In this study we employed three commonly used machine-learning augmented approaches, namely RF, AdaBoost and ElasticNet. RF and AdaBoost are considered non-parametric approaches while ElasticNet is considered parametric. For both classification tasks i.e., CD8 high vs. CD8 low and PD-L1 positive vs. PD-L1 negative, the performance between the different classifiers were comparable. Using a clinical cutoff of 80 lymphocytes per high power field, the ElasticNet based radiomic signature showed the highest discrimination between 46 CD8 high patients and 32 CD8 low patients with an AUC of 0.68 (95 % CI, 0.55–0.80). Also, based on tumor proportion score with a cut-off of > 1 % tumor cells expressing PD-L1, a radiomic signature of PD-L1 expression based on AdaBoost showed the highest discrimination between 59 PD-L1 positive patients and 19 PD-L1 negative patients with an AUC of 0.8 95 % CI: (0.66, 0.95). The closest comparison to our work is the radiomics approach to assess tumor-infiltrating CD8 cells and response to anti-PD-1 or anti-PD-L1 immunotherapy reported by Sun et al. In their retrospective multicohort study, they reported an AUC of 0.67; 95 % CI 0.57–0·77 to discriminate between CD8 high and CD8 low profile patients using a CT-based radiomics score. The signature was also able to discriminate inflamed tumors from immune-desert tumors (0·76; 0.66–0·86; p < 0·0001). As in their study, we also report discrimination between CD-8 high vs CD8-low patients with comparable performance, in single institution single protocol study. In addition, we also show good discrimination between PD-L1 positive and PD-L1 negative patients. Unlike Sun et al.’s, study, we use the radiomics metrics directly rather than creating a radiomics score, which helps us identify key radiomic variables driving the prediction performance, with a goal to understand the association of these metrics with immune response. Also, we focus on creating a radiomics only signature as opposed to a fusion of radiomics and sematic variables such as location of the ROI. We also construct robust predictive models which increase the reliability of our findings.
      Variable of importance (VOI) analysis revealed radiomic variables that drove the discriminative capability of the predictive radiomics models generated using ElasticNet and AdaBoost for CD8 and PD-L1 stratification, respectively. 3D radiomic metrics of graylevel co-occurrence matrix (GLCM) played an important role in driving the predictive performance of the radiomics signature for CD8-T cell infiltration, particularly the difference average (difAve) or dissimilarity. The dissimilarity metric captures the relationship between occurrences of pairs with similar and differing intensity values within the GLCM parametric map. High dissimilarity extracted from FDG-based PET images has been associated with poor overall survival and recurrence-free survival stage I–III NSCLC [
      • Lovinfosse P.
      • Janvary Z.L.
      • Coucke P.
      • Jodogne S.
      • Bernard C.
      • Hatt M.
      • Visvikis D.
      • Jansen N.
      • Duysinx B.
      • Hustinx R.
      FDG PET/CT texture analysis for predicting the outcome of lung cancer treated by stereotactic body radiation therapy.
      ,
      • Desseroit M.-C.
      • Visvikis D.
      • Tixier F.
      • Majdoub M.
      • Perdrisot R.
      • Guillevin R.
      • Cheze Le Rest C.
      • Hatt M.
      Development of a nomogram combining clinical staging with (18)F-FDG PET/CT image features in non-small-cell lung cancer stage I-III.
      ,
      • Hatt M.
      • Majdoub M.
      • Vallières M.
      • Tixier F.
      • Le Rest C.C.
      • Groheux D.
      • Hindié E.
      • Martineau A.
      • Pradier O.
      • Hustinx R.
      • et al.
      18F-FDG PET uptake characterization through texture analysis: investigating the complementary nature of heterogeneity and functional tumor volume in a multi-cancer site patient cohort.
      ]. 3D radiomic metrics of graylevel run-length matrix (GLRLM) played an important role in driving the predictive performance of the radiomics signature for PD-L1 expression, particularly, LRLGE. The LRLGE metric preferentially weights low graylevel values and long run lengths, therefore a high value will be observed in CT images with predominantly low attenuation values with a high frequency of occurrence. The results from our VOI analysis were similar to the predictive values of radiomics features for PD-L1 expression levels, as reported by Wen et al. [
      • Wen Q.
      • Yang Z.
      • Dai H.
      • Feng A.
      • Li Q.
      Radiomics study for predicting the expression of PD-L1 and tumor mutation burden in non-small cell lung cancer based on CT images and clinicopathological features.
      ]. In both studies, neighborhood-based texture assessment methods such as GLCM played a key role in driving the predictive power of the models.
      From our findings it is observed that texture metrics such as difAve and LRLGE can help differentiate between different levels of CD8-Tcell infiltration and PD-L1 expression, respectively. However, the reason for these metrics being significant compared to other texture metrics remains to be evaluated. In addition, the reasons for the differential performance of why the 3D versions of these metrics outperform the 2D versions also needs to be evaluated. In general, higher values of both metrics were observed at higher expression levels of both CD8 and PD-L1, which indicates higher spatial heterogeneity (textural differences with the ROI). Currently, little evidence exists in the literature that supports a correlation between spatial imaging heterogeneity (texture) and underlying mechanistic processes or biological heterogeneity. While our study identifies key radiomic metrics that strongly associate with immune markers, future studies with larges sample sizes and novel methodology such as cell network clustering and mapping with radiomics texture clustering are required to clarify the molecular mechanisms that are underplay connecting radiomic metrics and tumor microenvironments. Prior studies have shown that treatment ineffectiveness of cancer therapy is mainly due to the heterogeneous nature of the tumor microenvironment [
      • Reuben A.
      • Spencer C.N.
      • Prieto P.A.
      • Gopalakrishnan V.
      • Reddy S.M.
      • Miller J.P.
      • Mao X.
      • De Macedo M.P.
      • Chen J.
      • Song X.
      • et al.
      Genomic and immune heterogeneity are associated with differential responses to therapy in melanoma.
      ,
      • Rao S.-X.
      • Lambregts D.M.
      • Schnerr R.S.
      • Beckers R.C.
      • Maas M.
      • Albarello F.
      • Riedl R.G.
      • Dejong C.H.
      • Martens M.H.
      • Heijnen L.A.
      • et al.
      CT texture analysis in colorectal liver metastases: a better way than size and volume measurements to assess response to chemotherapy?.
      ]. However, currently, we do not have the inputs required to produce appropriate cancer models for ccRCC that would help us study the properties by which cancer is promoted and sustained. Various aspects may be involved causing the textural differences for example, there is a growing area of research exploring the interactions between malignant cells and other components of the tumor microenvironment including non-malignant cells and extracellular matrix (ECM) that contribute to metabolic heterogeneity and flexibility of metabolism between tumors and even within distinct regions of solid tumors [
      • Kim J.
      • DeBerardinis R.J.
      Mechanisms and implications of metabolic heterogeneity in cancer.
      ]. It may be possible that the metabolic heterogeneity may be related to the spatial heterogeneity seen in the multiphase CT images of the tumor. Future studies will help address these knowledge lacunae.
      In the past, statistical approaches such as ComBat were used to harmonize scans from different scanners and/or acquisition settings [
      • Orlhac F.
      • Boughdad S.
      • Philippe C.
      • Stalla-Bourdillon H.
      • Nioche C.
      • Champion L.
      • Soussan M.
      • Frouin F.
      • Frouin V.
      • Buvat I.
      A postreconstruction harmonization method for multicenter radiomic studies in PET.
      ]. A recent study showed that when scans have been harmonized, it may not preserve the original effect size. In some scenarios, it may introduce “unwanted effects”, which may exaggerate the prediction accuracy [
      • Cetin-Karayumak S.
      • Stegmayer K.
      • Walther S.
      • Szeszko P.R.
      • Crow T.
      • James A.
      • Keshavan M.
      • Kubicki M.
      • Rathi Y.
      Exploring the limits of ComBat method for multi-site diffusion MRI harmonization.
      ]. Further, not all radiomic features require harmonization. In our previously published radiomics reliability study using a radiomics phantom, we identified radiomic features that were robust without the necessity of harmonization [
      • Varghese B.A.
      • Hwang D.
      • Cen S.Y.
      • Levy J.
      • Liu D.
      • Lau C.
      • Rivas M.
      • Desai B.
      • Goodenough D.J.
      • Duddalwar V.A.
      Reliability of CT-based texture features: phantom study.
      ,
      • Varghese B.A.
      • Hwang D.
      • Cen S.Y.
      • Lei X.
      • Levy J.
      • Desai B.
      • Goodenough D.J.
      • Duddalwar V.A.
      Identification of robust and reproducible CT-texture metrics using a customized 3D-printed texture phantom.
      ]. During cross-validation, the robust features were much more likely to be retained as important predictors. The features with large difference between institutes and scanners would drop out since the unstable signals are likely to cancel each other [
      • Demirjian N.L.
      • Varghese B.A.
      • Cen S.Y.
      • Hwang D.H.
      • Aron M.
      • Siddiqui I.
      • Fields B.K.K.
      • Lei X.
      • Yap F.Y.
      • Rivas M.
      • et al.
      CT-based radiomics stratification of tumor grade and TNM stage of clear cell renal cell carcinoma.
      ].
      Our study had some limitations. Firstly, the sample size of 78 is small, but it is comparable to similar radiogenomic studies assessing immune response. With a dearth of such studies in ccRCC domain, our study shows the feasibility of using multiphase CT-radiomics signature to predict CD8-Tcell infiltration and PD-L1 expression, respectively. The rigorous cross-validation techniques in combination with use of the robust model provides a conservative estimate of performance with high reliability. Secondly, as we are a tertiary care center, therefore majority of lesions that are seen and treated by our institution is critical in nature. This may have contributed to the skewed distribution of samples in our patient cohort. Again, our rigorous cross-validation technique, increases our confidence in our results. Manual segmentation is time-consuming and raises concerns of inter-observer and intra-observer variability and validation; however, semi- or fully- automated segmentation approaches while rapid and unbiased are still being studied for applicability in different imaging modalities and cancers [
      • Parmar C.
      • Rios Velazquez E.
      • Leijenaar R.
      • Jermoumi M.
      • Carvalho S.
      • Mak R.H.
      • Mitra S.
      • Shankar B.U.
      • Kikinis R.
      • Haibe-Kains B.
      • et al.
      Robust Radiomics feature quantification using semiautomatic volumetric segmentation.
      ]. Also, such semi-or fully- automated segmentation errors may eventually introduce large errors into the calculation of radiomic features, leading to poor predictive models. In our study, the inter-observer segmentation reliability assessment between three radiologists showed that 65 % of radiomics features met ICC > 0.80, which represents consistent manual segmentations between the three radiologists.
      While the nephrographic phase provided the best delineation of the tumors when compared to other phases, there may be scenarios where all four phases of the CECT scans may not be available for analysis, due to factors including acquisition failure or poor scan quality due to motion artifact are typical of routine clinical scans. Therefore, developing an approach that is agnostic of a given phase (which may not be standardized across multiple centers) and rather based on tumor visibility is more practical in radiomics analyses in order to maximize sample size. Future studies including more balanced cohorts and larger sample sizes are warranted to evaluate suitability for clinical use. It was not our intention to establish an optimal cutoff to explore the performance of our prediction models. Therefore, we used clinical cutoffs for discriminating CD8 high vs. low and PD-L1 positive vs. negative patients. The field of radiogenomics is a potentially promising tool in constructing personalized cancer care, offering a novel non-invasive translational biomarker that can be used for molecular profiling of ccRCC. However, this field remains relatively immature, and the implementation of this technology in clinical practice is still a critical requirement [
      • Khaleel S.
      • Katims A.
      • Cumarasamy S.
      • Rosenzweig S.
      • Attalla K.
      • Hakimi A.A.
      • Mehrazin R.
      Radiogenomics in clear cell renal cell carcinoma: a review of the current status and future directions.
      ].
      To this end, in agreement with the literature, we report the feasibility of using multiphase CT-based radiomics signatures to predict CD8-Tcell infiltration and PD-L1 expression, respectively. Our findings suggest a promising future for the guiding of immunotherapy in ccRCC patients and deserve further in-depth study. Future studies should validate our results within large prospective validation cohorts to assess feasibility for clinical translation.

      Funding

      This work was supported by grants UL1TR001855 from the National Center for Advancing Translational Science (NCATS) of the U.S. National Institutes of Health and American Cancer Society (ACS) IRG –16-181-57 . The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health or the ACS.

      Ethics statement

      Informed consent was obtained for experimentation with human subjects. The privacy rights of human subjects was always observed. The study was IRB approved and HIPAA compliant.

      CRediT authorship contribution statement

      BV: Primary author, data processing, and analysis. SC: statistical design and analysis, HZ: Pathologist and performed PD-L1 analysis. IS: Pathologist and performed PD-L1 analysis. MA: Pathologist and performed CD8 analysis. AS: Overall guidance and provided the clinical insight and context to the study. SR: Overall guidance and provided the clinical insight and context to the study. XL: Overall guidance and provided the clinical insight and context to the study. MR: Overall guidance and provided the clinical insight and context to the study. DL: Overall guidance and provided the clinical insight and context to the study. DH: Overall guidance and provided the clinical insight and context to the study. DQ: Overall guidance and provided the clinical insight and context to the study. MD: Overall guidance and provided the clinical insight and context to the study. UV: Overall guidance and provided the clinical insight and context to the study. IG: Overall guidance and provided the clinical insight and context to the study. VD: Overall guidance and provided the clinical insight and context to the study. All authors reviewed the manuscript.

      Data Availability

      The code and extracted features generated for this study are available on request to the corresponding author.

      Declaration of Competing Interest

      The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

      References

      1. Key Statistics About Kidney Cancer. 〈https://www.cancer.org/cancer/kidney-cancer/about/key-statistics.html〉 (Accessed 20 August 2021).

        • Siegel R.L.
        • Miller K.D.
        • Fuchs H.E.
        • Jemal A.
        Cancer statistics, 2022.
        CA: A Cancer J. Clin. 2022; 72: 7-33https://doi.org/10.3322/caac.21708
        • Kaelin W.G.
        The von Hippel-Lindau tumor suppressor protein and clear cell renal carcinoma.
        Clin. Cancer Res. 2007; 13: 680s-684shttps://doi.org/10.1158/1078-0432.CCR-06-1865
        • Majmundar A.J.
        • Wong W.J.
        • Simon M.C.
        Hypoxia-inducible factors and the response to hypoxic stress.
        Mol. Cell. 2010; 40: 294-309https://doi.org/10.1016/j.molcel.2010.09.022
        • Semenza G.L.
        HIF-1 mediates metabolic responses to intratumoral hypoxia and oncogenic mutations.
        J. Clin. Investig. 2013; 123: 3664-3671https://doi.org/10.1172/JCI67230
        • Rooney C.M.
        • Leen A.M.
        • Vera J.F.
        • Heslop H.E.
        T lymphocytes targeting native receptors.
        Immunol. Rev. 2014; 257: 39-55https://doi.org/10.1111/imr.12133
        • Şenbabaoğlu Y.
        • Gejman R.S.
        • Winer A.G.
        • Liu M.
        • Van Allen E.M.
        • de Velasco G.
        • Miao D.
        • Ostrovnaya I.
        • Drill E.
        • Luna A.
        • et al.
        Tumor immune microenvironment characterization in clear cell renal cell carcinoma identifies prognostic and immunotherapeutically relevant messenger RNA signatures.
        Genome Biol. 2016; 17: 231https://doi.org/10.1186/s13059-016-1092-z
        • Choueiri T.K.
        • Motzer R.J.
        Systemic therapy for metastatic renal-cell carcinoma.
        N. Engl. J. Med. 2017; 376: 354-366https://doi.org/10.1056/NEJMra1601333
        • Motzer R.J.
        • Tannir N.M.
        • McDermott D.F.
        • Arén Frontera O.
        • Melichar B.
        • Choueiri T.K.
        • Plimack E.R.
        • Barthélémy P.
        • Porta C.
        • George S.
        • et al.
        Nivolumab plus Ipilimumab versus sunitinib in advanced renal-cell carcinoma.
        N. Engl. J. Med. 2018; 378: 1277-1290https://doi.org/10.1056/NEJMoa1712126
        • Motzer R.J.
        • Robbins P.B.
        • Powles T.
        • Albiges L.
        • Haanen J.B.
        • Larkin J.
        • Mu X.J.
        • Ching K.A.
        • Uemura M.
        • Pal S.K.
        • et al.
        Avelumab plus axitinib versus sunitinib in advanced renal cell carcinoma: biomarker analysis of the phase 3 JAVELIN Renal 101 trial.
        Nat. Med. 2020; 26: 1733-1741https://doi.org/10.1038/s41591-020-1044-8
        • Rini B.I.
        • Wilding G.
        • Hudes G.
        • Stadler W.M.
        • Kim S.
        • Tarazi J.
        • Rosbrook B.
        • Trask P.C.
        • Wood L.
        • Dutcher J.P.
        Phase II study of axitinib in sorafenib-refractory metastatic renal cell carcinoma.
        J. Clin. Oncol. 2009; 27: 4462-4468https://doi.org/10.1200/JCO.2008.21.7034
        • Rini B.I.
        • Escudier B.
        • Martini J.-F.
        • Magheli A.
        • Svedman C.
        • Lopatin M.
        • Knezevic D.
        • Goddard A.D.
        • Febbo P.G.
        • Li R.
        • et al.
        Validation of the 16-gene recurrence score in patients with locoregional, high-risk renal cell carcinoma from a phase III trial of adjuvant sunitinib.
        Clin. Cancer Res. 2018; 24: 4407-4415https://doi.org/10.1158/1078-0432.CCR-18-0323
        • Rini B.I.
        • Battle D.
        • Figlin R.A.
        • George D.J.
        • Hammers H.
        • Hutson T.
        • Jonasch E.
        • Joseph R.W.
        • McDermott D.F.
        • Motzer R.J.
        • et al.
        The society for immunotherapy of cancer consensus statement on immunotherapy for the treatment of advanced renal cell carcinoma (RCC).
        J. Immunother. Cancer. 2019; 7: 354https://doi.org/10.1186/s40425-019-0813-8
        • Rini B.I.
        • Motzer R.J.
        • Powles T.
        • McDermott D.F.
        • Escudier B.
        • Donskov F.
        • Hawkins R.
        • Bracarda S.
        • Bedke J.
        • De Giorgi U.
        • et al.
        Atezolizumab plus Bevacizumab Versus Sunitinib for patients with untreated metastatic renal cell carcinoma and sarcomatoid features: a prespecified subgroup analysis of the IMmotion151 clinical trial.
        Eur. Urol. 2021; 79: 659-662https://doi.org/10.1016/j.eururo.2020.06.021
        • Nishino M.
        • Ramaiya N.H.
        • Hatabu H.
        • Hodi F.S.
        Monitoring immune-checkpoint blockade: response evaluation and biomarker development.
        Nat. Rev. Clin. Oncol. 2017; 14: 655-668https://doi.org/10.1038/nrclinonc.2017.88
        • Tumeh P.C.
        • Harview C.L.
        • Yearley J.H.
        • Shintaku I.P.
        • Taylor E.J.M.
        • Robert L.
        • Chmielowski B.
        • Spasic M.
        • Henry G.
        • Ciobanu V.
        • et al.
        PD-1 blockade induces responses by inhibiting adaptive immune resistance.
        Nature. 2014; 515: 568-571https://doi.org/10.1038/nature13954
        • Zahoor H.
        • Duddalwar V.
        • D’Souza A.
        • Merseburger A.S.
        • Quinn D.I.
        What comes after immuno-oncology therapy for kidney cancer?.
        Kidney Cancer. 2019; 3: 93-102https://doi.org/10.3233/KCA-190053
        • Zahoor H.
        • Grivas P.
        The cancer immunogram: a pledge for a comprehensive biomarker approach for personalized immunotherapy in urothelial cancer.
        Eur. Urol. 2019; 75: 445-447https://doi.org/10.1016/j.eururo.2018.12.005
        • Aerts H.J.W.L.
        • Velazquez E.R.
        • Leijenaar R.T.H.
        • Parmar C.
        • Grossmann P.
        • Carvalho S.
        • Bussink J.
        • Monshouwer R.
        • Haibe-Kains B.
        • Rietveld D.
        • et al.
        Decoding tumour phenotype by noninvasive imaging using a quantitative radiomics approach.
        Nat. Commun. 2014; 5: 4006https://doi.org/10.1038/ncomms5006
        • Varghese B.A.
        • Cen S.Y.
        • Hwang D.H.
        • Duddalwar V.A.
        Texture analysis of imaging: what radiologists need to know.
        Am. J. Roentgenol. 2019; 212: 520-528
        • Sun R.
        • Limkin E.J.
        • Vakalopoulou M.
        • Dercle L.
        • Champiat S.
        • Han S.R.
        • Verlingue L.
        • Brandao D.
        • Lancia A.
        • Ammari S.
        • et al.
        A radiomics approach to assess tumour-infiltrating CD8 cells and response to anti-PD-1 or anti-PD-L1 immunotherapy: an imaging biomarker, retrospective multicohort study.
        Lancet Oncol. 2018; 19: 1180-1191https://doi.org/10.1016/S1470-2045(18)30413-3
        • Wen Q.
        • Yang Z.
        • Dai H.
        • Feng A.
        • Li Q.
        Radiomics study for predicting the expression of PD-L1 and tumor mutation burden in non-small cell lung cancer based on CT images and clinicopathological features.
        Front. Oncol. 2021; 11620246https://doi.org/10.3389/fonc.2021.620246
        • Xiao B.
        • Peng J.
        • Zhang R.
        • Xu J.
        • Wang Y.
        • Fang Y.
        • Lin J.
        • Pan Z.
        • Wu X.
        Density of CD8+ lymphocytes in biopsy samples combined with the circulating lymphocyte ratio predicts pathologic complete response to chemoradiotherapy for rectal cancer.
        Cancer Manag. Res. 2017; 9: 701-708https://doi.org/10.2147/CMAR.S150622
        • Ciccarese F.
        • Brandi N.
        • Corcioni B.
        • Golfieri R.
        • Gaudiano C.
        Complicated pyelonephritis associated with chronic renal stone disease.
        Radio. Med. 2021; 126: 505-516https://doi.org/10.1007/s11547-020-01315-7
        • Friston K.J.
        Statistical Parametric Mapping: the Analysis of Functional Brain Images.
        Elsevier/Academic Press, Amsterdam; Boston2007
        • Friedman J.
        • Hastie T.
        • Tibshirani R.
        Additive logistic regression: a statistical view of boosting (With discussion and a rejoinder by the authors).
        Ann. Stat. 2000; 28: 337-407https://doi.org/10.1214/aos/1016218223
        • Hastie T.
        • Tibshirani R.
        • Friedman J.
        Random Forests.
        in: Hastie T. Tibshirani R. Friedman J. The Elements of Statistical Learning: Data Mining, Inference, and Prediction. Springer Series in Statistics. Springer, New York, NY2009: 587-604https://doi.org/10.1007/978-0-387-84858-7_15
        • Demircioğlu A.
        Measuring the bias of incorrect application of feature selection when using cross-validation in radiomics.
        Insights Imaging. 2021; 12: 172https://doi.org/10.1186/s13244-021-01115-1
        • King G.
        • Zeng L.
        Logistic regression in rare events data.
        Political Anal. 2001; 9: 137-163
        • Liu X.
        Classification accuracy and cut point selection.
        Stat. Med. 2012; 31: 2676-2686https://doi.org/10.1002/sim.4509
        • Da-Ano R.
        • Visvikis D.
        • Hatt M.
        Harmonization strategies for multicenter radiomics investigations.
        Phys. Med. Biol. 2020; 65: 24TR02https://doi.org/10.1088/1361-6560/aba798
        • Da-ano R.
        • Masson I.
        • Lucia F.
        • Doré M.
        • Robin P.
        • Alfieri J.
        • Rousseau C.
        • Mervoyer A.
        • Reinhold C.
        • Castelli J.
        • et al.
        Performance comparison of modified ComBat for harmonization of radiomic features for multicenter studies.
        Sci. Rep. 2020; 10: 10248https://doi.org/10.1038/s41598-020-66110-w
        • Varghese B.A.
        • Hwang D.
        • Cen S.Y.
        • Levy J.
        • Liu D.
        • Lau C.
        • Rivas M.
        • Desai B.
        • Goodenough D.J.
        • Duddalwar V.A.
        Reliability of CT-based texture features: phantom study.
        J. Appl. Clin. Med. Phys. 2019; 20: 155-163https://doi.org/10.1002/acm2.12666
        • Varghese B.A.
        • Hwang D.
        • Cen S.Y.
        • Lei X.
        • Levy J.
        • Desai B.
        • Goodenough D.J.
        • Duddalwar V.A.
        Identification of robust and reproducible CT-texture metrics using a customized 3D-printed texture phantom.
        J. Appl. Clin. Med. Phys. 2021; https://doi.org/10.1002/acm2.13162
        • Liu R.
        • Elhalawani H.
        • Radwan Mohamed A.S.
        • Elgohari B.
        • Court L.
        • Zhu H.
        • Fuller C.D.
        Stability analysis of CT radiomic features with respect to segmentation variation in oropharyngeal cancer.
        Clin. Transl. Radiat. Oncol. 2020; 21: 11-18https://doi.org/10.1016/j.ctro.2019.11.005
        • Trebeschi S.
        • Drago S.G.
        • Birkbak N.J.
        • Kurilova I.
        • Cǎlin A.M.
        • Delli Pizzi A.
        • Lalezari F.
        • Lambregts D.M.J.
        • Rohaan M.W.
        • Parmar C.
        • et al.
        Predicting response to cancer immunotherapy using noninvasive radiomic biomarkers.
        Ann. Oncol. 2019; 30: 998-1004https://doi.org/10.1093/annonc/mdz108
        • Velcheti V.
        • Alilou M.
        • Khunger M.
        • Thawani R.
        • Madabhushi A.
        Changes in computer extracted features of vessel tortuosity on CT scans post-treatment in responders compared to non-responders for non–small cell lung cancer on immunotherapy.
        J. Thorac. Oncol. 2017; 12: S1547https://doi.org/10.1016/j.jtho.2017.06.067
      2. M. Alilou P. Vaidya M. Khorrami A. Zagouras P. Patil K. Bera P. Fu V. Velcheti A. Madabhushi. Madabhushi A. Quantitative vessel tortuosity radiomics on baseline non-contrast lung CT predict response to immunotherapy and are prognostic of overall survival, 2019, pp. 10950:109501F. 〈https://doi.org/10.1117/12.2513648〉.

        • Khorrami M.
        • Prasanna P.
        • Gupta A.
        • Patil P.
        • Velu P.D.
        • Thawani R.
        • Corredor G.
        • Alilou M.
        • Bera K.
        • Fu P.
        • et al.
        Changes in CT radiomic features associated with lymphocyte distribution predict overall survival and response to immunotherapy in non-small cell lung cancer.
        Cancer Immunol. Res. 2020; 8: 108-119https://doi.org/10.1158/2326-6066.CIR-19-0476
        • Del Re M.
        • Cucchiara F.
        • Rofi E.
        • Fontanelli L.
        • Petrini I.
        • Gri N.
        • Pasquini G.
        • Rizzo M.
        • Gabelloni M.
        • Belluomini L.
        • et al.
        A multiparametric approach to improve the prediction of response to immunotherapy in patients with metastatic NSCLC.
        Cancer Immunol. Immunother. 2021; 70: 1667-1678https://doi.org/10.1007/s00262-020-02810-6
        • Shafiq-ul-Hassan M.
        • Latifi K.
        • Zhang G.
        • Ullah G.
        • Gillies R.
        • Moros E.
        Voxel size and gray level normalization of CT radiomic features in lung cancer.
        Sci. Rep. 2018; 8: 10545https://doi.org/10.1038/s41598-018-28895-9
        • Mackin D.
        • Fave X.
        • Zhang L.
        • Fried D.
        • Yang J.
        • Taylor B.
        • Rodriguez-Rivera E.
        • Dodge C.
        • Jones A.K.
        • Court L.
        • Measuring C.T.
        scanner variability of radiomics features.
        Invest. Radiol. 2015; 50: 757-765https://doi.org/10.1097/RLI.0000000000000180
        • Mackin D.
        • Fave X.
        • Zhang L.
        • Yang J.
        • Jones A.K.
        • Ng C.S.
        • Court L.
        Harmonizing the pixel size in retrospective computed tomography radiomics studies.
        PLoS One. 2017; 12e0178524https://doi.org/10.1371/journal.pone.0178524
        • Shafiq-Ul-Hassan M.
        • Zhang G.G.
        • Latifi K.
        • Ullah G.
        • Hunt D.C.
        • Balagurunathan Y.
        • Abdalah M.A.
        • Schabath M.B.
        • Goldgof D.G.
        • Mackin D.
        • et al.
        Intrinsic dependencies of CT radiomic features on voxel size and number of gray levels.
        Med. Phys. 2017; 44: 1050-1062https://doi.org/10.1002/mp.12123
        • Lovinfosse P.
        • Janvary Z.L.
        • Coucke P.
        • Jodogne S.
        • Bernard C.
        • Hatt M.
        • Visvikis D.
        • Jansen N.
        • Duysinx B.
        • Hustinx R.
        FDG PET/CT texture analysis for predicting the outcome of lung cancer treated by stereotactic body radiation therapy.
        Eur. J. Nucl. Med. Mol. Imaging. 2016; 43: 1453-1460https://doi.org/10.1007/s00259-016-3314-8
        • Desseroit M.-C.
        • Visvikis D.
        • Tixier F.
        • Majdoub M.
        • Perdrisot R.
        • Guillevin R.
        • Cheze Le Rest C.
        • Hatt M.
        Development of a nomogram combining clinical staging with (18)F-FDG PET/CT image features in non-small-cell lung cancer stage I-III.
        Eur. J. Nucl. Med. Mol. Imaging. 2016; 43: 1477-1485https://doi.org/10.1007/s00259-016-3325-5
        • Hatt M.
        • Majdoub M.
        • Vallières M.
        • Tixier F.
        • Le Rest C.C.
        • Groheux D.
        • Hindié E.
        • Martineau A.
        • Pradier O.
        • Hustinx R.
        • et al.
        18F-FDG PET uptake characterization through texture analysis: investigating the complementary nature of heterogeneity and functional tumor volume in a multi-cancer site patient cohort.
        J. Nucl. Med. 2015; 56: 38-44https://doi.org/10.2967/jnumed.114.144055
        • Reuben A.
        • Spencer C.N.
        • Prieto P.A.
        • Gopalakrishnan V.
        • Reddy S.M.
        • Miller J.P.
        • Mao X.
        • De Macedo M.P.
        • Chen J.
        • Song X.
        • et al.
        Genomic and immune heterogeneity are associated with differential responses to therapy in melanoma.
        npj Genom. Med. 2017; 2: 1-11https://doi.org/10.1038/s41525-017-0013-8
        • Rao S.-X.
        • Lambregts D.M.
        • Schnerr R.S.
        • Beckers R.C.
        • Maas M.
        • Albarello F.
        • Riedl R.G.
        • Dejong C.H.
        • Martens M.H.
        • Heijnen L.A.
        • et al.
        CT texture analysis in colorectal liver metastases: a better way than size and volume measurements to assess response to chemotherapy?.
        U. Eur. Gastroenterol. J. 2016; 4: 257-263https://doi.org/10.1177/2050640615601603
        • Kim J.
        • DeBerardinis R.J.
        Mechanisms and implications of metabolic heterogeneity in cancer.
        Cell Metab. 2019; 30: 434-446https://doi.org/10.1016/j.cmet.2019.08.013
        • Orlhac F.
        • Boughdad S.
        • Philippe C.
        • Stalla-Bourdillon H.
        • Nioche C.
        • Champion L.
        • Soussan M.
        • Frouin F.
        • Frouin V.
        • Buvat I.
        A postreconstruction harmonization method for multicenter radiomic studies in PET.
        J. Nucl. Med. 2018; 59: 1321-1328https://doi.org/10.2967/jnumed.117.199935
        • Cetin-Karayumak S.
        • Stegmayer K.
        • Walther S.
        • Szeszko P.R.
        • Crow T.
        • James A.
        • Keshavan M.
        • Kubicki M.
        • Rathi Y.
        Exploring the limits of ComBat method for multi-site diffusion MRI harmonization.
        bioRxiv. 2020; (2020.11.20.390120. doi: 10.1101/2020.11.20.390120)
        • Demirjian N.L.
        • Varghese B.A.
        • Cen S.Y.
        • Hwang D.H.
        • Aron M.
        • Siddiqui I.
        • Fields B.K.K.
        • Lei X.
        • Yap F.Y.
        • Rivas M.
        • et al.
        CT-based radiomics stratification of tumor grade and TNM stage of clear cell renal cell carcinoma.
        Eur. Radiol. 2021; https://doi.org/10.1007/s00330-021-08344-4
        • Parmar C.
        • Rios Velazquez E.
        • Leijenaar R.
        • Jermoumi M.
        • Carvalho S.
        • Mak R.H.
        • Mitra S.
        • Shankar B.U.
        • Kikinis R.
        • Haibe-Kains B.
        • et al.
        Robust Radiomics feature quantification using semiautomatic volumetric segmentation.
        PLoS One. 2014; 9e102107https://doi.org/10.1371/journal.pone.0102107
        • Khaleel S.
        • Katims A.
        • Cumarasamy S.
        • Rosenzweig S.
        • Attalla K.
        • Hakimi A.A.
        • Mehrazin R.
        Radiogenomics in clear cell renal cell carcinoma: a review of the current status and future directions.
        Cancers. 2022; 14: 2085https://doi.org/10.3390/cancers14092085