\n", "
\n", "*Mathieu Boudreau1,2, *Agah Karakuzu1, Julien Cohen-Adad1,3,4,5, Ecem Bozkurt6, Madeline Carr7,8, Marco Castellaro9, Luis Concha10, Mariya Doneva11, Seraina Dual12, Alex Ensworth13,14, Alexandru Foias1, Véronique Fortier15,16, Refaat E. Gabr17, Guillaume Gilbert18, Carri K. Glide-Hurst19, Matthew Grech-Sollars20,21, Siyuan Hu22, Oscar Jalnefjord23,24, Jorge Jovicich25, Kübra Keskin6, Peter Koken11, Anastasia Kolokotronis13,26, Simran Kukran27,28, Nam. G. Lee6, Ives R. Levesque13,29, Bochao Li6, Dan Ma22, Burkhard Mädler30, Nyasha Maforo31,32, Jamie Near33,34, Erick Pasaye35, Alonso Ramirez-Manzanares36, Ben Statton37,Christian Stehning30, Stefano Tambalo25, Ye Tian6, Chenyang Wang38, Kilian Weiss30, Niloufar Zakariaei39, Shuo Zhang30, Ziwei Zhao6, Nikola Stikov1,2,40\n", "
\n", "\n", "\n", "\n", "1NeuroPoly Lab, Polytechnique Montréal, Montreal, Quebec, Canada,\n", "2Montreal Heart Institute, Montreal, Quebec, Canada,\n", "3Unité de Neuroimagerie Fonctionnelle (UNF), Centre de recherche de l’Institut Universitaire de Gériatrie de Montréal (CRIUGM), Montreal, Quebec, Canada,\n", "4Mila - Quebec AI Institute, Montreal, QC, Canada,\n", "5Centre de recherche du CHU Sainte-Justine, Université de Montréal, Montreal, QC, Canada,\n", "6Magnetic Resonance Engineering Laboratory (MREL), University of Southern California, Los Angeles, California, USA,\n", "7Medical Physics, Ingham Institute for Applied Medical Research, Liverpool, Australia,\n", "8Department of Medical Physics, Liverpool and Macarthur Cancer Therapy Centres, Liverpool, Australia,\n", "9Department of Information Engineering, University of Padova, Padova, Italy,\n", "10Institute of Neurobiology, Universidad Nacional Autónoma de México Campus Juriquilla, Querétaro, México,\n", "11Philips Research Hamburg, Germany,\n", "12Stanford University, Stanford, California, United States,\n", "13Medical Physics Unit, McGill University, Montreal, Canada,\n", "14University of British Columbia, Vancouver, Canada,\n", "15Department of Medical Imaging, McGill University Health Centre, Montreal, Quebec, Canada\n", "16Department of Radiology, McGill University, Montreal, Quebec, Canada,\n", "17Department of Diagnostic and Interventional Imaging, University of Texas Health Science Center at Houston, McGovern Medical School, Houston, Texas, USA, \n", "18MR Clinical Science, Philips Canada, Mississauga, Ontario, Canada,\n", "19Department of Human Oncology, University of Wisconsin-Madison, Madison, Wisconsin, USA,\n", "20Centre for Medical Image Computing, Department of Computer Science, University College London, London, UK,\n", "21Lysholm Department of Neuroradiology, National Hospital for Neurology and Neurosurgery, University College London Hospitals NHS Foundation Trust, London, UK,\n", "22Department of Biomedical Engineering, Case Western Reserve University, Cleveland, Ohio, USA, \n", "23Department of Medical Radiation Sciences, Institute of Clinical Sciences, Sahlgrenska Academy, University of Gothenburg, Gothenburg, Sweden,\n", "24Biomedical Engineering, Sahlgrenska University Hospital, Gothenburg, Sweden, \n", "25Center for Mind/Brain Sciences, University of Trento, Italy,\n", "26Hopital Maisonneuve-Rosemont, Montreal, Canada,\n", "27Bioengineering, Imperial College London, UK,\n", "28Radiotherapy and Imaging, Insitute of Cancer Research, Imperial College London, UK,\n", "29Research Institute of the McGill University Health Centre, Montreal, Canada,\n", "30Clinical Science, Philips Healthcare, Germany,\n", "31Department of Radiological Sciences, University of California Los Angeles, Los Angeles, CA, USA,\n", "32Physics and Biology in Medicine IDP, University of California Los Angeles, Los Angeles, CA, USA,\n", "33Douglas Brain Imaging Centre, Montreal, Canada,\n", "34Sunnybrook Research Institute, Toronto, Canada,\n", "35National Laboratory for Magnetic Resonance Imaging, Institute of Neurobiology, Universidad Nacional Autónoma de México (UNAM), Juriquilla, Mexico,\n", "36Computer Science Department, Centro de Investigación en Matemáticas, A.C., Guanajuato, México,\n", "37Medical Research Council, London Institute of Medical Sciences, Imperial College London, London, United Kingdom,\n", "38Department of Radiation Oncology - CNS Service, The University of Texas MD Anderson Cancer Center, Texas, USA,\n", "39Department of Biomedical Engineering, University of British Columbia, British Columbia, Canada,\n", "40Center for Advanced Interdisciplinary Research, Ss. Cyril and Methodius University, Skopje, North Macedonia\n", "
\n", "\n", "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# Abstract\n", "\n", "\n", "\n", "**Purpose:** T1 mapping is a widely used quantitative MRI technique, but its tissue-specific values remain inconsistent across protocols, sites, and vendors. The ISMRM Reproducible Research study group (RRSG) and Quantitative MR study group (qMRSG) jointly launched a T1 mapping reproducibility challenge. The goal was to investigate whether an independently-implemented image acquisition protocol at multiple centers could reliably measure T1 using inversion recovery in a standardized phantom and in brains of healthy volunteers.\n", "\n", "\n", "**Methods:** The reproducibility of a robust T1 mapping protocol and fitting algorithm from a well-known Magnetic Resonance in Medicine publication (Barral et al. 2010) were evaluated. Participants in the challenge were invited to acquire T1 mapping data on a standard ISMRM/NIST phantom and/or in healthy human brains. Data submission, pipeline development, and analysis were performed using open-source platforms for increased reproducibility and transparency. For participants acquiring data at multiple sites, inter-submission and intra-submission comparisons were conducted by selecting one dataset per submission.\n", "\n", "\n", "\n", "**Results:** Eighteen submissions were accepted using data collected from MRIs from three manufacturers (Siemens, GE, Philips) and at 3T (except one submission, at 0.35T), resulting in a total of 38 datasets acquired in phantoms and 56 datasets in healthy human subjects. Inter-submission mean coefficient of variation (CoV) was 6.1% for phantom measurements, nearly twice as high as the evaluated intra-submission CoV (2.9%). A similar trend was observed in the human data (inter-/intra- submission CoV was 6.0/2.9 % in the genu and 16/6.9 % in the cortical gray matter).\n", "\n", "Conclusion: The imaging protocol and fitting algorithm from Barral et al. (2010) resulted in good reproducibility of both phantom and human brain T1 measurements. Differences in implementations between submissions resulted in higher variance in reported values relative to intra-submission variability. This challenge generated a large open database of inversion recovery T1 mapping acquisitions across several sites and MRI vendors accessible at osf.io/ywc9g/. To allow for the wider community to access and engage with this dataset, we developed an interactive dashboard that is available at https://rrsg2020.dashboards.neurolibre.org." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\nFigure 1 Schematic of the system phantom (a) used in this challenge. Reproduced and cropped from Stupic et al. 2021 (Stupic et al. 2021) (Creative Commons CC BY license). ROI selection for the ISMRM/NIST phantom (b) and the human brain (c). b) The 14 phantom ROIs (shades of blue/green) were automatically generated using a script provided by NIST. In yellow are the three reference pins in the phantom, i.e. these are not ROIs or spheres. c) Human brain ROIs were manually segmented in four regions: the genu (yellow, 5⨯5 voxels), splenium (green, 5⨯5 voxels), deep gray matter (blue, 5⨯5 voxels), and cortical gray matter (red, three sets of 3⨯3 voxels). Note: due to differences in slice positioning from the single-slice datasets provided by certain sites, for some datasets it was not possible to manually segment an ROI in the genu or deep gray matter. In the case of the missing genu, left or right frontal white matter (WM) was selected; for deep gray matter (GM), it was omitted entirely for those cases.\n
\n\nManual ROIs were segmented by a single researcher (M.B., 11+ years of neuroimaging experience) using FSLeyes {cite:p}`McCarthy2019-qd` in four regions for the human datasets Figure 1-c): located in the genu, splenium, deep gray matter, and cortical gray matter. Automatic segmentation was not used because the data were single-slice and there was inconsistent slice positioning between datasets." }, { "cell_type": "markdown", "metadata": {}, "source": [ "### 2.4.3 | Analysis and Statistics\n", "\n", "Analysis code and scripts were developed and shared in a version-tracked public GitHub repository [^public-repo]. T1 fitting and main data analysis was performed for all datasets by one of the challenge organizers (M.B.). Python-based Jupyter Notebooks were used for both the quality assurance and main analysis workflows. The computational environment requirements were containerized in Docker {cite:p}`Boettiger2015-vd,Merkel2014-cu`, allowing for an executable environment that can reproduce the analysis in a web browser through MyBinder [^my-binder] {cite:p}`Project_Jupyter2018-ll`. Python scripts handled reference data, database handling, ROI masking, and general analysis tools, while configuration files managed the dataset information which were downloaded and pooled using a script (`make_pooled_datasets.py`). The databases were created using a reproducible Jupyter Notebook and subsequently saved in the repository.\n", "\n", "For the ISMRM/NIST phantom data, mean T1 values for each ROI were compared with temperature-corrected reference values and visualized in three different types of plots (linear axes, log-log axes, and error relative to the reference value). This comparison was repeated for individual measurements at each site and for all measurements grouped together. Temperature correction was carried out via nonlinear interpolation [^nonlinear] of the set of reference NIST T1 values between 16 °C and 26 °C (2 °C intervals), listed in the phantom technical specifications. For the human datasets, a notebook was created to plot the mean and standard deviations for each tissue ROI from all submissions from all sites. All quality assurance and analysis plot images were saved to the repository for ease-of-access and a timestamped version-controlled record of the state of the analysis figures. The database files of ROI values and acquisition details for all submissions were also saved to the repository.\n", "\n", "An interactive dashboard [^dashboard] was developed in Dash by Plotly (Plotly Technologies Inc. 2015) and hosted by NeuroLibre {cite:p}`Karakuzu2022-xq` to enable real-time exploration of the data, analysis, and statistics of the challenge results. The dashboard reports descriptive statistics for a variety of alternative looks at phantom and brain data, as well as some statistical comparisons (e.g., the hierarchical shift function [^hsf]). The data was collected from the pre-prepared databases of masked ROI values and incorporated other database information, such as phantom version, temperature, MRI system, and reference values. The interactive dashboard displays these results for all measurements at all sites." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "# 3 | RESULTS\n", "\n", "## 3.1 | Dashboard\n", "\n", "To disseminate the challenge results, a web-based dashboard was developed (Figure 2, https://rrsg2020.dashboards.neurolibre.org). The landing page (Figure 2-a) showcases the relationship between the phantom and brain datasets acquired at different sites/vendors. Navigating to the phantom section leads to the information about the submitted datasets, such as the mean/std/median/CoV for each sphere, % difference from the reference values, number of scans, and temperature (Figure 2-b, left). Other options allow users to limit the results by specific versions of the phantom or the MRI manufacturer. Selecting either “By Sphere” (Figure 2-b, right) or “By Site” tabs will display whisker plots for the selected options, enabling further exploration of the datasets.\n", "\n", "Returning to the home page and selecting the brain section allows exploration of information on the brain datasets (Figure 2-c, left), such as mean T1 and STD for different ROI regions, as well as selection of specific MRI manufacturers. Choosing the “By Regions” tab provides whisker plots of the datasets for the selected ROI (Figure 2-c, right), similar to the plots for the phantom." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "Figure 2 Dashboard. a) welcome page listing all the sites, the types of subject, and scanner, and the relationship between the three. Row b) shows two of the phantom dashboard tabs, and row c) shows two of the human data dashboard tabs Link: https://rrsg2020.dashboards.neurolibre.org\n", "
" ] }, { "cell_type": "markdown", "metadata": {}, "source": "## 3.2 | Submissions\n\n\nEighteen submissions were included in the analysis, which resulted in 38 T1 maps of the NIST/system phantom, and 56 brain T1 maps. Figure 3 illustrates all the submissions that acquired phantom data (Figure 3-a) and human data (Figure 3-b), the number of scanners used for each submission, and the number of T1 mapping datasets. It should be noted that these numbers include a subset of measurements where both complex and magnitude-only data from the same acquisition were used to fit T1 maps, thus the total number of unique acquisitions is lower than the numbers reported above (27 for phantom data and 44 for human data). The datasets were collected on three MRI manufacturers (Siemens, GE, Philips) and were acquired at 3T [^three-t], except for one dataset acquired at 0.35T (the ViewRay MRidian MR-linac) . To showcase the heterogeneity of the actual T1 map data from the independently-implemented submissions, Figure 4 displays six T1 maps of the phantoms submitted to the challenge.\n\n\n```{image} images/figure_3_full.png\n---\nwidth: 900px\nname: fig3-notebook\nalign: center\n---\n```\n\nFigure 3 Complete list of the datasets submitted to the challenge. Submissions that included phantom data are shown in a), and those that included human brain data are shown in b). Submissions were assigned numbers to keep track of which submissions included both phantom and human data. Some submissions included datasets acquired on multiple scanners. For the phantom (panel a), each submission acquired all their data using a single phantom, however some researchers shared the same physical phantom with each other (same color). Some additional details about the datasets are included in the T1 maps column, if relevant. Note that for complex datasets in the magnitude/phase format, T1 maps were calculated both using magnitude-only data and complex-data, but these were from the same measurement (branching off arrow). \n
\n\nOf these datasets, several submissions went beyond the minimum acquisition and acquired additional datasets using the ISMRM/NIST phantom, such as a traveling phantom (7 scanners/sites), scan-rescan, same-day rescans on two MRIs, short TR vs long TR, and 4 point TI vs 14 point TI. For humans, one site acquired 13 subjects on two scanners (different manufacturers), one site acquired 6 subjects, and one site acquired a subject using two different head coils (20 channels vs. 64 channels)." }, { "cell_type": "code", "execution_count": 3, "metadata": { "execution": { "iopub.execute_input": "2026-09-29T19:10:31.839883Z", "iopub.status.busy": "2026-09-29T19:10:31.785219Z", "iopub.status.idle": "2026-09-29T19:10:34.136065Z", "shell.execute_reply": "2026-09-29T19:10:34.135719Z" }, "tags": [ "hide_input", "remove_output" ] }, "outputs": [], "source": "from os import path\nfrom pathlib import Path\nimport os\nfrom repo2data.repo2data import Repo2Data\n\nif build == 'latest':\n if path.isdir('analysis')== False:\n !git clone https://github.com/rrsg2020/analysis.git\n dir_name = 'analysis'\n analysis = os.listdir(dir_name)\n\n for item in analysis:\n if item.endswith(\".ipynb\"):\n os.remove(os.path.join(dir_name, item))\n if item.endswith(\".md\"):\n os.remove(os.path.join(dir_name, item))\nelif build == 'archive':\n if os.path.isdir(Path('../../data')):\n data_path = ['../../data/rrsg-2020-neurolibre']\n else:\n # define data requirement path\n data_req_path = os.path.join(\"..\", \"binder\", \"data_requirement.json\")\n # download data\n repo2data = Repo2Data(data_req_path)\n data_path = repo2data.install()\n\n# Imports\n\nfrom pathlib import Path\nimport pandas as pd\nimport json\nimport nibabel as nib\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport matplotlib.animation as animation\nfrom IPython.display import Video\nimport glob\nfrom analysis.src.plots import *\nfrom analysis.make_pooled_datasets import *\n\n# Configurations\nif build == 'latest':\n configFile = Path('analysis/configs/3T_NIST_T1maps.json')\n data_folder_name = 'analysis/3T_NIST_T1maps'\n\n configFile_raw = Path('analysis/configs/3T_NIST.json')\n data_folder_name_raw = 'analysis/3T_NIST'\nelif build=='archive':\n configFile = Path(data_path[0] + '/analysis/configs/3T_NIST_T1maps.json')\n data_folder_name = data_path[0] + '/analysis/3T_NIST_T1maps'\n\n configFile_raw = Path(data_path[0] + '/analysis/configs/3T_NIST.json')\n data_folder_name_raw = data_path[0] + '/analysis/3T_NIST'\n\n# Download datasets\nif not Path(data_folder_name).exists():\n print(Path(data_folder_name))\n make_pooled_dataset(configFile, data_folder_name)\n\nif not Path(data_folder_name_raw).exists():\n make_pooled_dataset(configFile_raw, data_folder_name_raw)\n\nwith open(configFile) as json_file:\n configJson = json.load(json_file)\n\nwith open(configFile_raw) as json_file:\n configJson_raw = json.load(json_file)\n \ndef get_image(dataset_name, key2):\n # Load T1 image data\n t1_file = configJson[dataset_name]['datasets'][key2]['imagePath']\n t1 = nib.load(Path(data_folder_name) / t1_file)\n t1_volume = t1.get_fdata() \n\n # Load raw image data\n raw_file = configJson_raw[dataset_name]['datasets'][key2]['imagePath']\n raw = nib.load(Path(data_folder_name_raw) / raw_file)\n raw_volume = raw.get_fdata() \n\n # Handle 2D vs 3D volume case\n dims = t1_volume.shape\n if (len(dims) == 2) or (np.min(dims) == 1):\n im = np.rot90(t1_volume)\n TI_1 = np.rot90(np.squeeze(raw_volume[:,:,0,0]))\n TI_2 = np.rot90(np.squeeze(raw_volume[:,:,0,1]))\n TI_3 = np.rot90(np.squeeze(raw_volume[:,:,0,2]))\n TI_4 = np.rot90(np.squeeze(raw_volume[:,:,0,3])) \n else:\n index_smallest_dim = np.argmin(dims)\n numberOfSlices = dims[index_smallest_dim]\n midSlice = int(np.round(numberOfSlices/2))\n\n if index_smallest_dim == 0:\n im = np.rot90(np.squeeze(t1_volume[midSlice,:,:]))\n TI_1 = np.rot90(np.squeeze(raw_volume[midSlice,:,:,0]))\n TI_2 = np.rot90(np.squeeze(raw_volume[midSlice,:,:,1]))\n TI_3 = np.rot90(np.squeeze(raw_volume[midSlice,:,:,2]))\n TI_4 = np.rot90(np.squeeze(raw_volume[midSlice,:,:,3])) \n elif index_smallest_dim == 1:\n im = np.rot90(np.squeeze(t1_volume[:,midSlice,:]))\n TI_1 = np.rot90(np.squeeze(raw_volume[:,midSlice,:,0]))\n TI_2 = np.rot90(np.squeeze(raw_volume[:,midSlice,:,1]))\n TI_3 = np.rot90(np.squeeze(raw_volume[:,midSlice,:,2]))\n TI_4 = np.rot90(np.squeeze(raw_volume[:,midSlice,:,3]))\n elif index_smallest_dim == 2:\n im = np.rot90(np.squeeze(t1_volume[:,:,midSlice]))\n TI_1 = np.rot90(np.squeeze(raw_volume[:,:,midSlice,0]))\n TI_2 = np.rot90(np.squeeze(raw_volume[:,:,midSlice,1]))\n TI_3 = np.rot90(np.squeeze(raw_volume[:,:,midSlice,2]))\n TI_4 = np.rot90(np.squeeze(raw_volume[:,:,midSlice,3]))\n\n xAxis = np.linspace(0,im.shape[0]-1, num=im.shape[0])\n yAxis = np.linspace(0,im.shape[1]-1, num=im.shape[1])\n return im, xAxis, yAxis, TI_1, TI_2, TI_3, TI_4\n\n\nim_1, xAxis_1, yAxis_1, TI_1_1, TI_2_1, TI_3_1, TI_4_1 = get_image('wang_MDAnderson_NIST', 'day2_mag')\n\nim_2, xAxis_2, yAxis_2, TI_1_2, TI_2_2, TI_3_2, TI_4_2 = get_image('CStehningPhilipsClinicalScienceGermany_NIST', 'Bonn_MR1_magnitude')\n\nim_3, xAxis_3, yAxis_3, TI_1_3, TI_2_3, TI_3_3, TI_4_3 = get_image('mrel_usc_NIST', 'Session1_MR1')\n\nim_4, xAxis_4, yAxis_4, TI_1_4, TI_2_4, TI_3_4, TI_4_4 = get_image('karakuzu_polymtl_NIST', 'mni')\n\nim_5, xAxis_5, yAxis_5, TI_1_5, TI_2_5, TI_3_5, TI_4_5 = get_image('madelinecarr_lha_NIST', 'one')\n\nim_6, xAxis_6, yAxis_6, TI_1_6, TI_2_6, TI_3_6, TI_4_6 = get_image('matthewgrechsollars_ICL_NIST', 'magnitude')\nim_6 = np.flipud(im_6)\nTI_1_6 = np.flipud(TI_1_6)\nTI_2_6 = np.flipud(TI_2_6)\nTI_3_6 = np.flipud(TI_3_6)\nTI_4_6 = np.flipud(TI_4_6)" }, { "cell_type": "code", "execution_count": 4, "metadata": { "execution": { "iopub.execute_input": "2026-09-29T19:10:34.144220Z", "iopub.status.busy": "2026-09-29T19:10:34.138399Z", "iopub.status.idle": "2026-09-29T19:10:37.779710Z", "shell.execute_reply": "2026-09-29T19:10:37.131835Z" }, "tags": [ "report_output", "hide_input" ] }, "outputs": [ { "data": { "text/html": [ " \n", " " ] }, "metadata": {}, "output_type": "display_data" }, { "data": { "text/html": [ "\n", "\n", "\n", "