Sec. 9 Strategies for selecting climate models
Last update: Mon Feb 13 16:06:49 2023 -0500 (704da26)
Note: This Section has not be updated with CMIP6 data
One of the challenges facing students and practitioners of climate change impact assessment is deciding which models to use for the assessment. Have a look at Figure 9.1. The black line is the actual observed Toronto annual average \(T_{\textrm{mean}}\) (produced from the data that we downloaded from Environment and Climate Change Canada). Each of the coloured solid lines represents the historical scenario of some model from the r1i1p1 ensemble. The dashed lines, starting circa 2005, represent the RCP scenario-forced projections for future climate in the same grid cell.
Figure 9.1: Observed and GCM-modelled annual average mean temperature at Toronto (1981‒2100). The black line traces observed data from the Toronto climate station. Each coloured line represents the projection from one of 39 CMIP5 models. Solid lines indicate historical projections; broken lines are projections forced with the four IPCC RCP scenarios. Mouse over the figure for a detailed tooltip.
So, why do we see so much variation? There are a couple of factors at play here. Perhaps most importantly, is the fact that we are comparing apples to oranges, or, more specifically, an apple to an orchard! Each of these model projections is for a grid cell that contains the specific coordinates of the Toronto weather station. Toronto has a lot of local climate considerations like a strong Urban Heat Island and an important Lake Breeze, which makes it stand out in the context of the larger region. These processes are not well represented in the climate models, some of which don’t even consider the Great Lakes!
Another consideration is the fact that each model will display its own bias. The modelling groups that have created these models only focus on “local areas” in the broadest sense of the word (e.g., the Arctic), unless there is a very specific, large bias in a particular region that needs addressing. Biases typically arise due to biases in the representation of physical processes (usually in parametrization) or due to biases generated by inadequate horizontal or vertical resolution. Just like the climate of a model, the biases are emergent and are actually quite hard to remedy on a local scale. These biases mean that a model that performs very well over the deserts of Australia may not perform as well over the Arctic.
How, then, do we choose which models to use to inform our climate change impact assessment? There are a number of strategies that we can employ for the task, including using models at the extreme top and bottom of the projected range, using the average of all models in a multi-model ensemble, or selecting “validated” models based on the models’ ability to reproduce the observed baseline climate. We will detail these strategies in the following section.
9.1 Model selection strategies
Fenech, Comer, and Gough (2007) describe three approaches to selecting climate models: using the extremes; validating models based on their ability to reproduce some observed metric; and the use of multi-model ensembles.
If we select the model extremes, we would select both the smallest and the largest projected changes as reference limits for policy planning. In this sense, the extreme models can provide an indicator of the upper and lower limits of the full range of possible futures projected by the available models.
Another common approach to model selection is to validate models based on their ability to simulate the past and present climate. Some studies that have used validation methods include Ragettli et al. (2013), C. Xie et al. (2013), and Hewer and Gough (2016). When we use a validation-based approach, we are working under two assumptions: 1) that a model’s ability to simulate an observed baseline climate will be representative of the same model’s ability to project future climate, and 2) that the model is simulating a reasonable baseline climate for the right reasons.
Scientists may also choose to use all (or a subset) of the available models to create a multi-model ensemble. In a multi-model ensemble, the output from the selected models is averaged, and that average is used to project future climate. Multi-model ensembles often outperform individual models (Pierce et al. 2009). The ensemble approach partially addresses the varying strengths and weaknesses of individual models, with the hope that model biases will be cancelled out within the averaged output (Knutti et al. 2010). The ensemble approach has been shown to cancel offsetting errors present in individual models (Pierce et al. 2009), and can correct for differences in the parametrization of the models (Knutti et al. 2010). Thus, many studies choose to make use of a full ensemble of all available models, to account for the entire range of available projection data, such as Gough, Anderson, and Herod (2016).
9.2 Caveats to these strategies
The above strategies are easy to implement, and are commonly used in the literature, however, there are some important caveats and criticisms to these methods. For instance, a key criticism levelled against the use of the upper and lower extremes is that we typically do not have a priori information regarding why the relevant models offer extreme projections. Perhaps they truly are outliers, and we are creating an artificially wide range of possible scenarios.
Model validation is also problematic. Recall our earlier metaphor about an apple in an orchard. The spatial scales between the station data and the GCM grid are very different, and judging global model performance based on a sub grid-point observed data is like burning our orchard to the ground because we found a worm in a single apple. However, this mismatch does not only apply to the spatial scale. We also need to consider the temporal scale: Assuming a model simulates variability correctly, it can be hard to know whether this variability agrees with observations because we have a very limited historical record, and therefore a limited sample of observations. Modelled climatological processes can vary dramatically over the course of a long-term simulation. Sometimes this variability will line up well with our observations, and sometimes it won’t. If, upon validation, we happen to choose the model years in which our simulation does not look like observations, then our model will seem flawed. However, it could be the case that nature is actually very similar to our model, but we’ve only sampled one small piece of its total variability. Finally, and perhaps most importantly, we can’t be sure that the ability of a model to project the past is truly representative of the same model’s ability to project the future. In fact, even the ability of the model to reproduce the baseline presents a challenge related to equifinality; Gough (2001) showed that changes in model parametrization can greatly change the resulting future projections, even if the baseline is reproduced equally well by each run.
Finally, all three of our methods generally consider the available models to be independent options for the future. In truth, many of the climate models have a shared history, or use the same underlying code for some of their components (atmosphere, ocean, ice, land surface). In this sense, the models are not truly independent. If we perform a validation approach to select the top-ranked models, it is likely that we will find that interdependent models perform equally well or equally poorly. Likewise, in an ensemble-based study, including related models goes against our goal of cancelling out individual biases, as two interdependent models could artificially skew the overall mean slightly.
Given the above, we must carefully consider our approach to model selection to ensure that we are making good choices. In Lab 4 (Sec. 10) we will compare these model selection methods using a change factor approach. In Lab 5 (Sec. 12), we will see how model projections can be downscaled to a smaller scale to reduce the model bias and, therefore, increase model skill, without the need for validation-based model selection approaches.