The Atmospheric Model Evaluation Tool
Introduction
The Atmospheric Model Evaluation Tool (AMET) was developed to facilitate the evaluation of the meteorological and air quality models within the CMAQ modeling system. AMET helps users manage and expedite the evaluation of research and operational model simulations. Although it was developed specifically for the CMAQ modeling system, the AMET software can be adapted to work with other modeling systems.
AMET scripts first pair observations and gridded model output in space and time and then provide a suite of analysis methods to elucidate model performance for predicted or diagnosed atmospheric fields. There are separate modules in AMET to analyze meteorological and air quality model output. The partition is necessary because both the observed and predicted meteorological and air quality data are quite different, utilizing different file formats for both the observed and model data. In addition, the observed meteorological and air quality datasets are often obtained from networks that are not co-located and use a range of sampling periods, which can make pairing meteorological and air quality data difficult. One advantage of separate meteorological and air quality modules in AMET is that the modules can be used independently, allowing a user to reduce installation time, learning curve and complexity if only meteorological or air quality evaluation is needed.
Evaluation of model simulations, which usually includes comparing the model-predicted values against the available observed values, is critical to establishing the model’s credibility (Dennis et al., 2010). For the development of CMAQ, evaluation of model predicted concentrations for many different atmospheric pollutants is required since the model is used to develop sound regulatory decisions by the Federal as well as State governments. Establishing air quality model credibility is further complicated by the need to understand the uncertainties in the meteorology that drive the CMAQ chemistry, dispersion and transport.
Roughly a decade ago, CMAQ model simulations typically spanned only several weeks to a month (in some rare cases several months) and were limited spatially (for example, to the eastern half of the U.S.). The increase in computing power over time has allowed for larger model domains, longer simulation time periods and a greater number of model simulations (e.g. sensitivities and ensemble modeling studies).
Today, CMAQ simulations routinely span several months to a year and in some cases decades, and have expanded spatially to cover the continental U.S., Northern Hemisphere and more recently the entire Globe. The result is terabytes of model output available for evaluation against observed data, which conventional methods of data analysis (e.g. spreadsheets) are not well suited. In addition to longer duration simulations and larger domains, the shorter wall-clock model run times have resulted in many more model simulations being performed. The increase in the duration, spatial size and number of model simulations all require greater organization and efficiency to systematically evaluate the large amounts of data being produced.
Visit the AMET GitHub repository to see what's new in AMET Version 1.5 (Aug. 2022)
AMET-AQ Observation Data
AMET ready observation data have been made available for the years 2000 through 2023 from the CMAS Center Data Warehouse.
- 2000 - 2023 observation data from the CMAS Center Data Warehouse
- Meta data README file for the observation data
The network data available include: AERONET, AMON, AQS, CASTNET, CSN, FLUXNET, IMPROVE, NADP, NAPS, NOAA ESRL, SEARCH, and TOAR.
NetCDF files of hourly observed values from AQS that can be used for the creation of observation overlays in the VERDI visualization software are also available for the years 2000-2017 from the CMAS Center Data Warehouse.
AMET Overview
AMET is built on several open-source software packages. MySQL database software is leveraged to store the paired observation and model output along with various metadata. The pairing is performed by several scripts, which insert those data into the MySQL database using Fortran and R functions. Finally, the analysis of the data utilizes scripts and functions written for the R statistical software package. Figure 1 presents a flow of processing and analysis steps of the AMET package.
The initial step required as part of the AMET evaluation process is to pair the observed and modeled values in space and time. In the air quality (AQ) module of AMET, this is accomplished using the Site Compare software. Site Compare performs matching using the nearest neighbor approach, and applies temporal averaging to the modeled data if required. In most cases, the Site Compare program is able to read the observation data files that are provided directly from the major air quality networks in the United States.
The air quality networks that Site Compare/AMET currently support are the EPA's AQS, the Chemical Speciation Network (CSN), the Interagency Monitoring of PROtected Visual Environments (IMPROVE) network, the Clean Air Status and Trends Network (CASTNET), the National Atmospheric Deposition Program (NADP) networks (AIRMON, AMON and NTN), the SouthEastern Aerosol Research and CHaracterization Study experiment (SEARCH) and the Aerosol Robotic Network (AERONET), the National Air Pollution Surveillance Program (NAPS) and the Tropospheric Ozone Assessment Report (TOAR).
- AQS
- CSN
- IMPROVE
- CASTNET
- NADP - AIRMON, AMON and NTN
- SEARCH (note network ended in 2016)
- AERONET
- NAPS
- TOAR
The meteorological (MET) matching is performed using either bilinear interpolation or nearest neighbor using R functions that acquire observations for the period of the model simulation and perform the pairing and database insertion. The meteorological module primarily relies on observations from the Meteorological Assimilation Data Ingest System (MADIS). MADIS is a widely used, comprehensive and stable meteorological observational archive that has open access. More recent versions of AMET leverage global radiation observations from the Baseline Surface Radiation (BSRN) and SURface RADiation (SURFRAD) budget network. For the evaluation of precipitation, AMET uses Parameter-elevation Regressions on Independent Slopes Model (PRISM) and a new R function that automatically acquires and translates the data for model matching.
The MET module of AMET is designed to work with standard output formats of the Weather Research and Forecasting (WRF; Skamarock et al., 2008) model and the Model for Prediction Across Scales (MPAS; https://mpas-dev.github.io/). Additionally, AMETv1.5 has full support for the intermediate format of meteorology input to CMAQ termed Meteorology-Chemistry Interface Processor (MCIP) for users that do not have access to the raw meteorological model outputs from WRF or MPAS.
The AMET MET module supports the evaluation of both surface and upper-air meteorological fields. The surface fields that can be evaluated by AMET MET include standard 2-m and 10-m variables like temperature, water vapor mixing ratio, wind speed, and wind direction. Other surface-based fields are surface shortwave radiation and precipitation (daily, monthly and annual). Upper-air fields that can be evaluated are potential temperature, relative humidity, wind speed and direction using global rawinsonde soundings of the troposphere.
AMET Database
MySQL is open-source relational database software that can used to organize and quickly access large aggregates of data. Aside from providing a convenient method for storing the observation-model pairs, the use of a relational database allows an entire database to be easily clustered based on any criteria in the database tables (e.g. by date, time of day, geography or the data characteristics). The ability to subset the data quickly is advantageous since model performance often varies spatially/temporally and science questions often require flexible specificity. The MySQL software is free for most research applications and easily installed on all computing platforms.
R Statistical Software
The R project for Statistical Computing, “R”, is an open-source software for statistical computing and graphics based on the “S” language, and compatible on Windows and Unix-based computer platforms. The R software can be ran via the command line interface or through the use of scripts, and contains a large number of functions for computing common statistical metrics and generating graphics (e.g. box plots and scatter plots). Additionally, the R software contains a library of modules, including one to interface with MySQL database servers. All these features make R an attractive computational and plotting software option for AMET.
The information stored in the database is queried using a number of R analysis scripts. Using functions included in the R MySQL library, the database can be queried based on any of the information stored in the database tables. For example, one could limit their data analysis to a single hour of the day, a particular day or week, or to an individual state, a collection of states or the data itself like range of pollutant concentrations or temperature. The AMET scripts include instructions for performing common queries (e.g. queries based on a date range or location). However, anyone familiar with basic database queries could easily formulate their own query criteria to subset the data.
The AMET software includes a collection of prepackaged R scripts that perform analyses like computing standard statistical metrics including bias, error and correlation. These script create routinely used statistical plots such as scatter and time series plots as well as more options tailored for the evaluation of air quality and meteorology models. Table 1 provides a list of the R-based analysis scripts currently available in the AMET AQ and MET modules, along with a brief description of each. In addition to the provided R scripts, more advanced users could use leverage these scripts as examples to create new R scripts tailored for their particular application.
|
Script Name |
AMET Module |
Description |
|---|---|---|
|
MET AQ |
MET & AQ |
Compares air quality and meteorological data. Creates a spatial plot of correlation and a time series plot of bias. |
|
Spatial Surface |
MET |
A series of maps with each providing common statistics used to assess model performance over defined period or data characteristics (Figure 2). |
|
Summary |
MET |
Summary table and diurnal statistics based on defined criteria like a time period, geographic region, time of day, etc. |
|
Time series |
MET |
Time series plots of observed and modeled temperature, mixing ratio, wind speed and direction for a single or batch of observation sites plus quantitative statistics (Figure 3). |
|
Daily Statistics |
MET |
Daily statistics in the form of bar plots and text outputs for a defined batch of data. |
|
Upper-air/RAOB |
MET |
Spatial maps, profile statistics and curtain plots for the evaluation of upper-air meteorology. See Figure 4 for example of curtain plots. |
|
Shortwave Radiation |
MET |
Shortwave radiation evaluation using spatial, time series, diurnal and histogram analyses. |
|
Precipitation |
MET |
Daily, monthly and annual precipitation comparisons using PRISM database. Figure 5 is an example of interactive Leaflets plot output. Also provided are NetCDF model-PRISM and statistics output files. |
|
Box plots |
AQ |
Creates a box plot of observed and modeled quartiles. |
|
Bugle Plot |
AQ |
Creates a “bugle plot,” which plot bias and error as a function of concentration and includes curves representing performance goals. See Figure 6 for an example. |
|
PAVE/VERDI Overlay |
AQ |
Creates a PAVE/VERDI compatible overlay file of observed values. |
|
Spatial Plots |
AQ |
Numerous spatial plots available. See Figure 7 for an example. AMETv1.4 includes interactive spatial plots with map-zoom capability. See Figure 12 for an example. |
|
Scatter Plots |
AQ |
Various scatter plots available. Includes standard model/ob scatter plot, model to model scatter plot, model skill scatter plot, etc. See Figure 8 for an example. |
|
Soccer Plot |
AQ |
Creates a “soccer goal” plot, which plots bias verses error with performance lines in the shape of a soccer goal. See Figure 6 for an example. |
|
Stacked Bar Plots |
AQ |
Creates a stacked bar plot of total PM2.5 mass from networks with speciated PM2.5 data and color codes the contribution from individual PM2.5 species (e.g. sulfate, nitrate, ammonium, total carbon) to the total PM2.5 mass. See Figure 9 for an example. |
|
Stats Plots |
AQ |
Creates spatial plots of NMB, NME, FB, FE and correlation, along with a comma delimited file with various statistics. |
|
Time Series Plots |
AQ |
Various time series plots available (e.g. Model/Ob, Model/Model, time averaged). See Figure 10 for an example. AMETv1.4 includes interactive time series (Figure 11) and Leaflets plots (Figure 12) |
|
Unique Plots |
AQ |
Unique plots available such as the bugle plot, soccer goal plot and Kelly plot. See Figure 13 and Figure 14 for examples of the Kelly plots. |
References
Dennis, R., Fox, T., Fuentes, M., Gilliland, A., Hanna, S., Hogrefe, C., Irwin, J., Rao, S.T., Scheffe, R., Schere, K., Steyn, D., Venkatram, A. (2010). A framework for evaluating regional-scale numerical photochemical modeling systems. Environ. Fluid Mech., 10, doi: 10.1007/s10652-009-9163-2
Skamarock, W.C., Klemp, J.B., Dudhia, J., Gill, D.O., Barker, D.M., Duda, M.G., Huang, X-Y, Wang, W., and Powers, J.G. (2008). A description of the advanced research WRF version 3. NCAR Tech Note NCAR/TN 475 STR, 125 pp, [Available from UCAR Communications, P.O. Box 3000, Boulder, CO 80307.].
Related Publications/Presentations
Appel, K.W., Gilliam, R.C., Davis, N., Zubrow, A., and Howard, S.C. (2011). Overview of the Atmospheric Model Evaluation Tool (AMET) v1.1 for evaluating meteorological and air quality models, Environ. Modell. Softw., 26(4), 434-443.
Galmarini, S., Bianconi, R., Appel, K.W., Solazzo, E., Mosca, S., Grossi, P., Moran, M., Schere, K., & Rao, S.T. (2016). ENSEMBLE and AMET: two systems and approaches to a harmonized, simplified and efficient assistance to air quality models development and evaluation. Atmos. Environ., 53, 51-59.
Appel, K.W., & Gilliam, R.C. (2008). Overview of the Atmospheric Model Evaluation Tool (AMET). 9th Conference on Air Quality Modeling, October 9-10, 2008, RTP, NC.
Appel, K.W., & Gilliam, R.C. (2008). Overview of the Atmospheric Model Evaluation Tool (AMET). 7th Annual CMAS Models-3 Users’ Conference, October 6-8, 2008, Chapel Hill, NC.
Example Analyses
