PROFFASTpylot: Running PROFFAST with Python

Measurements of atmospheric greenhouse gas (GHG) concentrations are important to assess the effect of climate change mitigation policies. Additionally, climate models depend on a precise knowledge of greenhouse gas abundances and emissions. A variety of measurement methods are addressing these needs. The Collaborative Carbon Column Observing Network (COCCON) was established in 2019 as a supporting framework for users of the portable Fourier-Transform spectrometers EM27/SUN. The spectrometers measure precisely and accurately GHG column abundances from near-infrared solar absorption spectra. To ensure common quality standards across the COCCON


Statement of Need
The EM27/SUN solar Fourier-Transform Infrared (FTIR) spectrometer was developed by the Karlsruhe Institute of Technology (KIT) in collaboration with Bruker (Gisi et al., 2011;Frank Hase et al., 2016), was commercialized in 2014 and is in wide use today.GHG city emissions (Dietrich et al., 2021;F. Hase et al., 2015;Tu et al., 2022;Vogel et al., 2019), as well as long-term trends at selected sites (M.M. Frey et al., 2021;Mermigkas et al., 2021) have been investigated.A further goal is the validation of space borne GHG measurements (Alberti, Tu, et al., 2022;Tu et al., 2020).
The publication by M. Frey et al. (2019) forms the starting point of instrumental quality assurance by COCCON's central facility at KIT.Since then many more EM27/SUN spectrometers have been characterized.Recently, Herkommer et al. (2023) exploited the portability of the EM27/SUN to improve the inter-calibration of the Total Carbon Column Observing Network (TCCON) (Wunch et al., 2015), further interlinking the TCCON and the COCCON.
Recent developments improved the operability of the measurements (Aigner et al., 2023;Heinle & Chen, 2018).PROFFASTpylot targets the operation of the retrieval.
The software PROFFAST (Frank Hase, 2023;Sha et al., 2020) is required by the COCCON for processing the raw measurements (interferograms) collected by the EM27/SUN spectrometers.It is split into three program parts: 1. PROFFASTpreprocess: Conversion from the raw interferograms to atmospheric absorption spectra.2. PROFFASTpcxs: Tabulation of daily columnar absorption cross-sections as a function of the air mass for daily specific atmospheric conditions.3. PROFFASTinvers: Inversion of the column-averaged trace gas abundances.
The manual operation of PROFFAST has the following workflow: For each of the above described steps, the user has to create input files with the relevant parameters.A list of interferograms to be processed and specific input parameters are required for preprocess.Secondly, pcxs requires the specification of the atmospheric conditions.Finally, for invers the output generated by the previous steps has to be listed.PROFFAST creates several output files; only a single day can be processed at a time.The task repetition and file organization makes the processing of longer measurement series work intensive and prone to application errors.
The following requirements are addressed by PROFFASTpylot: • Significant improvement of usability by enabling a single set of input parameters and the simultaneous processing of many measurement days.• Untangling of raw data, processing files and output.
• Reduction of application errors by introducing various cross-checks and user warnings.
• Flexibility to allow experimental use cases besides the COCCON standard.
Already during development we received many comments and questions from the global COCCON user community indicating the great interest in this tool.PROFFASTpylot has already been used by Schmid (2023) and Herkommer et al. (2023).

Functionality and Design
In this section the main functionality and structuring of the program are explained.

Main functionality
The program is an interface for PROFFAST.Figure 1 gives an overview of the main functionalities.• Concatenate the final data of all processed days to a single output file.

Adaptability and error prevention
To ensure a simple user experience and fast error detection several measures have been taken: • The empirical instrumental parameters (ILS parameters) (Alberti, Hase, et al., 2022) are taken automatically from an internal list.• For auxiliary data, cross-checks are implemented that generate a warning or a controlled program stop (e.g.checking the correct location of atmospheric a-priori files).• Automatic handling of different time zones in interferograms and auxiliary data.
• Correct handling of various pressure records (different sampling intervals or data formats).

Design
The PROFFASTpylot consists of three layers which inherit from each other and an independent fourth class: The first layer is called prepare.This part creates a list of all days to be processed and creates the PROFFAST input files.This includes a call of the independent pressure class which includes the functionality to read, check and interpolate the pressure records.
The filemover is responsible for providing the necessary input data for each part and to hand over intermediate files to the next step.
The pylot interacts with the user: It contains methods to start the individual PROFFAST parts or to run them subsequently in a single request.