SimplyAgree: An R package and jamovi Module for Simplifying Agreement and Reliability Analyses

Accurate and reliable measurements are critical to quantitative research efforts. Based on citation counts of Bland & Altman (1986) and Weir (2005) alone1, researchers appear to highly value methods to quantify the accuracy and reliability of measurement tools used for research. This article introduces the SimplyAgree R package and jamovi module as userfriendly solutions for estimating agreement and reliability (R Core Team, 2020; The jamovi project, 2021) for continuous measurements. Updates and additional details on SimplyAgree can be found on the package’s website. SimplyAgree was created for applied physiologists to use when evaluating different physiological measurements (e.g, comparing the measurement of oxygen consumption between two competing gas analyzers). However, the functions within SimplyAgree can be utilized by any researcher wanting to evaluate agreement between two continuous measurements, or for evaluating the reliability of a continuous measure2.


Statement of Need
A number of new methods have been developed in the past three decades to improve the calculation of the limits of agreement (Lin, 1989;Shieh, 2019;Zou, 2011) and other measures of measurement reliability (Carrasco, Phillips, Puig-Martinez, King, & Chinchilli, 2013;Weir, 2005). However, to the author's best knowledge, statistical software -particularly open source software -to implement these statistical analyses are lacking. While some software may provide the limits of agreement analysis outlined by Bland & Altman (1986, 1999, few, if any, account for multiple observations within the same research subject (Zou, 2011) or include hypothesis tests of agreement (Shieh, 2019). blandr (Datta, 2017) and BlandAltmanLeh (Lehnert, 2015) exist to aid the creation of Bland-Altman type plots, but do not offer the tests detailed by Shieh (2019) or Zou (2011). MethComp (Carstensen, Gurrin, Ekstrøm, & Figurski, 2020) is the most comprehensive with regards to limits of agreement but also lacks the methods outlined by Shieh (2019). The cccrm package (Carrasco et al., 2013) does already provide concordance correlation coefficients as output, and is the basis of the concordance calculations within SimplyAgree. Lastly, psych (Revelle, 2020) and ICC (Wolak, Fairbairn, & Paulsen, 2012) provide some measures of reliability but lack the ability to calculate standard error of the measurement. Moreover, only blandr has a jamovi module but the capabilities are limited to just the traditional Bland-Altman limits.
SimplyAgree is also created to be easy to use for those with limited programming experience. The output from the functions is intentionally verbose and typically provides more than 1 measure of agreement or reliability. The hope is that researchers who use these functions will not limit their interpretations of the results to 1 numeric value, but instead look at the totality of the results (estimates and confidence intervals) for their interpretation.
Additionally, many researchers may not have the skills necessary to write the code, from scratch, in order to implement many of the newest techniques. The jamovi project (2021) is a open source statistical platform that provides a graphical user interface (GUI), and therefore is an accessible source for researchers, or even students, without coding experience. Therefore, a jamovi module of SimplyAgree was also created in order to reach those researchers who may not have the coding expertise required to effectively use the R package.

Current R Capabilities
The R package SimplyAgree, currently v0.0.2 on the comprehensive R archive network (CRAN), implements a number of useful agreement and reliability analyses.
The current release of the R package can be downloaded directly from CRAN in R: install.packages("SimplyAgree") Or, the developmental version, can be downloaded from GitHub: devtools::install_github("arcaldwell49/SimplyAgree") There are 2 vignettes that document the major functions within the package that can be found on the package's website (https://aaroncaldwell.us/SimplyAgree). Overall, there are 6 fundamental functions, all with generic plot and print methods, within the R package: 1. agree_test: Simple Test of Agreement. This is function performs agreement analyses on two vectors of the same length, and is designed for analyses that were described by Bland & Altman (1986, 1999. In addition to providing the traditional Bland-Altman limits of agreement, the function provides a hypothesis test (Shieh, 2019), and provides the concordance correlation coefficient (Lin, 1989).
2. agree_reps: Test of Agreement for Replicate Data. This function provides the limits of agreement described by Zou (2011) for data where the mean, per subject, does not vary. In addition, the concordance correlation coefficient, calculated by U-statistics, is also provided in the output (Carrasco et al., 2013).
3. agree_nest: Test of Agreement for Nested Data. This function provides the limits of agreement described by Zou (2011) for data where the mean, per subject, does vary. Similar to the replicate data function, the concordance correlation coefficient, calculated by U-statistics, is provided in the output (Carrasco et al., 2013).

loa_mixed:
Bootstrapped Limits of Agreement for Nested Data. This function calculates limits of agreement using a non-parametric bootstrap method, and can allow the underlying mean to vary (replicate data) or not (nested data).

blandPowerCurve:
Power Analysis for Bland-Altman Limits of Agreement. This function implements the formula outlined by Lu et al. (2016). This allows for power calculations for the Bland & Altman (1999) limits of agreement. The function find_n can then be used to find the sample size at which adequate power (defined by the user) is achieved.
6. reli_stats: Reliability Statistics. This function calculates and provides as output the statistics outlined by Weir (2005). This includes an array of intraclass correlation coefficients, the coefficient of variation, and the standard error of measurement.

Current jamovi Capabilities
The jamovi module can be added to the jamovi directly from the "add module" tab in the GUI. The SimplyAgree module is then available on the main menu, and within it there are three analysis options. The three analysis options essentially enable jamovi users to complete some of the same analyses available in the R package.
1. The simple agreement analysis incorporates the agree_test function. Users have the option of including the concordance correlation coefficient, and plots of the data. 2. The nested/replicate agreement analysis uses the agree_nest and agree_reps function to perform the analyses. The agree_reps function is used if "Assume underlying value does not vary?" is selected; otherwise agree_nest is used. 3. The reliability analysis utilizes reli_stats to calculate reliability statistics.