sweater: Speedy Word Embedding Association Test and Extras Using R

The goal of this R package is to detect associations among words in word embedding spaces. Word embeddings can capture how similar or different two words are in terms of implicit and explicit meanings. Using the example in Collobert et al. (2011), the word vector for “XBox” is close to that for “PlayStation”, as measured by a distance measure such as cosine distance. Word embeddings can also be used to study associations among words that are otherwise difficult to detect. For instance, Jing & Ahn (2021) used word embeddings to study how Democrats and Republicans are divided along party lines about COVID-19.

The same technique can also be used to detect unwanted implicit associations, or biases. For example, Kroon, Trilling, & Raats (2020) measure how close the word vectors for various ethnic group names (e.g. "Dutch", "Belgian" , and "Syrian") are to those for various nouns related to threats (e.g. "terrorist", "murderer", and "gangster"). These biases in word embedding can be understood through the implicit social cognition model of media priming (Arendt, 2013). In this model, implicit stereotypes are defined as the "strength of the automatic association between a group concept (e.g., minority group) and an attribute (e.g., criminal)." (Arendt, 2013, p. 832) All of these bias detection methods are based on the strength of association between a concept (or a target) and an attribute in embedding spaces.
Previously, the software of these methods is only available piecemeal as the addendum of the original papers and was implemented in different languages (Java, Python, etc.). sweater provides several of these bias detection methods in one unified package with a consistent R interface (R Core Team, 2021). Also, sweater provides several methods that are implemented in C++ for speed and interfaced to R using the Rcpp package (Eddelbuettel, 2013) 1 .

Usage
In this section, I demonstrate how the package can be used to detect biases and reproduce some published findings.

Word Embeddings
The input word embedding w is a dense m × n matrix, where m is the total size of the vocabulary in the training corpus and n is the vector dimension size. sweater supports input word embeddings, w, in several formats. For locally trained word embeddings, output from the R packages word2vec (Wijffels, 2021), rsparse (Selivanov, 2020) and text2vec (Selivanov, Bickel, & Wang, 2020) can be used directly with the packages primary functions, such as query 2 . Pretrained word embeddings in the so-called "word2vec" file format, such as those obtained online 3 , can be converted to the dense numeric matrix format required with the read_word2vec function.
The package also provides three trimmed word embeddings for experimentation: googlenews (Mikolov, Sutskever, Chen, Corrado, & Dean, 2013), glove_math (Pennington et al., 2014) , and small_reddit (An, Kwak, & Ahn, 2018). Query sweater uses the concept of a query (Badilla et al., 2020) to study associations in w and the ST AB notation from Brunet, Alkalay-Houlihan, Anderson, & Zemel (2019) to form a query. A query contains two or more sets of seed words (wordsets selected by the individual administering the test, sometimes called "seed lexicons" or "dictionaries"). Among these seed wordsets, there should be at least one set of target words and one set of attribute words.
In the situation of bias detection, target words are words that should have no bias and usually represent the concept one would like to probe for biases. For instance, Garg, Schiebinger, Jurafsky, & Zou (2018) investigated the "women bias" of occupation-related words and their target words contain "nurse", "mathematician", and "blacksmith". These words can be used as target words because in an ideal world with no "women bias" associated with occupations, these occupation-related words should have no gender association.
Target words are denoted as wordsets S and T . All methods require S while T is only required for Word Embedding Association Test (WEAT). For instance, the study of gender stereotypes in academic pursuits by Caliskan et al. (2017) used S = {math, algebra, geometry, calculus, equations, ...} and T = {poetry, art, dance, literature, novel, ...} In the situation of bias detection, attribute words are words that have known properties in relation to the bias. They are denoted as wordsets A and B. All methods require both wordsets except Mean Average Cosine Similarity (Manzini, Lim, Tsvetkov, & Black, 2019). For instance, the study of gender stereotypes by Caliskan et al. (2017) used A = {he, son, his, him, ...} and B = {she, daughter, hers, her, ...}. In some applications, popular off-the-shelf sentiment dictionaries can also be used as A and B (e.g. Sweeney & Najafian, 2020). That being said, it is up to the researchers to select and derive these seed words in a query. However, the selection of seed words has been shown to be the most consequential part of the entire analysis (Antoniak & Mimno, 2021;Du, Fang, & Nguyen, 2021). Please read Antoniak & Mimno (2021) for recommendations. Table 1 lists all methods supported by sweater. The function query is used to conduct a query. The function calculate_es can be used for some methods to calculate the effect size representing the overall bias of w from the query.

Example 2
Word Embedding Association Test (WEAT) (Caliskan et al., 2017) requires all four wordsets of S, T , A, and B. The method is modeled after the Implicit Association Test (IAT) (Nosek, Greenwald, & Banaji, 2005) and it measures the relative strength of S's association with A to B against the same of T . The effect sizes calculated from a large corpus, as shown by Caliskan et al. (2017), are comparable to the published IAT effect sizes obtained from volunteers.
In this example, the publicly available GLoVE embeddings made available by the original Stanford Team (Pennington et al., 2014) were used. In the following example, the calculation of "Math vs Arts" gender bias in Caliskan et al. (2017) is reproduced. In this example, the positive effect size indicates the words in the wordset S are more associated with males than are the words in wordset T .