Learning Simulator: A simulation software for animal and human learning

Summary Learning Simulator is a software for simulating learning in animals and humans, as studied for example in experimental psychology and animal cognition research. It is primarily intended for computational and behavioral biologists, ethologists, and psychologists, and it can also be useful to students and teachers in these fields


Introduction
Learning Simulator was developed to study learning in animals and humans. The current version implements associative learning (AL) and reinforcement learning (RL) algorithms, apt to study instrumental (operant) learning and Pavlovian (classical) learning (Bouton, 2016;Pearce, 2013), including in complex situations such as social learning or maze learning. A plugin system to add more learning mechanisms is planned for a future version.
The simulator uses a common framing of learning that comprises a subject interacting with an environment. The environment presents a stimulus to the subject, and the subject responds with a behavior. As a result, the environment presents the next stimulus that the subject responds to, and so on. See Figure 1.

Figure 1:
The subject and the world can be seen as two interacting dynamical systems, where the state variables in the subject determine the probabilities for its behaviors (the subject's output), and each behavior from the subject puts the environment in a state that determines (probabilistically) its output stimulus.
The stimuli that the environment presents and the behaviors that the subject can exhibit are pre-defined by the user of the program. Each stimulus is given a reinforcement value (corresponding to genetically determined values in biological organisms). A stimulus such as food would typically have a positive value, while the perception of harm to the body would have a negative value.
As seen in Figure 1, the consequence of responding with a behavior (say B) to a stimulus (say S) is that the subject meets the next stimulus (say S ′ ): Learning algorithms can then use the reinforcement value of S ′ as an indication of the quality of the response B to S. Specifically, this can be accomplished by updating one or more of the subject's memory state variables. These state variables control the probabilities of future responses: if the response B to stimulus S leads to a reward (a stimulus with high reinforcement value), the subject will be more likely to respond with B the next time it faces S.
The user of Learning Simulator specifies in a text script how the output stimulus from the environment depends on the subject's response to the previous stimulus. This script also specifies the values of all parameters used in the learning process. The simulation script, written in a simple and well-documented scripting language, is the only input to Learning Simulator, facilitating reproducible workflows. The script also specifies how to visualize the simulation data, for example how a memory state variable changes during the simulation. Learning Simulator can also export results to CSV files.
More information is available at https://www.learningsimulator.org.

Applications of associative learning
Associative learning theory has a rich tradition of computational modeling, which has recently converged with RL learning models from machine learning (Sutton & Barto, 2018). During the last decade or so, AL and RL algorithms have proven increasingly powerful.
Firstly, when applied to deep neural networks, RL has been used to teach computers to find optimal play and achieve human-level skills in chess (Silver et al., 2017) and the Chinese board game Go (Silver et al., 2016).
Secondly, behaving optimally (or near-optimally) is central to animals' adaptation to their environment. AL and RL can provide explanations for a wide range of learning phenomena in both humans and animals (Dayan & Niv, 2008;Enquist et al., 2016;Ghirlanda et al., 2020;Niv, 2009). Indeed, investigating predictions about animal and human behavior is the main motivation behind Learning Simulator, with the goal to understand biological learning and its numerous applications to animal welfare and training (McGreevy & Boakes, 2011), and to human health (Bernstein, 1999;Haselgrove & Hogarth, 2013;Schachtman & Reilly, 2011).

Statement of need
As a result of a strong theoretical tradition, and of the many application areas, there are now many AL and RL models with varying properties and varying predictive power in different environments.
This situation has created a need for a general simulation software offering a convenient way to investigate different learning models. The first aim of our software is to fulfil this need.
The second aim is to provide a generic, flexible way to describe unambiguously the design of learning experiments or, more generally, the features of any learning scenario.
The main advantage of Learning Simulator is its simple scripting language that provides a way to explore and understand learning mechanisms in a multitude of learning scenarios, and investigate the effects of varying their underlying parameters.
A further strength is the simplicity with which even complex learning environments can be specified, such as psychological experiments with complex dependencies between stimuli and behaviors. The scripting language has been developed to be useful to any researcher of learning phenomena -not necessarily computer programmers.
Our software has been used in scientific publications (Ghirlanda et al., 2020;Lind, 2018;Lind et al., 2019)  An open source license and an accessible scripting language enable further scientific exploration of learning phenomena by students and experts alike within the fields of biology, ethology, psychology, and neuroscience.

State of the field
Other simulating software either specialize in one specific mechanism (Schultheis et al., 2008a(Schultheis et al., , 2008bAlonso et al., 2012) or only include models of classical conditioning (George, 2019;Harris & Livesey, 2010;Thorwart et al., 2009), or provide only one learning algorithm or environment (Q-Learning Simulator, n.d.).

Repository
The program is written in Python and its source code repository is hosted on GitHub. Its documentation is hosted on Read the Docs. The repository incorporates continuous integration with automatic build of the documentation, linting, and execution of a test suite along with code coverage measurement.