Resources for Online Experiments

This page contains resources for online experiments. These software and platforms have different advantages and disadvantages, for example the difficulty to start creating a new experiment and the degree of customizability that they offer. You can find here a selection additional resources developed by other labs.

DISCLAIMER: These resources seem interesting and potentially useful for experimenters; however CELSS has not checked all of them, does not endorse them, and is not responsible for their use.

We group the software to create online experiments into two groups, based on whether they allow the design of interactive experiments. This partition is fuzzy, as some software is not designed for interaction but it can be achieved with good programming skills.

You can also check these guides for survey experiment (Chicago Booth) and for "virtual lab" experiments (UCSC)

Online Experiment (allow interaction)

oTree is a Python-based platform used for online tasks. It can be used for behavioral experiments (also for the laboratory), multiplayer strategy games, surveys, and quizzes, especially those that require customized or dynamic functionality.

oTree is the natural choice for online interactive experiments, but it requires some programming skills for the background calculations (Python) and interface (HTML).

You can find more information on the page "oTree"

Qualtrics offers a simple interface to create online surveys. It is often used with Amazon Mechanical Turk (MTurk) to distribute surveys or simple tasks (non-interactive) over a large number of potential participants.

With some additional programming skills, you can also design simple interactive experiments using Smartriqs (see details in the page below)

You can find more information on the page "Qualtrics and MTurk"

zTree is a widely used software package for developing and carrying out economic experiments in the lab.

Using the software zTree unleashed, you can run experiments online as if they were in the lab (see details in the page below)

You can find more information on the page "zTree"

LIONESS Lab is a web-based platform for interactive online experiments. The development of experiments by stages, and adding elements to those stages, largely follows the logic of zTree. 

It can be used for interactive online experiments.

More information:

Online Experiment (do not allow interaction)

jsPsych is a JavaScript library for running behavioral experiments in a web browser. The library provides a flexible framework for building a wide range of laboratory-like experiments that can be run online.

More information:

PsychoPy is a free cross-platform package allowing you to run a wide range of in the behavioral sciences.

More information:

LabVanced is an experiment builder, that allows you to build experiments (graphical interface, no coding necessary, basic features for free, premium features require a license) and collect data with their panel (pay-per-respondent).

More information:

Gorilla is an experiment builder, that allows you to build experiments for free (graphical interface, no coding necessary) and collect data with their panel (pay-per-respondent).

More information:

PsyToolkit is a free-to-use toolkit for demonstrating, programming, and running cognitive-psychological experiments and surveys, including personality tests. PsyToolkit can be used for academic studies, for student projects, and for teaching cognitive and personality psychology.

More information:

Preregister your research

AEA RCT Registry website:

The American Economics Association created a RCT (Randomized Controlled Trials) registry for economics and other social sciences. As RCT become more numerous, a central registry on which trials are on-going or complete (or withdrawn) becomes important for various reasons: as a source of results for meta-analysis; as a one-stop resource to find out about available survey instruments and data. Registration is free and you do not need to be a member of the AEA to register.

AsPredicted website:

An "AsPredicted" is a standardized pre-registration that requires only what's necessary to separate exploratory from confirmatory analyses. You will easily generate a pre-registration document that takes less effort to evaluate than it takes to evaluate the published study itself.

COS (Center for Open Science) website:

Preregistration separates hypothesis-generating  (exploratory) from hypothesis-testing (confirmatory) research. Both are important. But the same data cannot be used to generate and test a hypothesis, which can happen unintentionally and reduce the credibility of your results. Addressing this problem through planning improves the quality and transparency of your research. This helps you clearly report your study and helps others who may wish to build on it.

OSF (Open Science Framework) website:

By writing out specific details such as data collection methods, analysis plans, and rules for data exclusion, you can make important decisions early on and have a clear record of these choices. This can help reduce biases that occur once the data are in front of you.

G*Power - computes effect sizes and displays graphically the results of power analyses.

Sample Size Calculator - Determines the minimum number of subjects for adequate study power.

DeclareDesign - a set of software tools for describing, assessing, and conducting empirical research (requires R).

Matlab Sample Size resources (require Matlab).

Launch your Online Experiment

Amazon Mechanical Turk (MTurk) is a marketplace used to outsource computer labor to humans. Tasks range from several seconds (e.g. identify a photo) to hour-long surveys.

You can find more information on the page "Qualtrics and MTurk", including a tutorial and additional resources (e.g. Cloud Research tools for MTurk experiments). 

Prolific is a platform for online market research that allows experimenters to filter participants based on demographic information or generate a US representative sample.

More information:

Lucid is a platform for online market research that uses independent quality checks to select respondents and allows experimenters to filter participants based on demographic information.

More information:

Pavlovia is a platform used to run, share, and explore experiments online. Though it was originally conceived as a repository and launch platform for PsychoPy experiments, its open architecture makes it possible to support other open-source tools, such as jsPsych and lab.js.

More information:

EMPanel Online supplies B2B respondents, including business professionals, healthcare specialists (doctors, specialists, nurses), and other specialized professional groups (engineers, truckers, teachers, artists).

More information:

Volunteer Science is a platform for performing online behavioral research. They provide a software to develop experients and a platform to recruit participants.

You can find more information on the Volunteer Science website or on this separate page.

Other platforms:

- Qualtrics has its own panel of respondents

- SurveyMonkey also has its own panel of respondents

- Dynata can be used to reach a balanced sample or "hard-to-reach" individuals

- Clickworker is similar to mTurk and has a pool of subjects that is active mostly in tasks involving audio and natural language processing

- SWG (research institute with a subject pool used for market surveys, opinion and institutional polls)

- CE&Co (research and market analysis)