Types of Experiments

EXPEDITED ONLINE EXPERIMENT
Non-interactive, short experiment (e.g. 15min)

The expedited procedure is suggested for short experiments conducted with a large number of participants. The participants can complete the task whenever they want before the deadline or until the recruitment target has been reached.

RECRUITMENT

- Create an experiment with one session on ORSEE (same date as the deadline for the experiment) – registering for the session on ORSEE does not have any effect

- Assign and invite subjects, but edit the invitation email such that:

• It does not contain the link to the ORSEE calendar
• It contains information about the expected duration of the task and the range of rewards (similarly to the basic information typically provided on mTurk) and also the payment method(s)
• It should indicate that the study can be completed by MM/DD/YY, and the link will not be active after. Also, if you have a target number of participants, you should mention that the link will be closed after the recruitment goal has been reached
• It contains a link directly to the study (e.g. Qualtrics or oTree). At the beginning of the study, collect information for the payment (e.g. email for Amazon, or account for Venmo)

- If you want to send a new invite (only to those who did not complete the study already), the participants can be manually added to the sessions scheduled on ORSEE to record how many participants completed the task (but this is not necessary if you do not plan to send a second invite).

PAYMENT

- The payment can be delivered using Amazon gift cards or Venmo transfer. These methods allow to pay any amount to the participants promptly and without personal interactions

- Other methods can be used, but you should mention them explicitly in the recruitment and verify that you can get reimbursed with that (ISERP accepts the ones above, as well as PayPal, ApplePay and GooglePay)

- Experimenters can always decide to pay more than the amounts indicated below

- Minimum show-up fee $1. Regardless of the duration of the experiment, guaranteed for every participant who completes the task

- Minimum average payment $15/hour. Participants are on average receiving this payment or more ($2.5 / $5 / $7.5 for an experiment planned for 10 / 20 / 30min)

- Minimum guaranteed $8/hour. Participants are guaranteed to receive $1.3 / $2.7 / $4 after an experiment planned for 10 / 20 / 30 minutes.

​​​​​​STANDARD ONLINE EXPERIMENT
Non-interactive, long experiment (e.g. 60min)

The expedited procedure is suggested for long experiments conducted with a relatively small number of participants. The participants can complete the task whenever they want before the deadline or until the recruitment target has been reached.

RECRUITMENT

• Create an experiment with one session on ORSEE (same date as the deadline for the experiment) – registering for the session on ORSEE does not have any effect
• Assign and invite subjects, but edit the invitation email such that:
  • It does not contain the link to the ORSEE calendar
  • It contains information about the expected duration of the task and the range of rewards (similarly to the basic information typically provided on mTurk) and also the payment method(s)
  • It should indicate that the study can be completed by MM/DD/YY, and the link will not be active after. Also, if you have a target number of participants, you should mention that the link will be closed after the recruitment goal has been reached
  • It contains the link to a 1-min participation survey that requires 1) Columbia UNI and name (as it appears in their CU badge), 2) information about the payment method (Amazon/Venmo), 3) the statement “I commit to complete the by MM/DD/YY at hh.mm EST (New York time)” (date and time of the experiment)
  • If needed, the participation survey can contain additional questions (see example at the end of the document)
  • Within 24 hours after filling the participation survey, the experimenter needs to respond (automatically or manually)
    • If participation is allowed, sending the link to the experiment and the deadline for its completion
    • If participation is not allowed (too many participants, conditions not met, etc.), saying that they cannot participate in the current study
  • When the desired number of participants is reached, you should close the participation survey
  • If someone registers on ORSEE but not completes the participation survey, send them a reminder (and remove them from ORSEE if needed)
  • If someone completes the surveys but does not register on ORSEE, you should manually add them to the session
• Send a reminder 12 hours before the deadline to those who completed the participation survey but did not complete the experiment (it is up to you whether to be flexible with the deadline or not)
• When you want to stop the data collection, close the link (so no one will play the experiment without being paid for it)
• If you want to send a new invite (only to those who did not complete the study already), the participants can be manually added to the sessions scheduled on ORSEE to record how many participants completed the task (but this is not necessary if you do not plan to send a second invite).

PAYMENT

Same details (methods and amount) as for the Expedited Online Experiments.

VIRTUAL LAB EXPERIMENTS
Interactive (e.g. conducted with Zoom)

UC Santa Cruz Lab wrote these guidelines that are particularly useful for “virtual lab” experiments
https://docs.google.com/document/u/0/d/1M9Dnz66eTQJm5f7ahM34sji8Ikvrf4SmdgI000Dl6gc/mobilebasic?usp=gmail_thread

RECRUITMENT

- Create one or multiple sessions on ORSEE as if it was a usual lab experiment

- Assign and invite subjects, but edit the invitation email such that:

• It indicates that the experiment will be conducted on Zoom and participants need to have access to a computer with a stable internet connection (they cannot participate with a mobile phone or tablet)
• It contains information about the expected duration of the task and the range of rewards (similarly to the basic information typically provided on mTurk) and also the payment method(s)
• It still contains the link to the ORSEE calendar (because we have a limited capacity for each session)
• It contains the link to a 1-min participation survey that requires 1) Columbia UNI and name (as it appears in their CU badge), 2) information about the payment method (Amazon/Venmo), 3) the statement “I commit to connect through Zoom to participate to the experiment on MM/DD/YY at hh.mm EST (New York time)” (date and time of the experiment)
• If needed, the participation survey can contain additional questions (see example at the end of the document)
• Within 24 hours after filling the participation survey, the experimenter needs to respond (automatically or manually)
  • If participation is allowed, sending additional information about the session (Zoom link, if they need to have a document or CU badge ready at the beginning of the session, etc.)
  • If participation is not allowed (too many participants, conditions not met, etc.), saying that they cannot participate to the current study
• When the desired number of participants is reached, you should close the participation survey
• If someone registers on ORSEE but not completes the participation survey, send them a reminder (and remove them from ORSEE if needed)
• If someone completes the surveys but does not register on ORSEE, you should manually add them to the session

- Send a reminder 12 hours before the experiment (you can use ORSEE reminder function, but make sure you modify the text accordingly, e.g. include the Zoom link)

PAYMENT

Methods of payment as for the Expedited Online Experiments.

Average, minimum, and guaranteed amounts (including show-up fee) as for the Lab Experiments.

NO DECEPTION

- Experiments funded by CELSS must indicate the no-deception policy in the instructions: use the paragraph you find on the website https://celss.iserp.columbia.edu/research-projects/rules

PAYMENT

- Online surveys and experiments using online subject pools (e.g., MTurk) must pay a minimum of $1 and an average of $15 per hour.