Volunteer Science

DISCLAIMER: This service seems interesting and potentially useful for experimenters; however CELSS has not checked them, does not endorse them, and is not responsible for their use.


Volunteer Science (website: https://volunteerscience.com/) is a platform for performing online behavioral research. They provide a platform and software for social and behavioral scientists to create online experiments and recruit subjects from anywhere in the globe.

In the past five years, over 100 researchers have run half a million studies using Volunteer Sciences (VS). VS specializes in multi-person experiments, ranging from pairs of bargaining subjects to 40-person social dilemmas, meaning most economic, team, and network experiments can be done with groups anywhere in the world. VS also supports a variety of subject recruitment strategies including integrating with Mechanical Turk for incentivized research and existing subject pools. VS also has our own crowd of 50,000 volunteers. Finally, they offer tutorials and training to teach students and faculty how to create online studies from scratch using basic web languages.

Additionally to the Q&A section on Volunteer Science, you find below some extra questions that we asked for you

Our current researchers come from a broad swath of the social and behavioral sciences. But, to date, our economic-style researchers have been from or trained in business schools rather than economics departments. That said, we have performed or facilitated a lot of incentivized experiments typically either through Mechanical Turk or with our Ambassadors. Right now, we're working on two incentivized studies. One is with Computer Scientists at Bar-Ilan University using students in a large undergrad class who all get paid a base rate and the top 10% get a bonus payment. The second is with political scientists at Ohio State who are using Qualtrics' recruitment services to recruit pairs into a two-person bargaining game with a base pay and bonus for the winner. Lastly, we have done one incentivized study with our volunteers where the incentive was a $1500 donation to a nonprofit of the winner's choosing. 

For managing incentives, we provide a couple of tools. First, we have API-level integration with MTurk that makes creating and managing HITS, validating data, setting payment, and managing qualifications much easier than doing it manually through the AMT system. For ambassadors, we have internal accounting processes for validating participation and tracking payments that vary because ambassador projects have varied substantially (more on this in #3). If you have your own subjects, either in-lab or in-class, we have done one of two things. First, we've built the receipt into the study itself. Participants get the receipt in the game and submit it via email or print it out. Second, we have a secure certificate system that tells you whether someone participated in a study with a link to verify the certificate. If people are unable to come into the lab, researchers either use Amazon gift cards or paypal to transfer the funds. 

To your question about research standards, we provide research teams with the discretion to choose their own standards, methods, and techniques. We're like Qualtrics in that way. You can create a deceptive study or you can not. It's completely up to you. You're the researchers.

The types of studies on our website range dramatically. What you see on the front page are studies we think volunteers will want to participate in on their own. Hence they're short and tend towards more fun kinds of activities. But, we're currently running a 14-person game from Gallup that takes an hour - https://volunteerscience.com/about/500/. We also have an API for collecting participants' social media data on Facebook and Twitter. There are psychologists at Harvard working on a study where we're recording people's faces to build a machine-learning model of mindfulness. Our role at Volunteer Science is to provide researchers with the technical means to do studies you can't do on Qualtrics (or zTree/oTree necessarily). You can create what you want and run it with whatever subjects you want. However, we do perform some quality assurance on studies that get posted to the front page to make sure they're are appropriate for our volunteers.

What is the ambassador program? The ambassador program was built for groups. I mean that literally. We developed the program as part of a grant to develop group recruitment methods. Ambassadors are volunteers who are trained to recruit groups of people to participate. Most ambassadors develop a network of volunteers they are able to mobilize for an experiment. Sometimes these are friends from class or coworkers at work and sometimes ambassadors put up a table on campus or pass out fliers. That is to say, recruitment differs. We have ambassadors in the U.S., Nigeria, and China (an artifact of the grant). In the U.S. these are most often international students looking to get their first work experience with an American company. As volunteers, they get to establish their resume and we get subjects. The ambassadors in Nigeria and China are older students and young professionals (again an artifact of the grant). For an individual study, we first train ambassadors in how the study works and how to validate completion. This ensures the ambassador is able to identify any bugs, document them so the researcher can fix them, and mitigate if needed to help participants complete the study.  For incentivized experiments, the ambassador is given cash to give to the participant and does the paperwork. On the back side, we provide quality assurance that the payment was valid. Finally, ambassadors are paid a bounty for every group they recruit largely based on the difficulty of the recruitment (larger studies with longer time-commitments have larger bounties). We typically pay $1/person per completion for something like a 2 or 3 person study that's 5 minutes long. For the 14-person Gallup study, the bounty is $10/person per completion.

How does the program work financially? There are typically four types of costs involved in an ambassador project. The first cost is recruitment. If a project needs more ambassadors than we have, we have to recruit them. The cost of this varies by the ambassadors needed, but it's roughly $1,000 for every 10 ambassadors. Right now we have 20 ambassadors. The second cost is for training ambassadors to do the study. This varies by study complexity. If it's just a Qualtrics survey, then it would probably be free. A 2-player bargaining game, even if it's an hour, would likely be less than $500. A 20 person mobile game would be $1000-$2000 in training. The third cost is management which covers bug reporting, payment validation, transferring cash, etc. Again, this can vary by study size, duration, and complexity. But the biggest driver is stability. A simple study that rarely breaks, even if it's 10-20 people, will cost $500/month or less to manage. A study that breaks all the time, even for 2 people, can cost up to $1000/month if you're running a lot of subjects. Fourth, is the cash transfers themselves. Volunteer Science handles all of the cash. Our ambassadors know we'll pay them. This allows us to minimize cash transfer costs because our ambassadors are willing to wait to get paid (we're not doing a dollar to Yuan transfer every time a subject gets paid).

The rate of subject recruitment depends on the method of recruitment. We have an email list of 47,000 volunteers around the world and can bring in a couple hundred within a day with just one email. But the sample is very diverse geographically and demographically. You can put a study up on our main page and get anywhere from 5-50 people a day depending on the quality of the experience. If you're looking for multiplayer though, passive web traffic usually won't do it. We've had a lot of success partnering with the webgames subreddit. We've linked to our games there for free and gotten 2,000-5,000 participants over the next 72 hours (depending on game quality). We've also used Facebook ads to do groups. We spent $400 in ads over two hours and got 69 completed sessions of a three-player experiment. These were all volunteers. If you want to incentivize groups and recruit quickly, MTurk is still the easiest and fastest way to do it (though attrition is a killer on Mturk and I recommend ambassadors for any study needing more than 4 or 5 concurrent users).

Volunteer Science COVID announcement

Volunteer Science is offering free access to the platform to academic researchers through June 30th. Click the title for the full announcement.