Online Data Collection

Recommended experimental protocol:

  • 请确保您的IRB协议涵盖您的实验过程
  • Use Qualtrics 编写调查/实验代码，然后将链接发布到亚马逊土耳其机器人(MTurk), for a general how-to on MTurk: see here and here)
  • 关于素质和心理实验编程的教程: see here
  • For tips on coding in HTML: see this document
  • In the Human Neuroscience Laboratory, we have direct experience coding up experiments employing both word and picture stimuli testing old-new recognition memory, recall, source memory (e.g.(如空间位置)，以及各种创造性任务(如.g., Alternate Uses Task and Remote Associates Task). For templates contact Preston. 我们在质量方面也有平衡和循环的经验.g., see this video for a short how-to on looping)，这是适当和有效的实验设计所必需的!
  • If using Qualtrics, there are FOUR things that are critical:
    • 1a)确保监控非任务表现(用于相关出版物); Permut et al., 2019)
      • a) See here for the code to add in your Header (请参阅调查问卷的“观感”和“一般”部分)
      • b) See here for the code to add in your Footer (请参阅调查问卷的“观感”和“一般”部分)
      • c)确保按照Permut等人的详细说明添加嵌入式数据字段. (2019)
    • 1b) The code listed in 1a, in certain cases may not work, due to compatibility issues in Qualtrics
    • 2) At the end of online experiments, each MTurk worker is provided a unique code which they are asked to input into the MTurk survey. 他们必须在提交人类智能任务(HIT)之前这样做才能收到报酬. 如果参与者不输入此代码，则不会付款.
      • 例如，构建一个最后的问题，看起来像 this
    • 3) To ensure you create a Qualification in MTurk to ensure MTurkers cannot retake your experiment
      • Why do you need this? See here and here
      • To get your unique MTurk code: http://uniqueturker.myleott.com/
      • There are additional qualifications/restrictions you can setup in your MTurk requester account when posting your study, see here for the Human Neuroscience Laboratory settings (as you can see in the picture, 我设置了一个限制，这样的电子游戏正规平台之前没有参与过, here called, 'Remembering Lists')
        • E.g., We like recruiting participants with a HIT approval rate of >95% (meaning that those who participate are those that are likely to finish it as that is their prior history), and also having participants with at least 50 HITs approved meaning they have done a considerable amount of MTurk studies (note that these setting follow other published studies).
        • Note also that some restrictions, like age, cost additional money (so called, 'Premium Restrictions')
    • 4) How to save/extract your data (e.g., as a .csv file)
      • 使用问题旁边的齿轮按钮打开变量命名. Check box called 'variable naming': this will save the actual response selected as opposed to the position of the click or choice.
      • In the export section make sure to export the .CSV与'选择文本'选定(参见其他选项，如删除空格)
      • 关键是，不要假设你所有的数据/回复都会被保存. Ensure the output captures it!

Alternatives to Qualtrics

  • You can code in Java/HTML straight in MTurk, 然而，它不是基于gui的，学习和执行起来可能很耗时
  • 如果对这种方法感兴趣，看这些很有帮助 videos and tutorial
  • In addition, you can also use REDCap or PsyToolkit
    • REDCap is more survey-based. Research Services @BC are available to meet with members of the Boston College community to discuss individual REDCap projects. Individual consultations or customized class consultations are available by emailing researchservices@americangreens.net or dalgin@americangreens.net.
    • PsyToolkit has an amazing library of classic cognitive psychology experiment templates some of which you think would not be suitable for online data collection (e.g., Stroop Task, Mental Rotation, Posner Cueing Task, Wisconsin Cart Sorting Task, Go/NoGo task, Inhibition of Return, Multitasking, etc.)

(不全面)在线数据收集清单

These are suggestions and not rules (see here for additional information)

1)在客户端代码中隐藏敏感内容

  • Check your client-side code for sensitive content. 在浏览器窗口中打开实验程序. If you are using Chrome, click View > Developer > View Source. In Firefox, click Tools > Web Developer > Page Source. (谷歌“如何在[浏览器名称]中查看页面源代码”以查找其他浏览器的说明.) A new browser window will open with the source code for the particular page of your experiment you are viewing. Scroll through this code, 确保里面没有任何你不想让参与者看到的东西(e).g. hidden payoff values, logic that reveals your experimental manipulation, names of different conditions, etc.).
  • Obfuscate any JavaScript code that you are sending client-side so participants can’t view hidden values or the logic of your procedure by viewing the source code. Open-source tools like javascript-obfuscator 由Timofey Kachalov创建，可以用来混淆JavaScript代码. A Web UI for javascript-obfuscator@0.18.6 can be found at obfuscator.io. (更好的选择是将所有敏感内容保存在服务器端，但这并不总是可行的.)

2) Testing your procedure

  • Create a Mechanical Turk worker account so you can test your procedure as a participant before launching publicly
  • 设置自定义资格并将其仅分配给您自己的工人帐户
  • 使用分配给工人帐户的自定义资格启动您的实验, preventing other workers from viewing the HIT
  • Log into your worker account, and complete your HIT at least 8 times, 每次使用以下8种配置之一(浏览器+操作系统):
    • Chrome + Mac
    • Chrome + Windows
    • Firefox + Mac
    • Firefox + Windows
    • Edge + Mac
    • Edge + Windows
    • Safari + Mac
    • Safari + Windows
  • Either block mobile devices, 或在移动设备上测试您的HIT，以确保您的程序正常运行. (You can simulate a mobile request in Chrome by selecting View > Developer Tools, clicking on the Toggle Device Toolbar icon, selecting “mid-tier mobile” and selecting a mobile device model in the device toolbar that pops up. More information can be found on this page of the Google Developers Help Center.)
  • Check the URLs for each page in your procedure to ensure they do not contain any information that might reveal your experimental manipulation
  • 检查您的记录数据，以确保所有活动都被适当地跟踪
  • Have a few friends complete your survey and time the duration of their responses to each question type, so you have some sense of how long it *should* take the average participant to complete the procedure

3) Protecting yourself against low quality data

  • Always include a neutral answer (e.g. “Choose here”) as the preselected or default response in form elements such as drop down lists
  • Test multiple response styles (e.g. 径向、滑块、下拉)来确定响应类型是否会影响结果
  • 不将资格要求作为电子游戏正规平台注册过程的一部分, 相反，使用资格限制(如国家/地区), number of studies completed, etc.)或预先调查以限制资格(Siegel et al .)., 2015)
  • 通过Mechanical Turk API分配自定义资格来防止重修, or using Qualtrics’s built-in system
  • 收集IP地址和位置数据，因为这些数据可用于检测机器人活动
  • Use a transcription task at the start of your experiment that requires participants to transcribe a photograph of handwritten words
  • 使用不明显的模式(如.e. answers to multiple choice comprehension questions should not all be assigned to the top radial in each choice list)
  • Include a mandatory free-form response at the end of your survey (you can instruct participants simply to type “na” if they have no response/comments)
  • 使用自定义JavaScript计算主题离开页面的次数, 跟踪主题离开页面的时间, 或者甚至在参与者离开页面时显示弹出消息, warning her that the HIT will not be approved if she leaves the procedure before completing it. (You can find some sample code that implements this sort of tracking in Qualtrics, see above)

4) Reducing the negative impact of attrition

  • Ask personal demographic questions, 并在你的手术开始时宣布有机会获得额外的经济奖励, 因为这两种措施都被证明可以减少辍学率(Reips), 2002a)
  • Introduce a high hurdle task by concentrating motivationally adverse factors toward the start of the procedure
  • Ask participants to estimate the likelihood they will complete the whole experiment before they begin the procedure
  • Use a prewarning with an appeal to conscience (e.g. “Our research depends on good quality data, 当参与者未能完成整个过程时，这种质量就会受到影响.”) (Zhou & Fishbach, 2016)
  • Use practice trials, pilot new experimental protocols, or ask participants to complete a boring task prior to starting the focal experimental procedure

5) Analyzing your data

  • Wait until all data have posted to your server, or deactivate your Qualtrics survey, before beginning your analysis – some platforms (e.g. Qualtrics) do not post unfinished responses to the server for up to one week after a participant initiates her response, 在这些数据公布之前，你将无法衡量流失率
  • 检查你收集到的每一个自由形式的回复，寻找可疑的模式.g. 我在数据中检测到的一个脚本从周围的页面复制文本, mixes up the words, 然后将结果粘贴到自由格式响应框中)
  • 检查重复的IP地址和经纬度坐标, 因为大量这样的重复信息可能是机器人活动的标志(Bai, 2018)
  • 将浏览器和操作系统作为分析的因素, 因为系统配置可能会影响参与者与您的过程的交互, and have been found to be correlated with important individual differences that may affect your results (Buchanan & Reips, 2001)
Back To Top