10 Data Collection
When collecting original data as part of your study (i.e., you are administering your own survey, assessment, or observation), data management best practices should be interwoven throughout your data collection process. The number one way to ensure the integrity of your data is to spend time planning your data collection efforts (Northern Illinois University n.d.). Not only does planning minimize errors, it also keeps your data secure, valid, and relieves future data cleaning headaches.
If you have ever created a data collection instrument and expected it to export data that looks like the image on the left (Figure 10.2), but instead you export data that looks like the image on the right, then you know what I mean. Collecting quality data doesn’t just happen because you create an instrument, it takes careful consideration, structure, and planning on the part of the entire team.
10.1 Planning
Planning for data collection begins in the Data Management Plan phase when you first choose measures you plan to collect as well as the instruments you plan to use to collect those measures. Then, during the planning and documentation phase, you can begin to really flesh out the exact items you plan to collect, how those items will be recorded (e.g., type, name, values), and how they will be collected (e.g., online survey, paper form). And last in the create instruments phase, you start to choose tools to build your instruments in and begin constructing those instruments. As shown in Figure 10.1 above, you’ll see that creating data collection instruments is typically a collaborative effort between the project management and data management team members.
However, planning includes not only carefully designing your data collection instruments, but it also includes considering the data collection and handling process itself. While the project management team is typically in charge of overseeing data collection, there are still best practices that can be implemented during this time to collect better data. In this chapter we will not only review best practices for building your data collection instruments, we will also discuss our two other lines of defense which include data management in the field and implementing ongoing data checks to catch errors early on (DIME Analytics n.d.).
Before we dive into planning for building data collection instruments, it’s important to first review the ethical and legal considerations of your data collection effort. When working with human subjects it is likely that the Institutional Review Board (IRB) will need to review and approve all of your data collection instruments as well as any agreement forms that will be collected as part of your study. Our next section will provide an overview of the IRB and its requirements as well as best practices for creating agreement forms for participants and partners.
10.2 Institutional Review Board
The IRB is a formal organization designated to review and monitor human participant research and ensure that the welfare, rights, and privacy of research participants are maintained throughout the project (Oregon State University 2012). More likely than not, if you are conducting education research with human participants you will have some interaction and oversight with the IRB. Before reviewing potential requirements, lets review the history of this administrative body.
10.2.1 Background
In 1974 the IRB was established as part of the National Research Act in response to a long history of unethical research that had been conducted with human participants (Qiao 2018). In 1979, the Belmont Report established a set of ethical principles for doing research with human participants. Those ethical principles included the following (Duru and Sautmann n.d.; Huisman n.d.; Office for Human Research Office for Human Research Protections 2018):
- Respect for persons
- This included both protecting autonomy of participants by acquiring consent as well as providing a plan to protect participant privacy
- In practice this meant acquiring consent in a way that ensures participants can comprehend what is being asked of them, understand that their participation is voluntary, and understand the plan to protect their privacy
- This included both protecting autonomy of participants by acquiring consent as well as providing a plan to protect participant privacy
- Beneficence
- This involved maximizing good and minimizing harm in the study, for both participants and society at large
- In practice this meant taking time to assess risk and benefits of your study for both the intervention itself as well as the data collection efforts (e.g., how burdensome is the survey)
- This involved maximizing good and minimizing harm in the study, for both participants and society at large
- Justice
- This included providing additional care and consideration when working with vulnerable populations (e.g., children, prisoners), making sure your practices are non-exploitative and there is fair distribution of costs and benefits across all participants
- In practice this involved fairness in the selection of participants
- This included providing additional care and consideration when working with vulnerable populations (e.g., children, prisoners), making sure your practices are non-exploitative and there is fair distribution of costs and benefits across all participants
Heavily influenced by the Belmont Report, in 1991 the Federal Policy for the Protection of Human Subjects was published, establishing core procedures for human subject protections. The policy, 45 CFR part 46 (Office for Human Research Protections 2016), included four subparts. Subpart A, known as the “Common Rule” for the 15 federal departments and agencies which codified the policy in separate regulations, provided a set of protections for human subjects research including informed consent, review by an IRB, and compliance monitoring (National Institute of Justice 2007; Office for Human Research 2009).
In 2018 the Common Rule was revised in order to better protect research participants and to reduce administrative burden (Office for Human Research Office for Human Research 2018; U.S. Department of Health and Human Services n.d.). While many revisions were made, some changes that are applicable to education researchers include the following (Fordham University n.d.):
- Revisions and additions to exempt categories, many of which are applicable to research conducted in educational settings
- Reduced burden of continuing review, particularly for exempt and expedited studies
- Clarifications on how informed consent should be organized, written, and provided
10.2.2 Requirements
While each institution’s IRB submission process is different, typically if your study involves working with human subjects you are required to submit an application to the IRB. As part of your application you will be asked to state what review category your study falls under (Lafayette College n.d.; Northwestern University n.d.; University of California Berkeley 2022).
- Exempt
- These studies usually involve minimal risk and fit within categories predefined by your IRB (e.g., Evaluating the use of accepted or revised standardized tests). These studies typically involve a shorter review process and a quicker review than non-exempt studies.
- Expedited
- These studies also involve minimal risk but do not meet criteria for exempt status (e.g., collection of voice, video, or image data from non-vulnerable populations).
- Full Review
- If a study does not fall into one of the two categories above (e.g., collection of information about illegal behavior), it requires full review, discussed by the full board at a convened meeting.
As part of your application, common documents you may be required to submit include the following (Cabrini University n.d.; Duru and Sautmann n.d.).
- Certificates from human subjects training (e.g., CITI training)
- Research protocol (see Chapter 7)
- When writing your protocol, make sure to review your IRB’s rules around data handling and include this information in your plan. IRBs typically have specific rules for things such as how paper and electronic data must be stored and backed up, how long data should be retained, how data can be transferred and shared, and how data should be anonymized (Filip, n.d.).
- Study materials (e.g., recruitment materials)
- Copies of your instruments (e.g., surveys, interview guides)
- Note that these will need to be created before you can submit to your IRB so make sure to consider timing and start building your instruments early enough to give you time to submit to your IRB before data collection
- Copy of informed consent/assent forms
- Same as above, give yourself plenty of time to submit before you start participant recruitment
- If collecting data from sites (such as school districts) or sharing data between sites, supporting documentation from those partners may be required (MOUs, data use/sharing agreements, letters of support, confidentiality agreements)
- If partnering with other institutions, IRB approval letters from partner institutions may also be required
The review process can take several weeks and it is common for the IRB to request revisions to materials. Make sure to review your timeline and give yourself plenty of time to work through this process before you need to begin recruitment and data collection.
10.3 Agreements
There are several types of agreements that may be required for your research study for both ethical and legal reasons. Here we will discuss the most common type of agreements, informed consent and assent, as well as other agreements used when working with external partners including data sharing agreements, memorandum of understanding documents, and confidentiality agreements.
10.3.1 Consents
Informed consent involves obtaining a participant’s voluntary agreement to participate in your research study. As described in the Belmont Report, informed consent should meet the following criteria (Huisman n.d.):
- Describe the study and what is expected of the participant
- Use accessible language to ensure comprehension. Avoid technical jargon and explain terms that may not be easily understood.
- Explain that participation is voluntary
- Review how participant privacy will be maintained
With the revised Common Rule, additional requirements for informed consent were added (Fordham University n.d.).
- The top of the consent must begin with a concise review of key information that allows participants to make informed decisions
- All information must be presented with sufficient detail to make decisions, not just bulleted lists of facts
- The form must disclose any plans to use participant data for other future research
Figure 10.3 shows common elements that are included in a participant consent form (Bellevue College, n.d.; The Turing Way Community 2022).
Depending on the type of research study, a participant signature or a check box denoting consent may be required. If so, it can be helpful to put the above information on a cover/information sheet, and then have a separate page for signed consent. Before signing, participants should be required to acknowledge that they
- Have read and understood the information provided
- Have been given the opportunity to ask questions
- Understand that their participation is voluntary
- Understand that they may withdraw from the study at any time
As mentioned above, not all studies require active consent (University of Virginia n.d.). Some studies may allow passive consent that is obtained by providing the information sheet to all participants which says if they consent, they simply move forward with the study. If they choose to withdraw, they can notify a specified contact. Your institution’s IRB will let you know which type of consent is required for your study.
10.3.1.1 Data Sharing
While creating your consent is very important to consider how you want to gain consent for public data sharing. There are essentially three different ways you can go about obtaining consent for data sharing (Gilmore, Kennedy, and Adolph 2018).
- Include a line about public data sharing in your consent to participate to research, therefore a participant who consents is agreeing to both participate in the research study and have their data shared publicly.
- Have participant’s consent to data sharing at the same time you provide the research study consent, but provide a separate consent form for the purposes of public data sharing.
- Have participant’s consent to data sharing on a separate consent form, at a later time, after research activities are completed. Obtaining consent this way ensures the participant’s are fully aware of the data collected from them and can make an informed decision about the future of that data.
A limitation of using method 1, as discussed by Gilmore, et al. (2018), is that if a participant is uncomfortable with their data being publicly shared, you will then also loose them as a study participant. So method 2 or 3 may be your best option. If you choose to go with method 2 or 3, it is very important that you not only track your participant study consent status in your tracking database (as discussed in Chapter 9), but that you also add a field to track the consent status for data sharing so that you only publicly share data for those that have given you permission to do so.
Meyer (2018) provides some general best practices to consider when adding language about public data sharing to a consent form.
- Don’t promise to destroy your data (unless your funder/IRB explicitly requires it)
- Do incorporate data-retention and sharing plans including letting participants know who will have access to their data
- Don’t promise to not share data
- Do get consent to retain and share data (consider adding the specific repository you plan to share your data in). Considered offering tiered levels of consent for participants who may not want all of their data publicly shared but will allow some.
- Don’t promise that research analyses of the collected data will be limited to certain topics
- Do say that data may be used for future research and share general purposes (e.g., replication, new analyses)
- Do review the ways you plan to de-identify data but be thoughtful when considering risks of re-identification (ex: small sample size for sub-groups)
10.3.1.2 Assents
If your study involves participants under the age of 18, you may also be required to obtain a participant assent form, in addition to a parent/guardian consent form. The guidelines for when assent is needed varies across IRBs, but typically if a child is age 7 or older (Duru and Sautmann n.d.), both assent and parent consent is needed. While including similar information as above, these are usually shorter forms that require much more simplistic language depending on the age of the child.
10.3.1.3 Collecting
Last, many institutions have started collecting electronic consent rather than paper consents, especially with a rise in remote data collection efforts. A few considerations for electronic consent include the following (Lee, Hughes, and Marsh 2020):
- Make sure your IRB approves this method
- Use institution and IRB approved tools to collect consent (e.g., Qualtrics, DocuSign)
- Find out what information is required by your IRB (e.g., signature, typed name, check box, date)
- Consider how those consents will be stored (e.g., download PDFs, download spreadsheet, store in collection tool)
Templates and Resources
10.3.2 Other agreements
As we discussed in Chapter 7, a data use agreement (DUA) is a contractual document that lays out expectations for how data will be shared between two or more parties. While the terms data use agreement and data sharing agreement (DSA) are often used interchangeably, I want to differentiate between the two documents. Data use agreements are typically legally binding agreements that provide terms and conditions for working with restricted use data. DUAs are commonly written for data sharing with school districts. In this case, a DUA may include the terms for sharing, working with, and storing identifiable district level data.
When working with de-identified non-sensitive data, a data sharing agreement is a good option. A DSA is a less formal agreement but is still beneficial if you want to provide terms for how data is used, such as limiting the types of projects that use the data (LDbase n.d.a). We will talk more about these types of agreements in the Chapter 14.
Another type of agreement, commonly signed when working with partners such a school districts, is a memorandum of understandings (MOU), which establishes the framework for collaboration (National Center for Education Statistics n.d.; REL West, n.d.). This document is typically not legally binding, but establishes agreements around things such as responsibilities, communication, and expectations (Duru and Kopper n.d.b). An MOU can be a standalone document or can include a DSA or DUA as part of the document.
Last, confidentiality agreements and non-disclosure agreements (NDAs) are other types of agreement that may be needed. These documents restrict the use of proprietary or confidential information (University of Washington n.d.) and are legally enforceable agreements.
Templates and Resources
Source | Resource |
---|---|
Amy O’Hara | Sample text for data sharing agreements 50 |
Florida State University | Example data use agreement 51 |
REL West | Data use agreement checklist 52 |
University of North Carolina | Data use agreement decision making flow chart 53 |
Wilhelmina van Dijk, Sara Hart | Example data sharing agreement 54 |