Contributing to education research through survey and statistical support

Background

The Science Education Resource Center at Carleton College (SERC) connects geoscience teachers with professional development opportunities and researches best practices for teaching in the field. As part of their research initiative, SERC hired PDA to survey undergraduate geoscience instructors in 2012 and 2016 to understand current teaching practices and how engaged instructors are in professional development, education research, and scientific research.  In addition to administering the survey, PDA cleaned and processed the data, conducted analysis, and facilitated the manuscripts’ submission process.

Approach

To begin survey administration, we de-duplicated contact lists of over 10,000 geoscience instructors provided by SERC and loaded them into LimeSurvey, an open source online survey management application that PDA hosts locally. We then programmed survey questions and response options into the application with appropriate skip logic and tested the survey in several environments (device type and browser). We developed and programmed email invitations and reminders to be sent automatically through the application. While the survey was in the field, we monitored responses and response rates.

After the survey closed, we processed and cleaned the data; addressing missing data and outliers. We also conducted a data reduction via factor analysis on a subset of survey questions.

Impact

Once the data was ready for analysis, we provided statistical consulting to the SERC research team members on a project by project basis to address specific statistical needs. Some members needed assistance with study design and analysis, others conducted the analysis themselves but came to us for help with interpretation, and others used our services to provide a statistical review of their manuscript prior to submission.

We maintain the survey datasets and codebooks from four iterations of the survey (2004, 2009, 2012, and 2016) that are used by the SERC research team. We have consulted on study design and conducted ten statistical reviews prior to submission for publication.

In 2017, we conducted statistical analysis, drafted the statistical methods section, and provided reviews of a seminal article in the field of professional development for STEM educators, which was published in Science Advances magazine (Manduca 2017), using data from the first three survey administrations. Since the 2016 survey has closed, PDA has provided statistical support to ten studies with topics including active learning, motivation to change teaching methods, student skill development, and systems thinking. Three studies were presented at the Geological Society of America’s annual meeting in November 2018, the remaining studies are in process of being submitted for publication to peer-reviewed journals in late 2018 and early 2019.

Link to published manuscript abstract:

Manduca, C.A., Iverson, E., Luxenberg, M., Macdonald, R.H., McConnell, D., Mogk, D., and Tewksbury, B., 2017. Improving undergraduate STEM education: The efficacy of discipline-based professional development, Science Advances, Vol. 3, no. 2.