DataClassroom is often used for analyzing results of surveys. What are the pitfalls to avoid when designing a survey, and what are best practices?

You are going to want to analyze the results afterwards in order to draw conclusions, so make it easy on yourself.

1. Define the goal(s) of your survey

The goal of your survey will typically be to answer a research question. Write that question down, and ensure that it is a question that has a definite answer.

Don’t use: “I want to learn more about….”

Do use: “Does X have an effect on Y?”

The formal terminology is that you should state your “null” and “alternative” hypotheses. Also, it’s important that you state the question before you have collected your data. This is slightly counter-intuitive, but not doing this can invalidate your results. Google “p-hacking” for more detail.

2. Name your variables

Each question will be a variable in your dataset, and the column header will be the name of your variable. Use a survey tool that allows you to have both a question name and a description, so your column headers aren’t long, explanatory sentences.

Compare these alternatives. Your column headers will also be on the axes of any graphs you make!

With Google Forms you can have a short name, and a long description :

TIP: if you already have data with long column headers imported to DataClassroom, you can easily move these into a description field as described in the User Guide here.

3. Plan what you will do with the results

Once you have collected your data, it will need to be in a Tidy Data format which would typically be a spreadsheet, with a column for each question, and a row for each respondent.

How about making up some dummy data, just as you expect it to look, and import it to DataClassroom? See what your options are for visualizing it, and maybe doing a statistical test.

Ask yourself: Can the data you plan to collect answer your research question?

4. Keep it short

It may be tempting to add a large number of additional questions, but answering a long questionnaire can be tedious, even for the most willing recipient.

A short survey will mean that participants spend more time thinking about the questions and giving you accurate answers.

Also, more formally - you shouldn’t formulate new research questions after you’ve collected your data (you can, but you’ll need to do a new survey). So any questions that are not used for your written goals are not going to be usable for drawing conclusions, only for giving you ideas for a new study.

5. Formulate your questions carefully

This is a huge topic, and would need a whole blog post to do it justice. But just to quickly mention:

Use closed-ended questions - anywhere someone is writing a response rather than choosing between options or stating a number is not going to be easily analyzable. But you realized this when you made your test data, right?
Make sure answer options are balanced between extremes, so you don’t introduce bias
Avoid absolutes like “never” or allow for nuances like “almost never”

Good luck!

If you’ve followed the above, you will have a good chance of doing some research that you can draw some well-founded conclusions from.

Now you just need to find and contact your participants. Avoiding bias in the selection process, of course!

Have fun!

Designing a survey?