Let’s say you want to know if a treatment or category affects an outcome. Could be anything. Does a new pill reduce bloodpressure? Are left-handed people more athletic?

There is an answer, but it is hidden inside the bodies of the subjects. How to get that answer out of them?

You do an experiment!

You collect a number of experimental subjects, categorize them or subject them to one of the treatments you are trying to assess, and then measure the outcome or characteristic you are interested in.

(I’m assuming here you know your methodology, and are doing all the right blinding and randomization! But that’s another topic…)

You might get a result like this:

Wow! You are really on to something here. Those in group B have clearly got a lower result than those in group A.

If group B is those that had your new pill, and you are measuring blood pressure, think of the $$$ that await you!

But then the phone rings

It’s a colleague. They just did an identical experiment, and here’s what they got:

What’s going on here? You agree to run a hypothesis test on the results, to see what that says. You are already looking at your results in DataClassroom, so you hit the GDT button, which explains that a T-test is the right test to use with this data, and get your answers:

You (A higher than B) : P = 0.17, “some indication that the groups might be different”

Colleague (A similar to B) : P = 0.43, “little to no indication that the groups might be different”

Hmm. Not very convincing, all that “some” and “might” there! You recall from your interactive tutorial of the T-test that the number of samples seemed to be an important factor, and indeed you are right. Your P-value is telling you that there is quite a high chance (about 15%) that this result would have been obtained purely by chance, so your results are not very significant. And indeed, your colleague has got a completely different result, where there is no clear overall difference.

How to teach this?

Now you’re reading the DataClassroom blog, so I assume what you are really looking for is a way to impart the above insights to your students. Preferably in an education-friendly way that doesn’t involve putting pills in hundreds of experimental subjects and waiting a month.

Simulation in the classroom

You’ve probably guessed that we didn’t actually perform the above experiments to get the data for this blog post.

We used the DataClassroom Simulator.

The Simulator lets you perform simulations of an experiment, so you can investigate how the results might look.

What did we do?

First, we made a Simulation Model. We made a model of a test subject, with these properties:

A categorical variable ‘Treatment’ with two values which could e.g. represent “pill” or “no pill”. You can name these as you want, we chose A and B. These values will be assigned randomly.
A numerical variable ‘Outcome’. This could represent “change in blood pressure” or “athletic ability X”. It has a mean value, and will vary randomly in the population (you also set the Standard Deviation).
A connection to define the relationship between these variables. This specifies the effect that the categorical variable Treatment (A or B) has on the mean value of Outcome.

Note: step 3 (editing a connection) requires our College Student license (DataClassroom U).

Then we clicked “Simulate” to make a Simulation based on the model. There, we configured:

Which variable is the main output, and which variable(s) should also be observed.
How many samples to generate per “run” (i.e. per simulated experiment). We set this to 20.
That the output graph should plot a histogram of Outcome, grouped by values of Treatment.

Then we were ready to click the “Play” button:

This ran the simulator and generated 20 samples at random, applying the relationship between the variables that we had defined above.

You can open and look at the actual Simulation we made by clicking here. It’s ready to run, you just have to make your own copy.

Within the Simulation, you can also click to edit (or view) the internals of the Model it uses. Indeed, you can make your own copy of the model, which you can give a suitable name, and you can also rename and change the variables, to simulate whatever experiment your students might find most motivating. From your copied model you can then select Simulate from the left menu and make your own simulation.

Now you can play too!

The stage is set for really playing with data, once you have a simulation up and running. You can:

Quickly see a histogram of the result, and see the descriptive statistics like mean and standard error.
Run with more, or fewer samples.
Perform multiple runs and see how they vary.
Click to generate a DataClassroom dataset from any “run”, and perform a hypothesis test.

A simulation where three experiments have been run.

And the answer was….?

Oops. I forgot to mention what the relationship actually was. If you’ve been playing (and making enough samples), you’ll quickly be able to come to a conclusion about the hidden relationship:

There is no relationship. The categorical variable does NOT affect the numerical one.

But a negative result is just as valid as a positive one, of course!

I’m sure you can think of all sorts of ways this can be used to impart some really good and memorable intuitions regarding the collection and evaluation of evidence in science.

For example:

A P-value of 0.05 means that there is a 5% (1 in 20) chance that this result was due to random chance.

Well, now your students can run 20 experiments. What do you think they will see?

Give it a go!

The above is just one simple way to use the simulator. Do go ahead and have a play. There is detailed information in the User Guide, if you are one of those people who read the manual.

Note: editing model connections requires our College Student license (DataClassroom U)

Have fun!

If this sounds like what you’ve been waiting for, register for a free trial of DataClassroom or check out the website, have a browse through the searchable User Guide, watch one of our videos or get in touch and see how we can help!

Collecting enough evidence