Not for the first time, the psychometrician mentioned p-value. P-value? 

I was supposed to know what it meant. But I didn’t. Maybe I had known and forgotten. Maybe not. Either way, I was lost. 

Was I going to speak up and admit this in front of colleagues? No. 

Instead, I waited and googled it. That was not a good idea.

Under People also ask, I clicked What is the p-value in simple terms?

“So what is the simple layman’s definition of p-value? The p-value is the probability that the null hypothesis is true. That’s it. … p-values tell us whether an observation is as a result of a change that was made or is a result of random occurrences. In order to accept a test result we want the p-value to be low.

Simple layman’s definition? Null hypothesis? 

After 30 plus minutes, I gave up. 

Sidebar: Later, the colleague and psychometrician who reviewed the first draft of this post kindly informed me that the simple explanation offered by Google doesn’t even pertain to psychometrics.

(But you’re in luck. A wickedly clear definition of p-value is just two paragraphs away.)

This explains why I nearly stood up and cheered–Sis, Boom, Bah!–when, at a recent company meeting, Dr. Thomas, Kryterion’s Chief Psychometric Officer, provided an explanation that made sense right away. I knew you would want to hear it, too. 

P-value is a statistical measure that tells us how hard it is to correctly answer a test question. 

The standard range is 0 (zero) to 1 (one). 

If nobody answered the question correctly, the p-value = 0. If everyone gets it right, the p-value = 1. For everything else, the value is somewhere in between.

Thank you, Dr. Thomas!

Let’s make this real. 

Example, 100 test candidates answer Question A. Thirty answer it correctly. Seventy get it wrong.

What is the p-value of Question A?

The p-value “formula” is Total Correct Answers divided by the Total Answers. In this example: 

30/100 = 0.30. That would generally be considered a challenging test question. 

And if all 100 answered it correctly, what would the p-value be, then? 

100/100 = 1.00. In other words, Question A was too easy.

In practice, Dr. Thomas explained, a p-value of 0.85 and up indicates an easy item. A p-value of 0.40 and below tells us the item is relatively difficult.

And that, dear reader, is why great psychometricians are worth their weight in test items, item banks and test forms. Not only do they know their stuff, but they also know how to help the rest us understand it! 

If you found this moment of psychometric clarity refreshing, we invite you to stay tuned for more blog posts from our occasional series: Psychometric Insights.

Have a question about the psychometric status of your test development project or credentialing initiative? Reach out to Kryterion’s Psychometrics Team here. 

Our experts specialize in the delivery of clear and helpful insights. You might enjoy it.