Close this search box.
Close this search box.

From 500 Test Questions to Zero: A Psychometrician’s Cautionary Tale

While creating a certification program can be a lot of work, it doesn’t have to be difficult if you know what you’re doing. Unfortunately, many new programs stumble into avoidable pitfalls, often at a high cost. To help you avoid these potential dangers, I am distilling my 20+ years of experience into a series of articles on “Mistakes New Certification Programs Make and How to Avoid Them” to explain what the most successful programs do differently than the rest. 

So, let’s start with a story from my own experience when I was just starting my career. 

Many (many 😊) years ago, a tech company, eager to launch its inaugural certification program, organized a contest where employees contributed over 500 test questions related to the company’s technology.

Meeting with Stakeholders

I was hired as psychometrician to facilitate a review of these questions along with a team of subject matter experts (SMEs) who had flown in from all over the world.  Our very first meeting was with the CEO who stressed the importance of the program’s broader market appeal, insisting on a technology-agnostic approach. This one decision meant we had to discard most of the employee-contributed questions. 

Further discussions with key company stakeholders revealed a lack of consensus as to why a certification program was being developed (i.e., for what purpose) and for whom (i.e., for which job roles). As you probably guessed by now, we had to start the whole exam development process all over again… 

In fact, many certification programs fail because they rush into exam development without laying a solid foundation for success first. One of the most common mistakes is:  


Failing to Align Stakeholders Before Exam Development Begins. Mark Twain once said, “if you don’t know where you’re going, chances are you won’t get there”.  Little did he know that this also applies to designing a certification program. 

So, what do the most successful certification programs do differently?

Because successful certification programs know it’s almost impossible to effectively develop to a “moving target”, they:  

1. Make Key Decisions Early

Critical questions are answered before exam development begins. By taking the time upfront to research and design the program based on organizational and market needs and constraints, they not only significantly increase their chances of success, but also save themselves a lot of time, money, and frustration on the backend. 

2. Ensure Stakeholder Agreement

Key decisions are documented and shared with important stakeholders with the understanding that these choices cannot be changed later unless there is substantial evidence – not opinion or preference – to do so.  In fact, I’ll sometimes joke with stakeholders that this is one of those “speak now or forever hold your peace” moments because changes to these types of decisions often result in costly rework, reduced momentum, and a poorer quality program – and can be very demotivating to the team. 

3. Address Issues Promptly

“Stuff” happens. The best-laid plans almost always go awry. However, it is how these challenges are addressed that differentiates the most successful programs from the rest. Rather than ignoring potential problems, these organizations tackle them head on before they escalate into a bigger challenge.   

Once the overall design, feasibility, and direction of the program are defined and agreed upon, exam development can be expedited. In other words, once you know where you want to go, it’s a lot easier, faster, and cheaper to get there.   

Interested in knowing more about the types of questions that stakeholders should consider when designing a certification program? If so, stay tuned for a posting next week where I’ll provide a list of critical questions that all successful programs consider when creating a new certification program. 

If you enjoyed this article, please feel free to like and share it. Also, I’d love to hear about your experiences and key insights from working on a new certification program. Share your stories and let’s continue learning together!   

We can help you get where you want to go 

Kryterion’s Test Development and Psychometric Services is known for our exceptional service, extensive expertise, and for our practical and customer-focused approach. We work with each client to craft the best possible solutions given their goals and resource constraints. During this process, we educate our clients on various options and help them select the optimal path based on their specific situation. Our clients have recognized us for our responsiveness, flexibility, customer focus, and high-quality work product. In fact, we have repeatedly earned a perfect client satisfaction rating – a Net Promoter Score (NPS) of 100. Please feel free to contact us at anytime; we are happy to help.

Subscribe Today

Stay informed! Sign up to receive periodic business and
assessment industry news and updates. Unsubscribe at any time.

Take Control of Your Certification Program!

Want more information on how to start or expand a credentialing program? Fill out the form below and we’ll get back to you right away!

Read Reviews