Law #1: Identify the Primary Problem the Market Will Pay You to Solve

Discover the four laws of certification success from a professional psychometrician with 20+ years experience working with dozens of credentialing programs spanning multiple industries. Leslie Thomas, Ph.D., is Kryterion’s Chief Psychometric Officer.

In this special presentation originally delivered at the ATP 2019 conference, Leslie distills credentialing success into four immutable laws: all on-point, all informed by experience and all in plain-English. They’re perfect for a non-technical audience looking for the secrets to a winning credentialing program.

Today, you’ll hear Law #1: Identify the Primary Problem the Market Will Pay You to Solve. Responding to an unmet market need is the singlemost important step of the development process. It’s positively foundational. Get this right, and you’ll be in perfect position to understand and implement the three other laws, which will be released in succession, one per week over the next three weeks on this Kryterion blog.

Leslie’s presentation is ideal for an audience contemplating a new credentialing program. It offers benefits, reminders and mental audit points to stakeholders of existing programs as well.  

Transcript of Video

The Four Immutable Laws of Certification Success: A Psychometrician Speaks.
Law #1: Identify the Primary Problem the Market Will Pay You to Solve

If you’d rather read the transcript in PDF format, you can get that right here!

Leslie Thomas, Ph.D.
Chief Psychometric Officer
Kryterion, Inc.

Introduction

I want to welcome everyone here. Thank you very much for taking time out of your busy schedules to join me and the Kryterion Team for the next 60 minutes.

My goal today is to provide you with tips, tactics and strategies for credentialing success that you can take back home and use right away.

One of my own hopes, whenever I attend a conference, is to get information that I can take back and use. If I can do that, the event is worth the price of admission.

My name is Leslie Thomas. I head up Test Development and Psychometric Services at Kryterion. I’ve been with the company for about 15 years.

I’m going to be providing a lot of information to you today. I actually have a sticky note up here saying “Slow Down” because I have a tendency to speak much too fast. If anything comes up, let me know.

We’ll have questions and answers at the end of today’s session. Also, if there’s something you want to ask me, but you’d rather not ask it in front of the group, please feel free to stop by the Kryterion booth #514 in the exhibition hall.

If you see me in the hall, feel free to stop me. I am a little near-sighted so you may have to make it a little obvious. If I don’t see you, it’s not that I’m ignoring you, it’s that my eyes have been bothering me, and I’m not wearing my contacts.

Question: How many of you have worked with or are currently working with a psychometrician?

OK, it looks like most of you. So usually the biggest question I get—and I still get this from my Mom—is what is a psychometrician and what is psychometrics? Through the years, I’ve been introduced at conferences as the Chief Psychometronics Officer. All kinds of names.

Basically, psychometrics is the science of psychological testing. We test everything from personality tests and IQ tests to job-knowledge tests. Any kind of testing that you can imagine.

We work with organizations like yours to help design, develop, analyze, score and maintain tests.

A common misconception is that in order to help you develop an exam, we need to be domain experts. And that’s not the case. We’re experts in working with the client’s subject matter experts to figure out what needs to be measured and the best way to measure it.

Psychometricians facilitate the process and do a lot of statistical analyses on the back end to figure out if what’s been developed is working as intended. 

Leslie’s Background

So, just a little bit about myself as context for today’s talk. I’ve been in the industry for a little over 20 years. I started off at a company called ACT, formally known as American College Testing. I was right out of graduate school at the time. I worked with a lot of traditional programs. I had the luxury at the time of having data sets of 20,000 people. It was wonderful.

I’ve worked with traditional health care organizations, social workers, dieticians and professional engineers. I’ve worked with the Automotive Service Excellence group. I worked with licensure programs as well.

I then went on to another company called Brainbench, which had an adaptive testing driver. I had done some adaptive testing when I was at ACT. Brainbench was more in the IT/software certification space.

They were later sold to Provisor then SHL, then CEB. But Brainbench was a different type of environment because they were for-profit. At the time, they were working with employers to identify qualified professional job candidates. They also worked in the IT certification space. If you’ve worked in that area, you know that the exam content changes quickly, and that was one of the things that we had to figure out how to accommodate.

After that, I had my own consulting business for a couple of years and worked with Lehman Brothers helping them develop a selection system for their IT people and their traders. I also worked with the Army on a career-interest inventory helping them funnel down their 200+ MOS categories.

And then I landed at Kryterion where, as mentioned, I head up Test Development and Psychometric Services.

I tell you all of this to give you some context about my interesting career path. Most of us have had interesting paths. Regardless of the organization I worked with or the content area or domain area of the exam, I started to see patterns from my birds-eye view of working with different programs.

And those are some of the things that I’ll talk to you about today.

The 4 Immutable Laws of Certification Success will help you:

• Design a successful and sustainable credentialing program and/or improve an existing one.Much of what we discuss today applies to an existing program as well as a new program. Credentialing is not a one-and-done effort. It requires constant feeding and care.
• Better communicate the value of your credentialing program
• Recruit and get the most out of your Subject Matter Experts. Your best and brightest SMEs are always the busiest. This presentation will talk about how best to recruit them.
• Create a higher qualify credentialing program in less time

Why do we build credentialing programs?

The why is very important. It’s the foundation of your program. You have to be clear about your goal/outcome. If you get this wrong, everything else will be wrong, too.

At ATP, you may hear different reasons for developing a credentialing program like…

• Protecting the public safety and welfare
• Elevating the status of a profession
• Helping employers find qualified candidates
• Mitigating risk and driving compliance
• Increasing brand loyalty.

There are many reasons for developing a credential. A well-designed program can meet many, if not most, of the particular market needs above, but the programs that are really successful realize that in order to be successful, you need to be very clear about your “why.”

You have to be very clear as to the primary market need that you’re trying to solve. And more importantly, the need must be one that the market will pay for.

So, even though we want to elevate the status of a particular profession, if I ask test takers if they’re willing to pay money and spend time to take this credential in order to elevate the profession, they’re not willing to do that.

Likewise, if you work for companies that send their people to courses and pay for the credential, they (the companies) are going to want to have a clear business need or return on investment in order to do so.

Treat a credentialing program like a product

One of the things I’ve found, especially when working with programs of a nonprofit nature, is that you need to realize that, at the end of the day, your credentialing program is a product, and you need to think about it like a product.  

Case Study #1: Call Center Certification for Upper Level Managers

The sponsor wanted to elevate the status of the call-center industry as a whole, drive best practices across the industry and help upper-level call-center professionals differentiate themselves in the market. All lofty goals.

The problem was that the sponsor targeted the wrong job role. Job demand was strong for call-center professionals, who were more than aware of the shortage because of all of the headhunters calling them and offering them top dollar. Upper-level call center professionals, therefore, saw little need for the credential. Employers were so desperate to find qualified candidates that they frankly didn’t care that prospects were certified.

The sponsor was trying to solve a problem that didn’t need a solution. Their program ended up withering on the vine.

The sponsors had used the “if we build it, they will come” rationale for developing the credential. They never did the market research that would’ve revealed that they were targeting the wrong job role.

Case Study #2: Automotive Industry Certification for Automobile Technicians

The problem for both consumers and employers was how to find qualified mechanics.

I worked on this credentialing program while I was with American College Testing. The Automotive Service Excellence Program was, at the time, the largest voluntary credentialing program out there.

The certification allowed certificants to distinguish themselves in the market. It also helped employers make better hiring decisions. It also allowed them to promote the quality of their service to the marketplace. Lastly, the certification instilled confidence in consumers who understood that more highly trained mechanics could actually help save time and money.

The certification was a success because it fixed a problem that the market would pay for. The certification addressed a real pain point in the marketplace.

Identify the primary problem the market will pay you to resolve

• Get market feedback early and often. Make sure that your credential addresses a problem worth solving.
• Understanding the market pain points will streamline the test-development process and identify ROI metrics
• Communicate how your program addresses specific problems/pain points to increase its credibility and market value.
• “Your opinion, although interesting, is irrelevant.” It’s only your customer’s opinion that matters. Yours doesn’t.

Talk to your customers. Sending out a survey is inexpensive. Include a survey at the end of your exam. Talk to your test taker.

It’s easy from the comfort and security of your office to think you know what your market needs. But once you start to get feedback and input from your market, you’ll discover that market needs are not what you thought they were.

Given the time and expense of developing a credentialing program, you need to be making the best decisions possible throughout. Many of them need to be informed by your market.

You need to understand your market’s pain for two other reasons. First of all, it will help bring focus when you do your test development. Second, it will help you identify potential ROI metrics.

How do we measure ROI for a credentialing program?

The first thing is to focus on is the market pain. For example, a common pain point for companies is how to find a qualified employee to fill a position. Go out to companies and ask them, “What is it about this job role that candidates don’t have the required knowledge and skills to perform the position?”

Get specific feedback, which knowledge and skill gaps are preventing the employer from finding a qualified candidate.

What outcome measures are available that would have the biggest impact in determining the knowledge and skill gaps, for example, in areas of public safety and welfare.

I have worked with successful programs for hospitals, banks and insurance companies to identify outcome measures so that what they’re measuring on the exams is important to consumers and the companies.

Another market pain for IT companies is that in order to drive brand loyalty and product adoption, they must offer customer support, which can get expensive.

To lower these costs, an analysis of support tickets will identify common knowledge and skill gaps of their customers. For IT companies, the customer support department is often the best source of information for real-world scenarios about who’s calling in and the problems (gaps) they’re having. Knowing the gaps will inform training, certification and other things.

Once you start identifying those data measures, which become metrics, effectively, you can link them back to some sort of ROI. This can be messy. It’s not a precise process. Start with the data. See what it tells you.

Feel the pain

Dive deeply into market needs, which I call pain points. People will study for and pay for a credential because they’re being driven by a pain point that needs resolution. 

A certification is a product and needs to be marketed as a product that is eliminating pain, solving a market need or bestowing benefits. You then need to communicate those messages in the language of the marketplace (consumers).

I have worked with a number of companies over the years that had a great, quality product (certification) and were measuring all of the right things, but they weren’t communicating with their market in a way that was making the product visible and known.

It’s helpful to know how your market thinks about things and the language that they use when they talk about pain points. Knowing this will improve your marketing.

Identify the primary problem the market will pay you to resolve

• Do your research: competitive and “self” analysis.
• Ideal circumstances:
• Customer driven
• Captive audience (e.g., used for state licensure)
• Military/government adoption
• Partner with a well-known entity.

One of the first things that I ask new clients is whether they have any competition. If the answer is no, that could be a red flag, so more market research may be needed. I recommend a self-analysis. See what you’re seeing on social media on your program or those of your competitors.

Ideal circumstances (when market conditions make certification a logical choice for solving pain points)

Customer driven. Car manufacturers partnered with ASE to certify car mechanics. This was a mutually beneficial relationship. It works best when there is a pervasive need. If GM withdrew their financial support, for example, you want to know that there’s still enough demand among technicians or other car companies to keep the certification going.

Captive audience. If you’re a certification sponsor for a profession that later becomes regulated by the state. The state may decide to adopt your certification as the new licensure test, in which case everyone who wants to practice the profession has to take your exam.

Military/government adoption. Helping veterans transition from military careers to civilian life can boost your certification if it assists in that transition.

Partner with a well-known entity. You can approach big companies and offer to partner with them. You’ll be the essential third party that will develop a credential for the benefit of the industry and the company. Promoting the certification will be easier when it’s backed by a recognized company with credibility in the marketplace.

Contact Us! The Kryterion Psychometric Team would love to learn more about your first or next certification exam. You can start the conversation here.