Predictive Pre-Hire Talent Assessment? 15 Risks

Screen Shot 2016-10-12 at 9.16.28 AM

Over the past 30-plus years, businesses have spent billions on talent assessments. Many of these are now being used to understand job candidates. Increasingly, businesses are asking how (or if) a predictive talent acquisition strategy can include the use of pre-hire assessments. As costs of failed new hires continue to rise, recruiters and hiring managers are looking for any kind of pre-hire information to increase the probability of making a great hire.

What is a Real Predictive Solution?

For all of the marketing hype, Predictive Analytics boils down to three very simple steps.

● First, a system reads “input” data—perhaps assessment scores or CV information.

● Second, the system does some math to apply a “predictive model” to the input data.

● Finally, the results of the model are shown as “output” data of the model—perhaps the likelihood of the candidate achieving a certain level of Sales Performance or another KPI.  At heart, it takes “inputs” and turns them into “outputs” or predicted business outcomes.  But to build and validate a model, you need a healthy, logical set of both input and output data for that role in your company.

If you are using a talent assessment alone this is just input data. The talent assessment is just one piece of the system. For most companies, their current pre-hire talent assessments are wasted data. Results are delivered in an individual report that cannot be analyzed or aggregated. For most “legacy” talent assessments, it’s difficult or impossible to determine what positive (or negative) business effect the assessments are having. It often comes down to the question of “how much the HR person believes the results”. This is a bad measure of success.

If you’d like to begin a predictive talent acquisition project using talent assessments, it can be daunting to figure out what solutions are smoke and mirrors, and what solutions will actually deliver a predictive solution. To help, I wanted to share important factors to sort through “pretend predictive solutions” and “real, rigorous predictive solutions” that can deliver significant bottom line results. This decision can dramatically affect your business’s bottom line. It’s important.

15 Risks while Selecting a Predictive Talent Assessment Solution

1. The Predictive Company Itself. Are you dealing with an assessment company, who is trying to learn how to be predictive? Or is it a predictive company that also uses assessment data?  How long have they been doing predictive work? Are they invited to speak at predictive conferences or at basic HR conferences?

2. Their Predictive Team. Ideally the company will have Data Scientists on staff as well as IO Psychologists. This is important because Data Scientists tend to utilize more modern and rigorous methods for prediction and validation. IO Psychologists tend to be focused on the instrument, while Data Scientists tend to be concerned with predictive validity and business results.

3. Are They Predicting For Your Company, or For Everyone? There are companies that create “Industry Benchmarks,” that is, general performance predictions for general industry categories—such as Retail Sales or Customer Service. These predictions are significantly less accurate, because they are based on companies different from your own, with different cultures, goals, and regions. Not all “Customer Service” is the same.

4. Do They Care About Your Outcome Data? Generally these solutions predict attrition or performance for a candidate or employee. Has the assessment company asked you for the attrition or KPI data for your employees in your target role? If they don’t know your employee outcomes, how can they predict your outcomes? They can’t. Most job roles have multiple KPIs that describe performance—do they predict each of these separately?

5. Does the Solution Base Predictions on Outcome Data or a Job Fit, Job Match or Job Blueprint Survey? Data Science predicts what you ask it to predict. If you want lower attrition or higher KPIs, the models must be trained and validated with those data alone.  The process looks for fact-based patterns to drive your business. Surprisingly, many solutions don’t use this approach, but fall back to managerial bias.

6. Does the Solution Use Machine Learning to Recalibrate Your Predictive Models? How Often? Business needs, role descriptions, and culture changes over time. Local labor conditions change. For example, Service Representatives may be incentivized to cross-sell related products, or new regulations may require new compliance to be performed.

7. The New Validation Question: Criterion Validation? HR has been taught to ask if the assessment is validated. The first level of validation checks whether the assessment measures are self-consistent. Continue to ask this question. But ultimately you care about whether the assessment feeds predictions that accurately correspond to improved business outcomes. That is, are the predictions actually working?

8. Can You Easily Access / Download Your Company’s Talent Assessment Data? Talent assessment data is a critical dataset for your company. If your Talent Assessment vendor makes it difficult or impossible to access your talent assessment data—this is a good indication they are using pre-predictive technology and that they don’t appreciate that this data is your asset.

9. How Easy is it to Deploy the Solution into the Talent Acquisition Process and Use the Predictions? How much training is required? Do your talent acquisition professionals need to read long text reports, or get out a calculator to use the predictions?  The complexity of a prediction should be kept out of the way of daily operations. If your team still needs to “think” about what the answer is, it is probably not a predictive solution.

10. Is there a Different Assessment for Every Role?  Or One Assessment with Multiple Predictive Models? Multiple assessments make it impossible to predict one candidate’s performance against multiple roles. This may also be a signal that you are working with an older, legacy (less predictive) talent assessment supplier.

11. Is There an Answer Key for their Solution on the Web? For many assessments, there are answer keys and guides on how to fool or pass the test. A data science-driven model would be custom to your role in your company, and be continuously evolving—therefore very difficult for answer keys and spoofing to catch.

12. Does the Company Itself Specifically Tell You Not to Use their Solution for Hiring / Talent Acquisition? Some assessments, notably the Myers Briggs survey, specifically implore users to not use the tool for talent acquisition: “It is not ethical to use the MBTI instrument for hiring or for deciding job assignments.”

13. Ask to see their company policy on employee predictive modeling, discrimination, disparate impact and fairness. It is important that a predictive solution has thought through the specific outcomes of their models and how they fit into creating fair opportunity for all applicants. In particular it is vital for the solution to satisfy government requirements for hiring.

14. Do Your Own (Internal) Data Scientists Approve of this Predictive Solution? We recommend asking one of your own data scientists (from HR, marketing, or another area inside your own company) to accompany you in your evaluation. They know what is a rigorous approach and what is marketing fluff.

 15. How Does the Predictive Solution Regularly Prove to You that the Models Are Working? Ideally the company you select will be able to show you two to four times a year how your predictions are working. Only use a predictive model during talent acquisition if the predictions are accurate. If they’re not—you should stop using the models. You need this feedback.


Greta Roberts is CEO of Talent Analytics.