Tests during recruitment: testing your tests

So you’re recruiting- it happens from time to time (or every day, relentlessly).    Maybe you do just the basics of just interviewing, or maybe you do the full suite of cognitive testing, personality testing, team assessments, references; the whole nine yards, with some graphology thrown in for good measure.

Should you?  Should you not?  Should you use some, if so, which ones?

I’m not going to tell you which to use sorry.  But I will help you with some insight about how to evaluate what you are doing. 

The truth is out there

I don’t want to get too philosophical on you, nor do I want to invite epistemological debate, so I’m going to ask you to accept the premise- there is a truth.  The selection process is about finding the answer to two questions- will they successful in the job, and are they do it better than the others.  As a recruiter/hiring manager/employer, we want to reveal that truth, (do they in fact have the necessary capabilities) so we test. 

What it is to test

When we conduct a recruitment test, we are essentially measuring.  We are measuring a candidate’s suitability to do the job;  we are assessing the extent they have the desired characteristics that we have determined to be necessary for the job.  Furthermore, testing is more than just measuring, is it essentially measuring then discerning whether it is sufficient to a required standard.  For example, when teachers test students, they are measuring their students’ knowledge on the tested subject, and then assessing whether they know enough, and to the extent they know it well- pass/fail, A, B or C.

Some characteristics are hard to test

Some characteristics are easy to measure, some are difficult.  Some are strong predictors of success, some are less so.  Some truths are easy to know, and some are hard.   Some data are simple to measure and discern, (i.e. quantifiable and objective) some are more nebulous, subject and qualitative in nature.  Does the person know the NZ tax code is simpler to test than someone’s management capabilities.

Let’s use an example

If we needed a basketball player, we pre-establish many characteristics we perceive to be either a) necessary, or b) advantageous to success. These could be height, skilled at shooting baskets, and able to function well in a team.   

Some of these characteristics we measure accurately- we can get a tape measure and measure the distance from top of head to the soles of their feet, and know very certainly how tall they are.  Some we can reasonably certain about-we can give the player 100, 500, or 1000 shots at a basket and measure the number they get in.  But others- capacity to play well in a team- now that’s hard.  We can question, we can simulate, and we stress test.  But we can only get a rough approximation.

Tests are imperfect

So let’s from here we have these simple premises:

-        Recruitment is a calculated guess

-        No test is perfect

-        Some tests are better than others

No test is perfect, no assessment which makes predicts of the future will be right 100% of the time. 

Testing your tests

The biggest mistake we can make in recruitment is bad data.  Thinking we know something to be relatively true, but is in fact not.  We can see this is almost all bad hiring decisions- at some stage along the line, what we thought was the truth, was in fact not the case.    So how do we get bad data?  How do we end up with misunderstandings of what the truth about this candidate is? 

As a quick aside- maybe they lied to you, but let’s not just stop there.  All candidates mislead us a little, we are supposed to be watching for that. 

In many many fields of science, they assess their tests like this:

False negatives and false positives

  It’s important to recognise that we may be misled, mostly in two key ways:

-  False negatives.  This is where a test tells us that the person is not suitable due to this, but in fact this is incorrect. Examples could be:

o   A weak handshake (is the job arm wrestling?)

o   Being late (once isn’t proof of a pattern)

o   Not giving a good answer to one question (how important is having insight into how you’ve handled conflict in the past for a sales role?)

Note the above list are not examples of bad tools, but how data derived from tools may turn out to false data.  Even the most punctual person can get lost and be late.

-  False positives.  This is the opposite, where the test suggests suitability, but it may not be the actual truth.  Examples could be:

o   A great impression at interview (how important is smiling, engaging, looking in the eye for factory role?)

o   A positive reference from a colleague

o   A socially desirable personality assessment (is it important that a quality manager scores highly on agreeableness?)

o   Any form of graphology.

As above, this list is also not examples of bad tools (well the graphology is) but how ‘good signs’ might not turn out to be reliable. 

What should you take from this

Firstly, it’s a good tool for critiquing your own work.  We all recruit, and apply tests to candidates all the time, and we get it wrong regularly.  All recruitment is a calculated gamble, and not every hire pays off.  The question ‘what did we get wrong when hiring Johnny’ is a reflection that we should be asking all the time, because it helps us understand how to do it better next time.  When reviewing and reflecting, ask yourself- what were the false positives about that candidate that led you think that they were suitable but may not have been?  Were there any false negatives that led to declines of good candidates

Secondly, it’s a good tool for critiquing your process.  Questioning what tools you use to evaluate candidates is an important question to ask from time to time.  Even if you only do interviews, evaluate your questions (how accurate is the data I get from this question?).  Understanding what you’re gathering and whether it is accurate can save you a lot of pain in the long run.