By Minda Zetlin
This is a sample of the type of question put to IT job applicants at RTTS, a professional services firm that specializes in software quality and testing:
A boat on the river in spring has a ladder hanging over the side. The bottom of the ladder is two feet above the water. The river is rising three inches per hour. How long will it take for the water to reach the ladder?
Because successful candidates will work on-site at RTTS's clients -- many of them Fortune 500 companies -- they must have both top-notch technical abilities and excellent communication skills, explains Bill Hayduk, director of professional services. "We won't get that just by speaking to people," he says.
To make sure RTTS gets exactly the employees that it wants, hiring is a five-phase process that begins with reviewing resumes. Phase two is an interview in which applicants display their technical acumen by discussing their most recent projects or describing their home computer setup. Each is asked the same set of questions, designed to elicit specific types of responses (for instance, a description of working with others on a project). Each answer is given a grade between one and three.
In phase three, those who pass phase two are asked eight easy technical questions: if they answer at least five correctly, they proceed to phase four, which includes a written programming test and 25 math and logic questions, such as the one about the ladder on the boat. (The correct answer is forever, since the ladder will rise at the same rate as the water level.) If they can do the programming and get 18 or more questions right on the test, they advance to the final phase.
Phase five consists of one more interview in which the applicant is asked to write code on a whiteboard while four or five RTTS executives look on -- a test designed to reproduce the high-pressure atmosphere at some of RTTS's clients. "Some people do really well," Hayduk says. "Others crack completely."
For RTTS, skills assessment ensures that the product it delivers -- its programmers and technicians -- will meet or exceed customer expectations. But there are lots of good reasons besides hiring to measure IT staff members' skills. Among them:
- For promotion: If they've spent 2,000 hours onsite at client companies, on their own, and are recommended by their supervisors, RTTS employees are offered the chance to take a further skills assessment. If they pass, they are automatically promoted to senior engineer and given a pay increase. If they fail, they can build up their weaker skills and try again the following year. "Skills assessments give people a visual path toward career advancement, and that helps with retention," notes Andres Fortino, associate provost at Polytechnic University in New York.
- To create a stronger, more flexible work force: Evaluating IT staff skills across the board can alert management as to what skills might be underrepresented in an organization. It can also make it easier to move staff from one area to another. "If an organization has loose definitions as to who's a programmer and who's an analyst, there will be some people in some part of the shop who have the title but are not as skilled as others," Fortino explains. "With declared competencies for each position, the organization can align for evenness. And it can avoid moving staff into jobs they can't do."
- During workforce reduction: Intel made news in September when an anonymous employee using the name "Intel IT Guy" blogged about how the company was selecting employees for downsizing. "We are going through a skills assessment process for each employee, scoring them, comparing scores, and then determining which skills we can most afford to lose from our individual groups," the blogger wrote. "It's unpleasant, painful work, and just not going well -- at least not for my team."
Though downsizing is never easy, skills assessment might be the best way to go about it, Hayduk argues. "I personally know every employee in my company, but I doubt the CEO of Intel can say that," he notes. "So how do you figure out an organized way to reduce staff? Tying a number to each employee might not be a great way to do it, but nobody's come up with anything better."
Getting assessment right
Of course, skills assessment is only useful if it accurately predicts the performance of the person being measured.
"A well constructed exam has a lot of science behind it," says Eric Wenck, vice president, IT and corporate business segment for Prometric, which provides testing to corporate, academic and government clients.
What goes into an effective assessment? Here are some elements to consider:
- Start with a model "The second year we were in business, we got lucky and hired the perfect employee," Hayduk recalls. "I said, 'How do we work backwards and ask questions to wind up with him?'" Once they'd devised the questions, the perfect employee tested the test by taking it himself. "He scored really well," Hayduk says.
- Judge by committee Unless someone is taking a test with specific right and wrong answers, an evaluation will necessarily be somewhat subjective. It can gain objectivity, however, if several people do the job instead of just one. That way, the score is less likely to be affected by things like mood, personality mismatches, or a well-meaning manager's reluctance to give someone a bad score.
- Don't go by numbers alone Hiring candidates who pass its skills assessment allows RTTS to ensure customer satisfaction, Hayduk says. But, he adds, the candidates who do best on the test are not always the best performers on the job, and vice versa -- which underscores the fact that candidates are human and capable of infinite variation, so it's important to consider intangibles, such as personality, as well as a skills assessment score.
"Can someone be reduced to a formula? The answer is no," Hayduk says. "We do our best to measure and to quantify. But there has to be some kind of fudge factor in there, too."
Minda Zetlin is co-author, with Bill Pfleging of The Geek Gap: Why Business and Technology Professionals Don't Understand Each Other and Why They Need Each Other to Survive (Prometheus Books).