The predictive validity of cognitive ability tests: a UK meta-analysis.

Several recent surveys indicate that tests of general mental ability (GMA) and tests of specific cognitive abilities (e.g. numerical, verbal, spatial, etc.) are increasingly popular amongst employer organizations in the UK for selection and assessment purposes (e.g. Hodgkinson & Payne, 1998; Keenan, 1995; Ryan, MacFarland, Baron, & Page, 1999; Salgado & Anderson, 2002; Salgado, Ones, & Viswesvaran, 2001). Whereas in the USA numerous meta-analytic studies have provided predictive and criterion-related validity evidence to support the use of GMA tests in selection (e.g. Hunter & Hunter, 1984; Schmidt, 2002; Schmidt & Hunter, 1998), there has been a notable absence of validity generalization studies in the UK. This is a serious shortcoming in our understanding of the predictive efficacy of such tests. Given their increasing popularity amongst employers, selection psychologists and test suppliers in the UK are potentially open to claims of relying upon tests which have not been fully validated through independent meta-analytic procedures combining multiple proprietary tests.

US meta-analyses of cognitive ability tests

A number of meta-analyses have been carried out in the USA investigating the criterion-related validity of GMA and cognitive ability tests (see Schmidt, 2002, Appendix A, for a comprehensive summary of past findings). Amongst these, the largest meta-analyses based on occupational samples are those conducted by Hartigan and Wigdor (1989), Hunter (1986), Hunter and Hunter (1984), and Levine, Spector, Menon, Narayanon, and Canon-Bowers (1996). Overall, these have shown that the average operational validity for GMA and cognitive ability tests ranges from .38 to .47 for overall job performance and from .54 to .62 for training success (re-estimated using Hunter & Hunter's criterion reliability and range restriction estimates). Furthermore, Hunter and Hunter (1984) demonstrated that despite differences in jobs and organizations, the predictive validity of GMA and cognitive ability tests generalizes across samples and settings. Consequently, it has been concluded that GMA and cognitive ability tests are robust predictors for all types of jobs (Salgado, 1999; Salgado, Ones, & Viswesvaran, 2001; Schmidt & Hunter, 1998), and that their validity generalizes across occupations in the USA. However, despite the large body of evidence supporting the validity of GMA and cognitive ability tests, there are a number of limitations within the current body of research. Firstly, there has been a general tendency towards examining general mental ability as a predictor of work behaviour, as opposed to the predictive validity of specific cognitive abilities. Secondly, only limited research has examined the predictive validity of GMA and specific cognitive abilities across different occupational groups. Finally, and perhaps most importantly, in examining these issues, there has been a general reliance on predominantly US samples (Anderson, Born, & Cunningham-Snell, 2001; Schmidt, 2002). As highlighted by Herriot and Anderson (1997), the findings from US meta-analyses have been unreservedly cited as being generalizable to the UK, without consideration of possible cultural, social, legislative, and recruitment and appraisal differences between countries. These differences, it can be argued, may well impact on the magnitude of validities observed in GMA test validity between the USA and UK (see also Salgado & Anderson, 2002, 2003).

European and UK meta-analyses of cognitive ability tests

A comprehensive review of the published studies revealed that no previous meta-analysis which considered the criterion-related validity of GMA tests in the UK has been published. Robertson and Kinder (Robertson & Kinder, 1993; see also Salgado, 1996) published a meta-analysis using data collected in the UK, but this meta-analysis focused on the validity of personality measures. Their meta-analysis did, however, examine the incremental validity of personality measures after partialling-out the variance in the criterion measure attributable to cognitive tests. In their series of recently published papers, Salgado, Anderson, and colleagues have investigated the criterion-related validity of cognitive tests across other countries in the European Union, but no UK-specific meta-analysis appears to have been published to date (Salgado & Anderson, 2003; Salgado, Anderson, Moscoso, Bertua, & de Fruyt, 2003a; Salgado et al., 2003b). This is undoubtedly a notable shortcoming in our understanding of the efficacy of cognitive ability tests for employee selection in the UK. According to Levy-Leboyer (1994), there are important differences between the US and European organizations in how selection procedures are carried out. This is borne out by subsequent analyses by Salgado and Anderson (2002) into the popularity of cognitive ability tests in Britain, Europe, and the USA as indicated by previous surveys of GMA test use in these countries. Across 16 major surveys conducted over the last 25 years, Salgado and Anderson found that organizations in the UK tended to use GMA measures substantially more than organizations in the USA, despite the dearth of British meta-analytic evidence to support this widespread popularity. Viswesvaran and Ones (2002) have further pointed out that countries in the European community, if considered individually, are relatively homogeneous compared with the USA as they have less within-country diversity. Of any European country, of course, it can be argued that the UK is closest to the USA in terms of its employment legislation (having opted out of the EU Social Chapter, for instance), hours of work, job security, and human resource management practices. As noted by Roe (1989) selection practices and perspectives in other European countries follow less the classical American predictivist model. Instead, they emphasize the social negotiation perspective (e.g. Herriot & Anderson, 1997), prospective employee rights in the procedure, and applicant privacy and expectations of equitable and fair treatment by the prospective employer organization (Levy-Leboyer, 1994). Other researchers have argued that another relevant difference is the difference in size typically between US and European organizations (see, for instance, Salgado et al., 2003a, 2003b). Again, comparisons between the UK and the USA are particularly interesting given the cultural differences between the UK and other European countries, and the adoption by UK organizations of American HR procedures and working practices. Several of the tests upon which primary studies were based in our dataset were either developed in the USA but are popular in the UK for GMA measurement (e.g. the Minnesota Clerical Test, the Differential Aptitude Test, Bennett's Mechanical Comprehension Test), or were UK-developed but are now used also in the USA (e.g. Raven's Progressive Matrices: Jensen, 1998). These overlaps further suggest that similar predictor-criterion relations could be expected across both countries.

Issues concerning the theoretical groundings, development, and use of cognitive ability measures for employee selection have been at the forefront of debate in US industrial, work, and organizational psychology recently (e.g. Ones & Viswesvaran, 2002; Viswesvaran & Ones, 2002). Indeed, the journal Human Performance has published a seminal special issue entirely dedicated to the role of GMA in selection and job performance. Given that cognitive tests are used considerably more extensively for selection in Britain than in the USA, it is timely and fitting that debate in the cultural and legislative context of the UK is encouraged. Indeed, major issues such as criterion-related validity, adverse impact, test construction, validation procedures, and claims for the efficacy of cognitive tests for employee selection in the UK have received scant attention (see for instance, Murphy, 2002; Ones & Anderson, 2002; Reeve & Hakel, 2002). As will be highlighted in the following sections, such limitations necessitate a comprehensive analysis of these issues. What is more, in view of the lack of comparable meta-analyses conducted on British samples, a country specific analysis of the validity of GMA and specific cognitive ability tests is warranted in order to accurately assess the predictive validity of such tests in the UK. Therefore, the current investigation sought to address these limitations by conducting the first independent and comprehensive metaanalysis of GMA and specific cognitive ability tests across a range of occupations consisting exclusively of UK samples.

General versus specific cognitive abilities

An extensive body of research conducted over the last 50 years has led to the general consensus that cognitive abilities manifest a hierarchical structure (see for example, Carretta & Ree, 2000; Carroll, 1993; Jensen, 1998; Ree & Carretta, 1998). In conjunction with this, many tests have been developed to measure both GMA and specific cognitive abilities, such as numerical, spatial, verbal, and perceptual ability. However, even in the USA, in contrast to the extensive research regarding the predictive validity of GMA, very little research has been conducted examining the predictive validity of specific cognitive ability tests. For example, Hunter and Hunter (1984) and Hartigan and Wigdor (1989) partially examined this issue by examining the predictive validity of a cognitive ability composite and a perceptual ability composite (as assessed by the GATB) within civil settings. The results from both of these studies revealed that the perceptual ability composite had generally lower predictive validity than the cognitive ability composite. For example, in Hunter and Hunter's (1984) presentation of the US Employment Service validation studies, the mean validities found for the cognitive ability composite ranged from .23 to .58 for job performance, and from .50 to .65 for training success (depending on the job complexity). However, in the case of the perceptual ability composite, mean validities ranged from .24 to .52 for job performance and from .26 to .53 for training success. A further piece of research which supports the conclusion that perceptual ability tests have generally lower predictive validity than general cognitive ability is Hunter's (1980, 1984, cited in Hunter, 1986) reanalysis of Ghiselli's data (1966, 1973). These results revealed that for general cognitive ability validities ranged from .27 to .61 for job performance, and from .37 to .87 for training success (corrected for measurement error and range restriction). However, for perceptual ability, lower estimates ranging from .20 to .46 were found for job performance.

On the question of general versus specific cognitive abilities as predictors of subsequent job performance, findings from meta-analyses conducted in the US have been unequivocal. Several studies indicate GMA to be the most robust predictor with specific abilities adding little or no incremental validity to predictor-criterion relationships (e.g. Carretta & Ree, 1996; McHenry, Hough, Toquam, Hanson, & Ashworth, 1990; Olea & Ree, 1994; Ree & Carretta, 1994; Ree & Earles, 1991; Ree, Earles, & Teachout, 1994). However, tests of specific cognitive ability are highly popular for selection purposes in the UK, with for instance, many organizations using notionally separate tests of verbal, numerical, and abstract reasoning (that is, regardless of underlying construct correlations with g). Meta-analyses in the USA have typically examined the issue of the incremental validity of tests of specific abilities, however, not their 'stand-alone' validity ff used by selection practitioners as multiple tests of different aspects of cognitive ability. This is typically the way in which specific ability tests are used for selection in the UK, regardless of existing findings that specific abilities correlate very highly with GMA.

Although not examining the validity of specific cognitive ability tests across a range of job groupings, some research has been conducted within narrower job groupings (e.g. Hirsh, Northrop, & Schmidt, 1986; Levine, Spector, Menon, Narayanan, & Cannon-Banister, Slater, & Radzan, 1962.; Pearlman, Schmidt, & Hunter, 1980: Vinchur, Schippmann, Switzer, & Roth, 1998). For example, Levine et al. (1996) examined the criterion validity of perceptual and cognitive ability tests for craft jobs in the utility industry. In their study, they found that perceptual tests demonstrated a corrected validity of .34 when predicting job performance, and .36 when predicting training success. However, these validity estimates may not accurately represent the predictive validity of perceptual ability tests, since the classification of tests under their perceptual test category is problematic.

The main conclusion to be drawn from these US results is that …