Psychometrics: How Psychological Measurement Actually Works
- Written by
- IQCognify Editorial Team
- Reviewed for accuracy
- IQCognify Research Review Process
- Last updated
- Reading time
- 6 min
Quick answer
Psychometrics is the science of measuring things that cannot be observed directly — ability, knowledge, attitude, personality. Its central concepts are reliability and validity, and the single most important fact about them is one that almost every article gets wrong. A test is not valid. An interpretation of a test score, for a particular purpose, is what can be valid or invalid.
Validity is not a property of a test
This is the load-bearing idea of modern psychometrics, and it is routinely stated backwards.
Validity refers to the degree to which evidence and theory support the interpretations of test scores for proposed uses of tests.— Standards for Educational and Psychological Testing (AERA, APA & NCME, 2014)
Read the sentence carefully. Validity attaches to interpretations, for proposed uses. It does not attach to the instrument. The question is never “is this test valid?” It is always “is this interpretation of this score, for this purpose, with this person, supported by evidence?”
Why the distinction is not pedantic
The same WAIS score may be a valid basis for identifying an intellectual disability and an invalid basis for predicting who will be a good manager. Nothing about the test changed between those two sentences. What changed was the inference being drawn from it.
Once you hold this idea, a great deal of argument about intelligence testing resolves. The question of whether IQ tests are valid is malformed. The answerable questions are: valid for what, for whom, and on what evidence.
Reliability: consistency, not correctness
Reliability asks whether a test gives you the same answer twice. It says nothing whatever about whether the answer is worth having.
A bathroom scale calibrated four kilograms heavy is perfectly reliable. It will report the same wrong weight every morning. Reliability without validity is exactly that: consistent, repeatable, and about nothing.
| Type | The question it answers |
|---|---|
| Test–retest | Does the same person score the same on two occasions? |
| Internal consistency | Do the items within the test agree with one another? |
| Inter-rater | Do two examiners score the same performance the same way? |
| Alternate forms | Do two versions of the test give the same result? |
Reliability places a ceiling on validity
A test cannot correlate with anything else more strongly than it correlates with itself. Unreliable measurement attenuates every relationship it touches — which is why meta-analyses correct for it, and why doing that correction badly can invert a literature.
That last point is not hypothetical. In 2022 Sackett and colleagues showed that a systematic error in how one such correction was applied had inflated the apparent validity of cognitive ability tests across decades of research on hiring.
Does IQ predict success? — what happened when a statistical correction was audited after twenty-five years.Every score is a range
No measurement is perfect, and psychometrics is unusual among the sciences in insisting that its imperfection be reported alongside the result.
The standard error of measurement quantifies how much a person's observed score would bounce around across hypothetical repeated testings. A professional score report does not say “your IQ is 112.” It says 112, with a 95% confidence interval of roughly 107 to 117.
The consequence nobody draws
The difference between two index scores has a larger standard error than either score alone, because the errors in both compound. A twelve-point gap between a child's verbal and spatial indices may look like a profile. Statistically, it is frequently indistinguishable from nothing.
This is precisely the finding that independent factor analyses of the Wechsler scales have driven home. When the general factor accounts for around 70% of common variance and an individual index adds as little as 11% of unique reliable variance, the peaks and troughs that reports invite you to interpret are largely measurement error with names attached.
The WISC-V test — sixteen competing models, and why its authors say to interpret the Full Scale IQ.Norming: what makes a number mean something
A raw score is a count. Twenty-eight items correct. It becomes information only when placed against a reference group.
Standardisation means administering the test, under controlled conditions, to a sample chosen to represent the population — stratified by age, sex, education, region and other variables that matter. Your score is then expressed as your position within that distribution. David Wechsler's deviation IQ, introduced in 1939, is exactly this: mean 100, standard deviation 15, position on a curve.
- Norms decay. Populations change and score distributions drift, which is why tests are restandardised every decade or two — and why the Flynn effect makes an old test hand out scores that are too high.
- Norms are population-specific. A test normed in one country does not automatically yield meaningful scores in another.
- Norms are age-specific. Comparing a fifty-year-old to a twenty-year-old on raw performance would confound ability with the ordinary trajectory of cognitive ageing.
- Without a representative norm sample, a percentile claim is an extrapolation. This is the defect shared by essentially every free online test.
Latent variables: measuring what you cannot see
Nobody has ever observed intelligence. What is observed is performance on tasks. The inference from one to the other is the entire discipline.
Charles Spearman's insight in 1904 was that performance on any collection of diverse mental tasks correlates positively — the positive manifold — and that a single common factor could be extracted from those correlations. That factor, g, is a latent variable: a construct posited to explain a pattern in observations, not a thing anyone can point to.
Factor analysis is the tool that extracts such variables, and it comes with a warning that is easy to overlook. Different models can fit the same data adequately, and the choice among them is not settled by the data alone. When a test publisher reports only the models that support the structure it sells, that is a methodological problem rather than a finding.
The task impurity problem, generalised
Every task measures the ability you care about plus everything else the task happens to require. This is why serious research uses latent variables extracted from several dissimilar tasks rather than trusting any single score.
Myths and facts about psychometrics
| Myth | Fact |
|---|---|
| A test is either valid or invalid | Validity attaches to the interpretation of a score for a proposed use, not to the instrument. The Standards are explicit. |
| A reliable test is a good test | Reliability is consistency. A scale calibrated four kilograms heavy is perfectly reliable and perfectly wrong. |
| Your IQ is a precise number | Every score has a standard error of measurement. Professional reports give a confidence interval, typically about ±5 points. |
| A gap between two index scores is a cognitive profile | The difference between two scores has a larger error than either alone. Many apparent profiles are statistically indistinguishable from noise. |
| A score means something on its own | Only against a representative norm sample. Without one, a percentile is an extrapolation rather than a measurement. |
| Factor analysis proves a test's structure | Different models can fit the same data. Reporting only the models that support your structure is not evidence. |
Frequently asked questions
What is psychometrics?+
Psychometrics is the science of measuring psychological attributes that cannot be observed directly — ability, knowledge, attitudes, personality traits. It covers how tests are constructed, how scores are made meaningful through norming, and how the reliability and validity of score interpretations are established.
What is the difference between reliability and validity?+
Reliability is consistency: does the test give the same answer twice? Validity is meaning: is a particular interpretation of the score, for a particular use, supported by evidence and theory? A bathroom scale calibrated four kilograms heavy is perfectly reliable and completely wrong, which shows the two are independent.
Is a test valid or invalid?+
Neither, strictly. The Standards for Educational and Psychological Testing define validity as the degree to which evidence and theory support the interpretations of test scores for proposed uses. Validity attaches to an interpretation for a purpose, not to the instrument. The same score can support a valid inference for one use and an invalid one for another.
What is the standard error of measurement?+
It quantifies how much an individual's observed score would vary across hypothetical repeated testings. It is why a professional report states an IQ of 112 with a 95% confidence interval of roughly 107 to 117, rather than a bare number. No measurement is perfect, and psychometrics requires the imperfection to be reported.
Why should I not interpret differences between index scores?+
Because the difference between two scores carries a larger standard error than either score alone — the errors in both compound. When the general factor accounts for around 70% of common variance and an index adds as little as 11% of unique reliable variance, most apparent peaks and troughs in a profile are measurement error with names attached.
What is norming and why does it matter?+
Norming means administering a test under controlled conditions to a sample chosen to represent the population, so that any individual's raw score can be expressed as a position within that distribution. Without a representative reference group, a score is just a count of correct answers and any percentile claim is an extrapolation.
Why do IQ tests need to be renormed?+
Because population score distributions drift. The Flynn effect means an older test, normed against an earlier and lower-performing sample, hands out scores that are systematically too high. Restandardisation every decade or two keeps the reference group current.
What is a latent variable?+
A construct posited to explain a pattern in observations, which cannot itself be observed. Intelligence is one: nobody has ever seen it, only performance on tasks. Spearman's general factor, extracted by factor analysis from the positive correlations among diverse mental tasks, is the classic example.
Sources
This guide draws on standard psychometric references and peer-reviewed research:
- 1.American Educational Research Association, American Psychological Association, & National Council on Measurement in Education (2014). Standards for Educational and Psychological Testing.
- 2.Spearman, C. (1904). “‘General Intelligence,’ Objectively Determined and Measured.” American Journal of Psychology, 15.
- 3.Carroll, J. B. (1993). Human Cognitive Abilities: A Survey of Factor-Analytic Studies. Cambridge University Press.
- 4.McGrew, K. S. (2009). “CHC theory and the human cognitive abilities project.” Intelligence, 37(1).
- 5.Canivez, G. L., Watkins, M. W., & Dombrowski, S. C. (2017). “Structural validity of the Wechsler Intelligence Scale for Children–Fifth Edition: Confirmatory factor analyses with the 16 primary and secondary subtests.” Psychological Assessment, 29(4), 458–472.
- 6.Sackett, P. R., Zhang, C., Berry, C. M., & Lievens, F. (2022). “Revisiting meta-analytic estimates of validity in personnel selection: Addressing systematic overcorrection for restriction of range.” Journal of Applied Psychology, 107(11), 2040–2068.
- 7.Wechsler, D. (1939). The Measurement of Adult Intelligence. Williams & Wilkins.
- 8.Pearson — Wechsler Adult Intelligence Scale, Fifth Edition (WAIS-5), 2024.
- 9.Deary, I. J. (2020). Intelligence: A Very Short Introduction (2nd ed.). Oxford University Press.
- 10.American Psychological Association (APA)
Sources are provided for further reading. Organization links point to official sites; academic works are cited in full. See our research standards and editorial team.
Find out your IQ
Take the free IQ test and get your score, percentile, and a full cognitive breakdown in about 12 minutes.
Start Free Test