Imagine a test where the questions you see depend on how well you’re doing. Get an item right, and the next item gets harder. Get it wrong, and the next one gets easier. The test calibrates itself to you, in real time, item by item, narrowing in on your true ability with each response.
This isn’t science fiction. It’s computer adaptive testing – CAT – and it has been quietly revolutionizing assessment for decades. The GRE, the GMAT, large parts of medical licensing, and increasingly large-scale international assessments like PISA all use some form of adaptive testing. Even Georgia ran a national CAT-based exam program for nearly a decade.
Most students who take these tests have no idea that the experience is being shaped to them in real time. They just see the questions. But under the hood, sophisticated psychometric machinery is making decisions about what they should see next, based on what they’ve already done.
How Traditional Testing Works
To understand what CAT changes, it helps to see what it’s changing.
In a traditional test, every student takes the same items, in the same order, regardless of ability. The test is constructed in advance through a process that combines content blueprints with psychometric analysis. Items are written, piloted, and analyzed. Through standard setting and pilot data, you learn which items are easy, which are hard, and how each item performs. You assemble a final form, and that form is the test. Everyone gets the same questions.
This approach has worked for over a century. It’s how most exams are still administered today. But it has a fundamental limitation: a single fixed test can’t be optimal for every test taker. A test that’s appropriate for the average student will be too easy for the strongest and too hard for the weakest. The information you collect is uneven, precise in the middle of the ability distribution, vague at the extremes.
The Core Idea of CAT
Computer adaptive testing solves this by personalizing the test on the fly. The items a student sees depend on their performance so far.
If you’re a strong student, the algorithm quickly moves you to harder items, because that’s where the most precise measurement happens for you. If you’re a weaker student, the algorithm gives you easier items, both because they’re more informative about your ability and because they’re more appropriate to your level. After each response, the system updates its estimate of your ability and selects the next item that will provide the most information.
The result is a test that adapts as you take it. Every student ends up with a different sequence of items, optimized for their individual ability level.
Why This Is Better
There are two main advantages of adaptive testing, and they operate on different levels.
The first is motivational. Traditional fixed tests can be frustrating in both directions. Strong students slog through items that are too easy and may lose engagement, feeling that the test is wasting their time. Weak students face items that are too hard and may give up entirely, once you’ve answered a dozen questions and feel you didn’t understand any of them, the temptation to stop trying becomes overwhelming. Adaptive testing keeps everyone in their zone of productive challenge: items are hard enough to engage strong students, but not so hard that weak students disengage.
The second is precision of measurement. This is the more important advantage from a psychometric perspective. When items are matched to the test taker’s ability level, each response carries maximum information about that person’s true ability. The result is more precise score estimates, particularly at the extremes of the distribution where fixed tests perform poorly. You can also discriminate more reliably between, say, two very high-performing students. This is something fixed tests struggle to do because their items are calibrated for the middle of the distribution.
Better motivation, better measurement. Both come from the same simple principle: give people items that fit them.
Why IRT Makes CAT Possible
None of this would work without Item Response Theory. CAT is technically possible only because IRT provides the underlying machinery.
Classical test theory, the older framework that most introductory statistics courses still focus on, gives you information about the total test score. It tells you how reliable the overall test is, how the score correlates with criteria, and so on. But CTT doesn’t give you what you need for adaptive testing – detailed information about each individual item, calibrated on a common ability scale.
IRT does exactly that. For each item, IRT estimates parameters like difficulty (where on the ability scale the item is most informative) and discrimination (how sharply the item distinguishes between students of different abilities). These parameters live on the same scale as student ability, which means at any point during the test, you can calculate exactly which item, given the test taker’s current estimated ability, will provide the most information.
This is the key insight: after each item, the system needs two things. It needs to update its estimate of the student’s ability based on the response. And it needs to select the next item that will be most informative given that updated estimate. IRT provides the framework for both of these computations.
Georgia’s CAT History
What many people don’t realize is that Georgia has its own history with computer adaptive testing. From 2011 to 2018, Georgia ran CAT-based school-leaving exams as part of its national assessment system. Students completing secondary education took adaptive exams to earn their school finishing certificate.
For nearly a decade, this was the only large-scale implementation of computer adaptive testing in Georgian education. It represented a genuinely modern approach to assessment at a national scale, something many countries with much larger psychometric infrastructures haven’t attempted. The program ended in 2018, but the technical and methodological work that supported it represents a significant chapter in the history of Georgian educational assessment.
That period is worth remembering, because it shows that adaptive testing isn’t just something that happens in wealthy Western testing programs. The capacity to design and run CAT-based national exams was developed and demonstrated here. The question now is what comes next.
PISA and Multistage Testing
PISA, the OECD’s flagship international assessment of 15-year-olds, has used a particular flavor of adaptive testing since 2018: multistage testing, or MST.
The difference between full CAT and MST is granularity. In full CAT, the test adapts after every single item – answer one question, the system updates its estimate of your ability, then chooses the next individual item. In MST, the test adapts after every block of items. Students complete a block of (typically) 15 or so items, the system estimates ability based on the block’s results, and then routes them to an easier, same-level, or harder block. Within each block, the items are fixed.
So PISA students might start with a routing block, then receive a stage-two block matched to their performance, then a stage-three block matched again. The test still adapts, but in larger chunks.
Why MST Instead of Full CAT?
Multistage testing has several practical advantages over fully item-adaptive CAT.
The first is content control. When you pre-assemble blocks of items, you can guarantee that each block covers the full content blueprint: appropriate proportions of different topics, item types, cognitive levels. With fully adaptive CAT, content coverage becomes harder to ensure, because the items you receive depend on your responses, not on a pre-planned content distribution.
The second is easier test review. Psychometricians and content experts can review complete forms, fully assembled blocks, before they go live. They can check that the blocks make sense, flow well, and represent the construct fairly. In full CAT, the “form” each student receives is generated on the fly, so this kind of review isn’t possible.
The third is lower operational risk. MST only requires routing decisions between stages: three or four points during the test where the system needs to make a decision about what comes next. Full CAT requires routing decisions after every single item. This makes MST simpler to implement, easier to debug, and more robust to technical issues during actual administration.
For a high-stakes international assessment with hundreds of thousands of students across dozens of countries, these practical considerations matter enormously. MST trades some of the theoretical efficiency of full CAT for substantial gains in manageability and content control. That’s the trade-off OECD chose for PISA, and it’s a defensible one.
The Downsides of Adaptive Testing
Adaptive testing isn’t a free lunch. There are real costs and constraints that limit when it makes sense to use.
The first and most obvious is the infrastructure requirement. Adaptive testing has to be done electronically. You can’t run a CAT on paper. This means every test administration site needs reliable computers, stable internet (or robust offline software), and the technical capacity to manage thousands of simultaneous test sessions. In many parts of the world, including in many schools globally, this infrastructure simply doesn’t exist yet.
The second is the item bank requirement. IRT-based testing in general needs more items than CTT-based testing, because each item needs to be precisely calibrated. CAT and MST need even larger item banks, because the system needs many items at every ability level to choose from. Building a high-quality calibrated item bank takes years of work, pilot testing, and ongoing maintenance.
The third is the piloting requirement. To estimate IRT parameters reliably, each item needs to be administered to a substantial number of test takers during piloting. CAT and MST require even more pilot data, because the precision of item parameter estimates matters more when those parameters are driving routing decisions. The number of test takers per item during piloting goes up significantly compared to traditional fixed-form testing.
These costs aren’t prohibitive for major testing programs like the GRE or PISA. But they do mean that CAT isn’t appropriate for every assessment context. A classroom teacher writing a midterm doesn’t need adaptive testing, and couldn’t reasonably build the infrastructure required for it.
Is It Worth It?
For institutions and educational systems that have the resources to invest in it, my view is clear: yes, adaptive testing is worth the investment.
The gains in measurement precision are real and substantial. Better measurement means better decisions – about placement, about progression, about diagnosis of strengths and weaknesses, about evaluating educational programs. When you’re making important decisions based on test scores, the precision of those scores matters.
For Georgia specifically, there’s also institutional memory worth preserving. The country has demonstrated, in the 2011-2018 period, that national-scale CAT is technically and operationally feasible here. That capability is a foundation that can be built on, not a one-time experiment.
The world of large-scale assessment is increasingly adaptive. PISA, TIMSS, PIRLS – all major international assessments are moving toward computer-based and adaptive designs. Educational systems that want their assessment infrastructure to match the international state of the art will need to invest in adaptive testing. The question isn’t whether adaptive testing is the future of assessment. It already is. The question is who’s prepared for that future.
—
Giorgi Tchumburidze
June, 2026