If you’re doing manual test assembly, chances are there is a significant gap between the test that you’ve assembled and a test that does the best job of measuring what you intend. Of course, it doesn’t seem like this should be the case. After all, you’ve got a blueprint and a pool of items to fulfill it. However, the problem is not so much what you’ve been able to accomplish with manual assembly; it’s all the variables you’ve overlooked.
You mean I’ve got choices?
One of the main reasons that manual assembly isn’t such a great idea is that humans are just not that well-suited to make choices involving a large number of possibilities. And test assembly involves a lot of possibilities. A quick reference to the mathematics of combinations: if you had a question bank comprising 200 items and needed to choose 100 items to make a test, you would have 9.055 x 1058 possible ways of doing that. Yes, you read that correctly. And, when the set of possible solutions is astronomically large, how is a human supposed to choose a good one? How would you even know that the one you chose was good when compared to what’s possible?
This is the situation you’re in when manually selecting items for tests. You may have A solution, but that solution may not be particularly good, and you probably won’t even know it’s not very good.
So what makes a good solution?
Manual test assembly only considers a small number of variables in order to make the selection tractable. In the example above, only the exam blueprint was considered. But many other factors affect the quality of your test. For example, you might want to have an equal number of questions about males and females; children, adults, and the elderly. You may want your exam to share 30 questions with some baseline exam, have an approximately equal number of As, Bs Cs, and Ds, and have a particular number of questions that are tied to passages or images. You also may want this exam to be no more difficult than the last one.
Automated test assembly allows you to take account of all these variables (and many more) by specifying to the software the kinds of solution characteristics that you would find acceptable… i.e., good solutions. The best thing about ATA? It’s easier than manual. It’s especially easier if you just get Spire to build it for you. ;-) But once you have all the constraints and characteristics of the test built, you can have the tool select the items to review. If during review, you want to remove a couple of them, a new version can quickly be generated that excludes the unwanted items but still meets the constraints that you set out. So you can use your time to validate the test rather than copy and paste items from one spreadsheet to another.
If you’d like to learn more about ATA, or to learn about ways to refine your existing test assembly process, contact the experts at Spire! info@spirepsychometrics.com