Some of you may have heard about the promise of Automatic Item Generation (AIG)… some of you may have heard me give various presentations over the years about AIG and its application to content development in high stakes testing. This blog is about a different angle in using AIG to build out a test bank, one that avoids the need to purchase software, or even to train your subject matter experts on how AIG works (although that never hurts!). I’ll explain, but first let me give a high-level description of AIG.
AIG is like item cloning but on a much bigger scale. In item cloning, a well-performing existing item is chosen and a few words in the stem and alternative are ‘swapped out’ in order to create a new question, based on the same structure as the original. AIG takes this cloning idea much (much!) further, by identifying variables in the stem that could take on different values and specifying a set of possible values for those variables, and by creating a larger option set to choose from when any particular question is made.
Here’s an example of an AIG item model, a framework that has the potential to create many items.
A patient is experiencing respiratory distress and is exhibiting the following signs and symptoms: S-S1, S-S2, S-S3. What is the most likely cause of this patient’s respiratory distress?
a. <Cause1> **Key
b. <Cause2>
c. <Cause3>
d. <Cause4>
As you can see, there’s no specific question there, but many such questions could easily be derived if the model is fed with the right information. And here’s the ‘right information’:
Even a quick inspection of these two elements, the item model and the above table reveals that many, literally thousands of unique items could be created by varying the symptoms and causes to be inserted into the item model. This is the essence of AIG and why many folks in testing find it so compelling.
But my guess is that you’re not using AIG to develop your own test content, even if you know about the method. And I have a theory about that… you’re just too busy. You don’t have time to learn about item and task models, how to use a new piece of software to build these items, you don’t want to figure out how to sort through the 1000’s of items that any one template might create. You’re too busy and your item development program is working just fine, mostly. Except you could use more items.
So, here’s a new approach, brought to you by Spire Psychometrics. You and I sit down in a webinar for an hour or two and I teach you how to create a table like the one above on Respiratory Distress. It could be about anything you need items for, and the rows and columns could represent almost anything. (I recently helped a client develop a table like this that described the elements and sequence of stages in negotiating a settlement. And also one about relating specific examples to styles of communication.) Then, you leave it to me. I consider the structure of the table and the content and select item models accordingly. To take the respiratory distress example, I might choose the above item model, but also:
Which of the following signs or symptoms is associated with <cause of respiratory distress>?
a. <S-S1>
b. <S-S2>
c. <S-S3> **Key
d. <S-S4>
A patient is experiencing respiratory distress and is presenting with S-S1 and S-S2. What additional information would definitively determine the cause of this patient’s respiratory distress?
a. <S-S1>
b. <S-S2>
c. <S-S3>
d. <S-S4> **Key
Then, I would generate say 50 very distinct items for you to incorporate into your item bank as experimental items. Beyond a cursory examination of the item set for validation, you would have no need for all the arduous steps required in item development like editorial and group review, never mind learning about the nuances of AIG. And those items will be much (much!) cheaper than your usual per item costs.
Want to learn more? Call me at 250.307.0076, email at greg@spirepsychometrics.com, or visit me at my virtual booth at CNAR 2020!