Creating and maintaining psychometrically sound examinations suitable for professional certification is no small feat. Here’s an overview of the thorough research, expert input, and many-tiered review that goes into the process.
How we develop our tests
We solicit questions from certificants and from people recognized as leading experts in the field—a status we judge by way of their publishing or presenting information about dog training and behavior. The questions are drafted in multiple-choice format, including a reference.
The raw questions (also known as “items”) are submitted to a psychometrician at our testing company who prepares them in proper question format and places them in our Item bank, marked “ready for review.”
We convene a meeting of subject matter experts who have previously passed the examination, and they review the items. The meetings are held either online or in person, which enables us to select experts from different parts of the country and the profession, thus bringing many perspectives to the discussion. The subject matter experts review all items against the content outline for the examination and retain only questions that pertain to it.
Next, they look at wording to make sure the question is succinct and clear, that the four answers are plausible and parallel, and that there’s one and only one correct answer. Once the experts have gone through all items and made their edits, the items are placed back in the Item Bank marked “ready for examination.”
An examination review panel approves each new form of an examination before it’s administered. They convene an in-person meeting, led by a psychometrician from the testing company. During that meeting they are presented with a draft copy of a new form of the examination, with the questions weighted according to the test content outline.
The review panel goes over every question on the examination, ensuring once again that they are appropriate for the examination, are clearly written, and have only one correct answer. This entire process of examination development constitutes the validation stage.
Next, the examination review panel looks at the statistical performance of the previous form of the examination, noting things like the high and low scores, reliability numbers, and a breakdown of performance by domains for both the passing and failing candidates. After reviewing the statistics and going through a psychometrically valid exercise,the board determines the passing score for the newly approved examination.
Following the administration of a new form of the examination, the psychometricians at the testing company look at the performance of each item on the test. If any item performed poorly (fewer than 40% of the candidates answered it correctly), that item is referred back to the subject matter experts for additional review. In most cases, the question is simply a difficult question and the correct answer stays as it is. But occasionally the board finds the wrong answer identified in the answer key and the testing company then changes this.
Only after this final validation of the questions on the examination is complete do we score the exams and send score reports to candidates.
Test development glossary
Psychometrician: A psychologist who devises, constructs, and standardizes psychometric tests. Psychometric tests are a scientific method for measuring an individual’s knowledge.
Body of Knowledge: A term used to denote the core teachings, skills, and research in a professional field or industry.
Testing company: A company that provides a range of services around professional testing and examination. For example test development and administration, psychometric consulting, and role delineation studies.
Role delineation study: A systematic collection of data that describes the tasks carried out by and responsibilities born by a professional in a given field.
Test development FAQs
Can I contribute a question?
Absolutely, if you’re a current certificant. We urge certificants to write and submit questions. The only stipulation is that questions must pertain to a section of the content outline and cite a reference. We then place your questions (“items”) into our Item Bank, marked “ready for review,” and they go through the above mentioned vetting process.
What does it mean that a test is psychometrically sound?
A sound examination meets two important criteria: validity and reliability. Validity is the term for ensuring that the examination covers what it should cover. That’s the process of item reviews and examination reviews described above. Reliability refers to the consistent performance of the examination over time. A reliable examination is one in which, if you were to have two groups of test takers who are similar in background and experience take the same examination, their scores would be the same. The reliability coefficient on the last examination was 0.95, which demonstrates high reliability.