Creating and maintaining psychometrically sound examinations suitable for professional certification is no small feat. Here’s an overview of the thorough research, expert input, and many-tiered review that goes into the process.
How We Develop Our Tests
Step 1: Job Analysis/Role Delineation
A valued and legally defensible credentialing examination is built upon a well-designed, current job analysis. The periodic performance of a job analysis study is critically important to the validity and defensibility of a certification examination program and ensures that the certification program stays up-to-date and is reflective of current best practices.
The job analysis process begins with a properly documented, accurate, broadly agreed upon delineation of the body of knowledge essential to competent practitioner roles and functions. The process of job analysis results in identification of the major domains of practice. The accompanying task statements and knowledge area statements identify the knowledge, skills, and abilities needed for competent performance. The content outline is then validated by professionals in the industry. Results from the study are used to develop the content outline and corresponding test specifications for the examination.
Step 2: Item Writing Training and Item Writing
We solicit questions (also known as “items”) from certificants and from people recognized as leading experts in the field—a status we judge by way of their publishing or presenting information about dog training and behavior. The questions are drafted in multiple-choice format and include a reference.
The items are submitted to test development specialists at our testing company who prepare them in proper question format and places them in our item bank, marked “ready for review.”
Step 3: Item Review
We convene a meeting of subject matter experts who have previously passed the examination, and they review the items with the facilitation and guidance from a test development specialist from our testing company. The meetings include experts from different parts of the country with varied expertise in the profession, thus bringing many perspectives to the discussion. The subject matter experts review all items against the content outline for the examination and retain only the questions that pertain to it.
Next, they look at wording to make sure the question is succinct and clear, that the four answers are plausible and parallel, there is no bias or discrimination in the item, and that there’s one and only one correct answer. Once the experts have gone through all items and made their edits, the items are placed back in the item bank marked “ready for examination.”
Step 4: Examination Development and Review
A psychometrician from the testing company constructs new forms of the examinations from items in the item bank. Examinations are constructed paying attention to factors such as the number of items in each content area and performance of items on past examinations, including the difficulty level and how effectively they discriminated between candidates with higher and lower performance levels. To ensure fairness to all candidates, items are then selected to cover equivalent content as well as equivalent difficulty to match difficulty levels between different forms of the examination.
An examination review panel approves each new form of an examination before it’s administered. They convene a meeting, led by specialists from the testing company. During that meeting they are presented with a draft copy of a new form of the examination, with the questions following the test content outline.
The review panel goes over every question on the examination, ensuring once again that they are appropriate for the examination, are clearly written, are free from bias and discrimination, and have only one correct answer. They will also check that the items are still current. New items are validated by administering them as pre-test items, meaning they are not scored on the examination and do not contribute to a candidate’s scores.
Step 5: Standard Setting
A standard setting committee is convened to set an appropriate passing score for each new examination form after the job analysis is conducted. A criterion-referenced methodology is used, which is the most widely acceptable methodology used on certification examinations. With this approach, a passing score is set by determining the number of items that must be answered correct in order to pass the examination. All forms after this are then equated back to this form.
Since every examination form is made up of a different mix of items, the difficulty level may vary slightly from form to form. Equating takes into consideration these small and unintended differences in difficulty among different examination forms by appropriately adjusting the passing point to ensure comparable levels of knowledge on each form is required to pass the examination. With equating, once the standard is set using the criterion-referenced methodology, the same standard will be upheld for future examination forms.
Step 6: Scoring
Following the administration of a new form of the examination, the psychometricians at the testing company look at the performance of each item on the test. If any item performed poorly, that item is referred back to the subject matter experts for additional review. In most cases, the question is simply a difficult question and the correct answer stays as it is. Occasionally the board finds the wrong answer identified in the answer key and the testing company fixes it.
Only after this final validation of the questions on the examination is complete are the exams scored and score reports sent to candidates.
To ensure fairness and consistency across examination forms, a statistical process called equating is used to account for any slight variations in difficulty level across forms. Scores on the examination are reported using scaled scoring, which converts the candidates’ raw score (i.e., total number of correct questions) onto a consistent and standardized scale. Scaled scores allow candidate scores to be comparable from one exam form to the next. The scale range for the examination is 200 to 800 with a passing point of 500.
Subject Matter Expert (SME) Recruitment
The Certification Council for Professional Dog Trainers (CCPDT) recognizes the need for, and is committed to, the goal of producing examinations that reduce bias and respect diversity, equity, and inclusion (DEI). The CCPDT does not discriminate against any individual based on race, color, ethnicity, national origin, gender or gender identity, age, disability, physical limitation, marital or familial status, sexual orientation, religion, political beliefs, or other characteristics prohibited by law.
CCPDT recruits and enlists subject matter experts (SMEs) representing various geographic locations, practice settings, years and types of experiences, educational backgrounds, genders, races, ethnicities, and ages. SMEs are given the option to provide demographic details that help confirm that the CCPDT is meeting its goals. In addition, SMEs are trained in item writing best practices, including strategies for ensuring that items are free from bias and stereotyping. Item and exam reviews further serve to catch and control for possible bias. CCPDT conducts reviews of the optional candidate demographic questions asked on the exam application, updating periodically to promote inclusivity. Candidate demographic data is aggregated anonymously for psychometric analyses, including differential item functioning (DIF) analyses, to identify and rectify possible biases in the examination.
Test Development Glossary
Body of Knowledge: A term used to denote the core teachings, skills, and research in a professional field or industry
Content Outline: The domains, task statements, and knowledge areas and what percentage of questions on the exam are devoted to each domain; also known as test specifications or test blueprint
Domains: The main categories into which the tasks are grouped
Knowledge Areas: The subjects or skills that a dog training professional or behavior consultant draws upon to be able to perform a task or work activity
Psychometrician: A psychologist who devises, constructs, and standardizes psychometric tests. Psychometric tests are a scientific method for measuring an individual’s knowledge
Role Delineation Study: A systematic collection of data that describes the tasks carried out by and responsibilities born by a professional in a given field; also known as a job analysis study or a job task analysis study
Scaled Scores: A raw score that has been transformed so that different forms of the examination can be reported on a common metric; this allows comparisons of scores across different examination administrations
Task Statements: A specific work activity performed by the dog training professional or behavior consultant
Testing Company: A company that provides a range of services around professional testing and examination, including test development, test administration, psychometric consulting, and role delineation studies
Test Development FAQs
Can I contribute a question?
Absolutely, if you’re a current certificant. We urge certificants to write and submit questions. The only stipulation is that questions must pertain to a section of the content outline and cite a reference. We then place your questions (“items”) into our item bank, marked “ready for review,” and they go through the vetting process outlined above.
What does it mean that a test is psychometrically sound?
A sound examination meets two important criteria: validity and reliability. Validity is the term for ensuring that the examination measures what it intends to measure. Reliability refers to the consistent performance of the examination over time. A reliable examination is one in which if you were to have two groups of test takers who are similar in background and experience take the same examination, their scores would be comparable. The reliability coefficient for each examination administration is reviewed to ensure it meets psychometric guidelines.
Where can I learn more about the most recent job analysis/role delineation studies?
Download the CPDT-KA RDS Executive Summary 2018
Download the CBCC-KA RDS Executive Summary 2018