Planning
Specification Tables
To ensure content validity of a test it is necessary to ensure that items are representative of the curriculum. The specification table will confirm the representativeness of the evaluated learning by the test and produce the results obtained by students in relation to the combinations of items defined in the specification table.
In addition, the specification table allows to verify the congruence between the content of a test and the learning to be made as part of a program. If, in a course, students are asked to solve problems or to analyze concepts, the test should measure all these abilities or skills and not only those related to the acquisition of knowing (memorization, for example).
It may be noted that at the time of preparation of the databases that will be used by the EduStat software, wet took linking items (that is to say the variables) to the specification table. This information will perform statistical compilations into account these variables groupings.
The information related to the preparation of a specification table were grouped according to the following steps of preparation:
- Analyze the program to build a table of specification
- Establish the specification table
- Validate the specification table
The last section of this document summarizes the links established between an EduStat database and the table specific to the concerned test.
A. Analyse the curriculum
To build a specification table, we must first analyze the curriculum. This analysis aims to identify the learning that will make the measurement object and what is their relative importance, that is to say their weighting.
We have then to consider the elements constituting the program. However, it should not be limited to only components related to content. Also we determine what mental processes or intellectual skills that students need to treat these content: should they store or analyze them? Should they apply rules, solve problems, evaluate situations?
Also, ideally, we must take into account other elements that make up a program, such as the characteristics of the disciplinary approach or aims of the training. These elements have an impact on the format of the test or items.
Analysis of the program is based on the purpose of the test and selecting among the listed items, those that are most significant, representative or vital and that will be the measure used by the test.
We must first consider the related content (notions, concepts, themes) and then the intellectual abilities or skills that students need to exercise to treat these related content. Also, determine the relative importance or weighting of these elements.
a) The notional content
Depending of the intended purpose, the related content should be analyzed, selected and grouped into representative sub-groups to establish a list of concepts to measure.
To avoid asking too many questions on a secondary aspect of the discipline or neglect important aspects, we must remember what is essential and representative in the program.
The notional content selected and organized to be neither too general nor too specific. If they are too global, they will lead to multiple interpretations; if they are too specific, they will force designers to the test to retain a large number of questions.
When it is a summative examination which aims are to draw up a competency report at the end of session or academic year, it is better to enclose elements which often correspond but not always the terminal objectives or a combination of terminal objectives. During the school year, if we want to develop a test to identify students’ difficulties, it is more adapted to evaluate different concepts or steps of the thought process.
The following table provide an example of an analysis of the related content of the curriculum.
EXAMPLE
According to the notional result of the content analysis of a mathematics program (primary) for the preparation of a synthetic test, we get the following list:
Natural numbers
Geometry
Measurement
|
b) Intellectual skills or expertise
We must identify and explain the mental or intellectual skills processes that refer to the discipline in which students must address the related content. We have to fix which cognitive process students put into motion to achieve the planned learning in a given program.
Each skill or competence should match a list of events and observable behaviors. Intellectual skills usually refer to taxonomies. If the reviewed program does not explicitly refer to a particular taxonomy, we must select one or define an appropriate one.
The following table is an example listing the skills or competences retained touching a curriculum.
EXAMPLE
Analysis of the required intellectual skills of students in a mathematics program for primary education leads to the following list: UNDERSTANDING - Knowledge of mathematical concepts and make connections between them. Students who master this skill manifest observable behaviors such as those listed below:
APPLICATION - Carry or designated transactions or data transformations. Students who master this skill manifest observable behaviors such as those listed below:
SYNTHESIZE - Linking a given solution and the problem, or find a solution to a given problem. Students who master this skill manifest observable behaviors such as those listed below:
|
c) The weighting of related content and intellectual skills or expertise
We must determine the weighting or relative importance for related content and intellectual skills or abilities. The weight allocated and used to determine the importance of each content and each skill or competence within the test and the number of questions to design for each component. An example of such a weighting integrating the two tables above.
EXAMPLE
In mathematics at the primary level, there were assigned the following weights to related content and intellectual skills:
The notional content
The intellectual skills
|
The relative importance of program elements has a direct impact on the importance will have each question in relation to the whole test. If given the same number of points to each question, there is a proportional number of questions corresponding to the weights.
The weighting of program components may already be indicated in the program. Otherwise, it must be determined with the help of content experts. In one case as in the other, it may be necessary to revise the weighting with the light of the purpose of the test. It's basically a matter of consensus among experts.
At the end of the step to analyze the program, one should have an idea of what to measure with a test: what are the related content checking and how students must treat. This step led to the development of a specification table.
B. Establish the specification table
The specification table is a planning element for structuring data relevant for the development of a test. It is, somehow, a design on notional contents and levels of cognitive functioning attached to each item. The specification table is an efficient way to specify the sample of questions to include in a test.
Specifically, the specification table is a table usually with double entry: one for the list of related content to master, the other to the list of intellectual skills or expertise to measure. It also indicates the relative importance or weighting of related content and intellectual abilities or skills.
The specification table also shows the overlap between related content and intellectual abilities or skills. Called "dimension", it is the intersection of skill or competence and content. Each precise dimension measurement object from which the questions will be drafted.
It is possible that a dimension is empty, that is to say the intersection of skill or competence and a concept does not exist for the discipline to evaluate. So this is a dimension that will not be used in order to measure the objects in the test.
A specification table is usually present in the form of a pattern such as this one.
SPECIFICATION TABLE
| Notion A N (%) | Notion B N (%) | Notion C N (%) |
Skill or competence A N (%) | dimension 1 | dimension 2 | dimension 3 |
Skill or competence B N (%) | dimension 4 | dimension 5 |
|
Skill or competence C N (%) | dimension 6 |
| dimension 7 |
The use of the specification table when developing a test ensures the link between the test and the program. Indeed, if we prepare questions according to program goals covering all the specification of array dimensions while respecting the weights indicated therein, it is ensured that the test will be the true picture of the program. The specification table is a set of data from which several tests involving the same program can be developed.
In addition, the specification table is of great value at the time of analysis. It helps to identify strengths and weaknesses of a student or all students who use the test. Indeed, as we know how dimension refers to each question of the test, we can determine from failed and successful questions, what are the content or skills that are not problematic or not.
Here is a sample specification table. The numbers of items attached to various dimensions are indicated in the table.
EXAMPLE
In this table of specification of maths test to teach primary level, we see that there are seven dimensions from which questions were drafted to ensure student learning in this discipline. Mathematics in primary education | ||||
CONTENTS SKILL OR COMPETENCY |
Nombers |
Geometry |
Mesurement |
Total (N %) |
Understanding | # 1, 4, 7 | # 5 | # 6 | 5 (50%) |
Application | # 2, 8 | # 9 |
| 3 (30%) |
Synthesize | # 3 |
| # 10 | 2 (20%) |
Total (N %) | 6 (60%) | 2 (20%) | 2 (20%) | 10 (100%) |
C. Validate the table of specification
To ensure content validity of the specification table, we have to consult the experts: teachers, specialists of the discipline. The consensus between content experts is the best way to check, depending on the intended purpose, if the table is representative of learning linked to the program and if it really reflects the goals of the training.
More specifically, the validation of the table covers the following elements:
- The representativeness of learning provided in the program:
- the related content
- the intellectual skills or expertise
- the links between content and skills or competencies (dimensions)
- the aims of the formation
- Weights according to :
- notional content
- intellectual abilities or skills
We provide below a questionnaire with items that will validate the specification table of a test.
According to the purpose of the test, answer the following questions:
Are selected representative elements of the program? |
|
|
|
| Yes | No | Uncertain |
| Yes | No | Uncertain |
| Yes | No | Uncertain |
Is the list of selected elements accompanying the specification table exhaustive and unambiguous? |
|
|
|
| Yes | No | Uncertain |
| Yes | No | Uncertain |
Are the skills or competencies translated in terms of observable behaviors? | Yes | No | Uncertain |
Are the skills or competencies in relation to taxonomic levels? | Yes | No | Uncertain |
Are withheld relevant dimensions? (Do they really represent objects of measurement in connection with the program?) | Yes | No | Uncertain |
The relative importance or weight given to elements of the specification table is it relevant? |
|
|
|
| Yes | No | Uncertain |
| Yes | No | Uncertain |
Links between the table of specification and EduStat database
At the time of preparing a database associated to a test, each question should be associated with certain parameters that allow the processing of the information.
Compared to the specification table, the EduStat software includes three groupings. Here is the information provided to the software according to each grouping.
Groupings
The first group concerns the "domains". The mathematics test covers three areas:
- Arithmetic
- Metric system
- Geometry
The second grouping lists "skills" evaluated by the test; it is thereof:
- Knowing
- Comprehension
- Application
- Analysis
- Synthesic
The third set was not selected for this test.
Email address: info@mcpythagore.com
