Substantive stage

Theory-based (including previous research and observations)

Generate theoretical and empirical definitions

Gather content-related evidence

Consider construct’s under-representation and relevancy

Step 1

Identify the behavior(s)/construct(s) you want to measure (Theory-based construct or observation)

Step 2

Identify the behaviors to represent the construct(s)

Step 3

Define the behavior(s) and identify the proportion of items needed to measure the construct(s).  Generate theoretical and empirical definitions and develop a set of test specifications.

Step 4

Generate an item pool

Step 5

Determine guidelines for scoring and interpretation

Step 6

Conduct an expert evaluation (cognitive interviews, expert review, item-content matching, . . .)

Step 7

Consider the inclusion of validation items “social desirability”

Step 8

Preliminary item try out and revise as necessary (wordings, understanding, . . .)

Structural stage (internal relations among the questions)

Item/subscale intercorrelations (item analysis-alpha/ reliability)

Generalizability theory (reliability analysis)

Item response theory (item analysis - reliability)

Exploratory factor analysis (not hypothesis driven)

Confirmatory factor analysis (hypothesis driven)

External stage (relations with other constructs)      

Correlations with other tests (partial correlations, regressions, ...)

Multitrait-multimethod matrix

Group differentiation (Anova, Ancova, Manova, Mancova, Discriminant analysis, ...)

existing or know groups

experimental manipulations

Structural equation modeling

Step 9

Develop a validation plan (sample size, sample characteristics, stratifications, . . .)

Step 10

Validation:  Include more than one method in each of the group of methods presented above.

Step 11

Assess the behavioral stability of the test (short and long term)

Step 12

Determine the preliminary psychometric properties of the test, revise and redo some of these steps as needed.