Schedule and Presentations 2007
Overview: This year’s workshop will focus on psychometric techniques used to refine screening instruments so that they maximize sensitivity and specificity for dementia across linguistic and educational groups. We will use data from the Canadian Study on Health and Aging (CSHA) to explore: 1) whether there are Modified Mini-Mental State Exam (MMMSE) items that may be eliminated without loss of sensitivity or specificity for identifying subjects at highest likelihood of having dementia; and 2) whether there are additional items from the neuropsychological battery used in the CSHA that may be useful to identify subjects at highest likelihood of having dementia. Learning objectives will include:
- Different goals of measurement and desired psychometric characteristics of cognitive measures
- Importance of a valid screening tool in the first stage of studies that employ a 2-stage sampling design
- The concept of a cognitive domain and how to assign items/tests to appropriate domains
- How to develop and interpret confirmatory factor analysis models designed to assess adequacy of domain assignments
- Basic principles of recursive partitioning and how to interpret output
- Basic principles of receiver operator characteristic curves (ROC curves), how to interpret them, and how to use results in conjunction with item response theory
- Comparison of IRT and testlet response theory; importance of the unidimensionality assumption for IRT.
Approach: The CSHA performed population-based screening with the MMMSE. They employed a 2-stage sampling design, in which a cut-off point was used to determine who was asked to return for further neuropsychological testing. In addition to those who screened positive by virtue of low MMMSE scores, a random sample of subjects who screened negative for dementia were also asked to return for neuropsychological testing. In all at wave 1, some 1600 subjects received neuropsychological testing, all of which has been entered at the item level. Of the 1600 subjects with neuropsychological data from Wave 1, about 800 screened positive on the MMMSE and the other 800 screened negative.
As a first step, we will identify items that appear to be assessing general cognition and/or items from the neuropsychological battery that appear to be assessing domains from the MMMSE. Initial confirmatory factor analysis approaches will be used to identify a sufficiently unidimensional subset of items to proceed with IRT. The IRT methods used to obtain an “item pool” for further analyses will be demonstrated. Several potential techniques will be discussed and demonstrated to select items from the neuropsychological battery that may be especially useful for screening purposes. These include recursive partitioning, use of a testlet approach to fit data, and wedding receiver operator characteristic (ROC) curve technology to address the question of item selection. Furthermore, we will demonstrate the difference in techniques for a single threshold analysis (e.g., demented vs. not) versus a two-threshold analysis (e.g., normal vs. CIND vs. demented), and we will explore whether the characteristics of the scale change if the goal is to distinguish accurately between normal and CIND. We will illustrate how differential item functioning (DIF) can cause problems with a 2-stage sampling technique.