Workgroup 1: Validation of classification
Description: Clinical diagnosis of cognitive impairment is commonly used for staging severity of brain injury/degeneration and predicting future decline. Studies of machine learning and other algorithmic forms of classification typically evaluate classification accuracy using comparisons with “gold standard” clinical diagnosis, but studies examining how algorithmic classifications and clinical diagnosis relate to relevant validation criteria are limited. It is important to directly demonstrate the utility of algorithm classifications, but also of clinical diagnosis. This workgroup will validate clinical diagnosis and various forms of algorithmic classification against disease biomarkers and future progression of cognitive impairment. We will compare associations of longitudinal MRI variables and other biomarkers with different forms of dementia classifications including clinical, algorithmic classification, and machine learning-based predictions.
Dataset: UC Davis ADRC
Workgroup 2: Feature engineering for machine learning classification
Description: What resolution of data elements should be considered best practice for inclusion as features in dementia classification algorithms? Is it better to capitalize on the higher reliability of summary measures (e.g., factor scores, sum scores) or the unique variance provided by individual item-level scores? Can item-level scores be reduced to a smaller set of predictive features without sacrificing classification accuracy or introducing bias? Are there other methods for engineering new features from item-level data? This workgroup will explore differences in coding of variables used in machine learning algorithms for dementia for constructs including demographic, objective cognition, subjective cognition, subjective IADL functioning, informant reports of functioning, and more. Different approaches will be compared based on accuracy and relative bias introduced.
Dataset: UC Davis ADRC
Workgroup 3: Approaches to develop and validate a dementia algorithm when one does not have a reference standard
Description: Algorithmic classification is an efficient alternative to reference-standard clinical diagnosis in large population-based cohort studies, but how can we truly show that we are measuring the construct we are claiming to measure in settings without clinical diagnosis or high-quality ADRD biomarkers? This workgroup will explore (1) approaches to develop algorithmic dementia classification without a reference standard and (2) different forms of validation, including evaluating subsequent longitudinal changes in cognition, loss to follow-up, and death, and considering patterns in dementia and MCI prevalence by age, education, and other demographic characteristics. We will evaluate the degree to which algorithms need to differ within different demographic subgroups. Some forms of validation may leverage unique aspects of study designs (e.g., longitudinal data and calibration samples with more detailed data on cognition and independent functioning).
Datasets: NHATS, HCAPs, VIP
Workgroup 4: Bayesian Approaches to Dementia Classification
Description: This workgroup will examine how Bayesian generative models can classify cognitive impairment and dementia in research studies. Most epidemiologic studies rely on fixed cognitive thresholds to define mild cognitive impairment (MCI) or dementia. These operational rules treat impaired cognitive performance as evidence of underlying brain disease. However, cognitive impairment can arise from many causes, including neurodegeneration, psychiatric illness, sensory impairment, educational differences, and testing conditions. Threshold-based classifications therefore risk misclassification and do not formally account for uncertainty in the disease process. Bayesian generative models offer an alternative approach. Instead of assigning deterministic diagnostic categories, these models represent how latent disease processes, cognitive test performance, demographic factors, measurement error, and alternative causes jointly generate observed data. Using Bayes’ theorem, investigators can compute the posterior probability of disease for each participant given the available evidence. In this workgroup we will (1) review how dementia and MCI are currently operationalized in epidemiologic studies, (2) examine the inferential limitations of threshold-based classification, and (3) develop simple Bayesian models that estimate the probability of dementia given cognitive test performance and other study variables. Participants will compare probabilistic classifications from Bayesian models with conventional rule-based approaches. Analyses will be implemented in R (e.g., brms/Stan) or Mplus Bayesian estimation, depending on participant expertise and interest.
Dataset: HRS/ADAMS
Workgroup 5: Modifications to MCI criteria to improve predictive utility across varying populations
Description: Mild cognitive impairment (MCI) is meant to capture individuals at greatest risk of developing dementia due to Alzheimer’s disease and related dementias (ADRD). However, several studies have reported up to 50% revert back to normal over the course of 5 years, particularly among population-based studies. Different criteria for MCI have been proposed, but it is unclear which MCI criteria may improve its predictive utility in cross-national settings. This workgroup will evaluate different modifications to MCI criteria (i.e., differential cognitive domain weighting, number of domains impaired required, different cognitive and functional impairment thresholds, with or without subjective cognitive complaints, etc.) to minimize reversion to normal and maximize prediction of incident dementia and subsequent cognitive decline.
Dataset: HCAPs