Article Text

Original research
Evaluation of the Montreal Cognitive Assessment as a screening tool for cognitive dysfunction in SLE
  1. Sudha Raghunath1,2,
  2. Yifat Glikmann-Johnston3,
  3. Eric Morand1,2,
  4. Julie C Stout3 and
  5. Alberta Hoi1,2
  1. 1Centre for Inflammatory Disease, School of Clinical Sciences, Monash University, Clayton, Victoria, Australia
  2. 2Department of Rheumatology, Monash Health, Clayton, Victoria, Australia
  3. 3Turner Institute for Brain and Mental Health, School of Psychological Sciences, Monash University, Clayton, Victoria, Australia
  1. Correspondence to Dr Sudha Raghunath; sudha.raghunath{at}monash.edu

Abstract

Objectives Cognitive dysfunction in SLE is common and associated with significant morbidity but is currently underdetected. Early detection requires the use of screening tests, as formal diagnostic cognitive testing is time-consuming. This study aims to evaluate the Montreal Cognitive Assessment (MoCA) as a screening tool for cognitive dysfunction in SLE.

Methods Patients with SLE (n=95) and demographically matched healthy control participants (n=48) underwent cognitive testing using the 1-hour neuropsychiatric test battery recommended by the American College of Rheumatology for use in SLE and the MoCA. We used regression analyses to determine associations between MoCA and cognitive test scores. We assessed several MoCA cut-offs for predicting cognitive impairment in terms of sensitivity, specificity, positive predictive value and negative predictive value. Receiver operating curve analyses were used to determine the diagnostic accuracy of the MoCA cut-off thresholds.

Results We found a significant correlation between MoCA score and 9 of the 10 cognitive endpoints studied (all p<0.001). Receiver operating curve analysis suggested that a MoCA cut-off of <27 had highest diagnostic accuracy across the cognitive impairment definitions (area under the curve 0.76–0.78). Using a screening cut-off of <28, the MoCA had sensitivity of 83%–94% and specificity of 46%–59%, depending on the impairment definition used.

Conclusions The MoCA correlates strongly with cognitive test results in SLE and has sufficient sensitivity for use as a screening tool with a cut-off of <28 as the optimal threshold. This tool can be incorporated into clinical practice for screening for cognitive dysfunction in SLE.

  • systemic lupus erythematosus
  • autoimmune diseases
  • epidemiology

Data availability statement

All data relevant to the study are included in the article or uploaded as supplemental information.

http://creativecommons.org/licenses/by-nc/4.0/

This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/.

Statistics from Altmetric.com

Request Permissions

If you wish to reuse any or all of this article please use the link below which will take you to the Copyright Clearance Center’s RightsLink service. You will be able to get a quick price and instant permission to reuse the content in many different ways.

Key messages

What is already known about this subject?

  • Cognitive dysfunction is common in SLE and has significant morbidity, but is currently underdetected as formal diagnostic testing is time-consuming and requires specialised staff.

  • The Montreal Cognitive Assessment (MoCA) is a freely available, practical and brief cognitive screening test; the utility of the MoCA as a screening test for cognitive dysfunction in SLE has not been comprehensively studied.

What does this study add?

  • Performance on the MoCA correlates strongly with formal cognitive test results and the MoCA has sufficient sensitivity to be used as a screening test for cognitive dysfunction in SLE.

  • The optimal screening cut-off threshold for use of the MoCA in SLE is <28, which is higher than the cut-off of <26 used for mild cognitive impairment and dementia.

How might this impact on clinical practice or future developments?

  • Screening for cognitive dysfunction with the MoCA should be incorporated into routine assessment of patients with SLE.

Introduction

SLE is a chronic multisystem autoimmune disease associated with significant morbidity and reduced life expectancy.1 Cognitive dysfunction is common in SLE and can present insidiously. Many patients with SLE report cognitive dysfunction as one of the most distressing symptoms of their condition,2 3 adversely impacting function and employment.4 5 Although the pathogenesis of cognitive dysfunction in SLE remains poorly understood, early detection and recognition of cognitive changes may help us develop strategies that improve patients’ quality of life.

Formal evaluation by a clinical neuropsychologist including cognitive testing remains the gold standard for the diagnosis of cognitive impairment, but is not practical for use on a routine basis in the SLE clinical care setting. Neuropsychological assessments themselves are time-consuming, and are expensive for routine use, because these services are often not covered by healthcare benefits or insurance. This further adds to the potential utility of a screening tool as a first step to identify patients who may benefit from comprehensive cognitive testing.

The ideal characteristics of a screening test include sufficient sensitivity to detect potential cases and that it should be easily accessible and simple to administer. It is essential that a screening tool is based on objective cognitive tests, as patient-reported symptoms are frequently discordant with objective tests and may be affected by factors such as mood disorders.6 7 Cognitive dysfunction in patients with SLE is known to affect a wide range of cognitive domains8 and hence it is also important that screening covers a broad range of domains.

The Montreal Cognitive Assessment (MoCA) and the Mini Mental State Examination (MMSE) are both cognitive screening tools that meet these characteristics and have undergone preliminary assessment for use in SLE.9 The MoCA is a freely available brief screening tool that was initially designed in 1996 to screen for early cognitive impairment in the setting of dementia.9 Three studies have assessed the MoCA for use in SLE,10–12 and have suggested that it is a more sensitive test than the MMSE.10 11 Of the studies assessing the use of the MoCA in SLE, only one used a broad conventional cognitive test battery to define cognitive dysfunction.10 However, this study did not specify the method, definition or threshold used to define cognitive impairment on the cognitive test battery comparator and did not evaluate a range of MoCA cut-off thresholds. In addition, this study excluded patients with SLE with cerebrovascular disease or mood disorders despite these being common comorbidities in SLE which may contribute to cognitive dysfunction,13 14 making these results less applicable to a typical SLE cohort in clinical practice. The Automated Neuropsychological Assessment Metrics test has also been evaluated in SLE,15 but practical considerations limit its use as a screening tool, including accessibility, cost and testing time.

We aimed to address methodological deficiencies in these previous studies by analysing across different definitions of cognitive dysfunction using the American College of Rheumatology (ACR) recommended conventional cognitive test battery and testing multiple MoCA cut-offs to determine the optimum threshold for clinical application. The first objective was to evaluate the construct validity of the MoCA for screening in SLE by determining if there was an association between MoCA scores and performance on conventional cognitive testing. The second objective was to evaluate the performance of the MoCA as a screening tool in SLE by determining the sensitivity, specificity, positive predictive value, negative predictive value and diagnostic accuracy of various MoCA cut-off thresholds.

Methods

Participants

Study participants for the SLE group (N=95) were recruited consecutively from the Monash Lupus Clinic site of the Australian Lupus Registry and Biobank (ALRB) between October 2018 and February 2020. The ALRB is a national registry of patients with SLE prospectively collecting longitudinal clinical data, blood and tissue samples since 2007.16 All enrolled patients fulfil either the 1997 ACR17 or the 2012 Systemic Lupus International Collaborating Clinics (SLICC) classification criteria.18 For this study, adults over the age of 65 years were excluded to avoid potential comorbid cognitive disorders associated with ageing. Patients with neurological conditions definitively not related to SLE (such as traumatic brain injury) were also excluded. Disease activity and damage were measured with the SLE Disease Activity Index 2000 (SLEDAI-2K) and SLICC-ACR Damage Index (SDI), respectively, as previously described.19

A healthy control (HC) group was recruited as a comparator (N=48). The mean and range of age and premorbid IQ of the HC group were matched to the SLE group. HC participants were excluded if they had a history of autoimmune disease (except stable thyroid disease), any organ failure, central nervous system neurological condition or were on immunosuppressive therapy. HCs were recruited from family and friends of the SLE participants and via advertisement in the local community. All participants in both groups were English speaking and had completed at least part of their secondary schooling in English in order to ensure sufficient English language proficiency for the cognitive assessments. Participants provided informed consent and received no monetary compensation. Patients and the public were not involved in developing the study design. The Standards for Reporting of Diagnostic Accuracy Studies (STARD) guidelines for reporting diagnostic accuracy studies were used to ensure completeness and transparency.20

The MoCA

The MoCA is a cognitive screening tool that takes approximately 10 min to administer, does not require specialised training and is freely available (it can be accessed here: https://www.moca.test.org/the-moca-test/).9 It consists of tasks examining visuospatial/executive function, naming, memory (delayed), attention, language, abstraction and orientation. The MoCA is scored out of 30 with 1 point added as an education adjustment for individuals who have completed less than 12 years of education.9 In screening for mild cognitive impairment or Alzheimer’s disease, the recommended cut-off score is <26.9 In this study, the MoCA was administered prior to the conventional cognitive test battery for all patients.

Conventional cognitive test battery

A single trained assessor (SR) administered the cognitive assessment using the 1-hour conventional neuropsychological test battery recommended by the ACR for use in SLE.21 The ACR battery has been validated in SLE against a more comprehensive 4-hour neuropsychological test battery22 with 90% agreement.14 The cognitive assessment component of this study was conducted under the guidance of a clinical neuropsychologist (YG-J).

Within the 15 subtest scores obtained from ACR test battery, there is some overlap in the domains tested. For example, the battery includes several tests of psychomotor speed and tests that tap overlapping aspects of memory. Therefore, for the purpose of defining cognitive impairment, seven subtest scores with significant magnitude of effect in the SLE group were chosen as outcome measures to represent seven domain groups (see online supplemental file for detailed cognitive test descriptions):

Supplemental material

  • The Rey-Osterrieth Complex Figure Test recall task score was used to assess visual memory.23

  • The California Verbal Learning Test immediate recall (sum of trials 1–5) was used to measure verbal learning and memory.24

  • The Controlled Oral Word Association Test F, A and S trials summation score was used to assess verbal fluency.25

  • The Letter Number Sequencing Test from Wechsler Adult Intelligence Scale (WAIS) fourth edition was used to assess verbal working memory and attention.26

  • The Coding Test from the WAIS fourth edition (previously known as the Digit Symbol Substitution Test) was used to assess processing speed and attention.26

  • The Trail Making Test B time was used to assess complex attention and cognitive sequencing.27

  • The Finger Tap Test using the dominant hand (average number of taps per 25-second trial) was used to measure fine motor speed.28

  • The Test of Premorbid Functioning (TOPF) was used to estimate premorbid IQ.29 Because the pronunciation of known words is relatively resistant to impairment from brain pathology, the TOPF yields an estimate of premorbid verbal ability that is robust in the context of possible cognitive impairment.

Defining cognitive impairment

The percentage of patients with SLE with cognitive impairment was defined using SD from the HC group. The ACR 2007 response criteria for neurocognitive impairment in SLE clinical trials proposed the use of two SD thresholds to determine cognitive dysfunction30: >2 SD below normative data (the bottom 2.5th percentile) is defined as ‘cognitive impairment’, and >1.5 SD below normative data as a lesser level of cognitive dysfunction. Focal impairment was defined as involving a single cognitive domain and multifocal impairment being more than one cognitive domain affected. We included three definitions of cognitive impairment using these thresholds: (1) two cognitive domains with SD >1.5 below the HC group mean, (2) one cognitive domain with SD >2 below the HC group mean and (3) two cognitive domains with SD >2 below the HC group mean. To capture the spectrum of cognitive impairment in SLE, we applied these three thresholds independently to categorising each participant as cognitively impaired or unimpaired. We added a fourth category of cognitive impairment pooling the first three definitions, categorising patients meeting any definition as being cognitively impaired.

Statistical analysis

To determine associations between MoCA scores and performance on conventional cognitive testing, we used multivariate analysis adjusting for age and premorbid IQ. We used linear regression to examine for relationships between MoCA scores and test performance in the seven individual cognitive domains, which were represented as z-scores in comparison with HC group data. We used logistic regression to examine associations between MoCA scores and cognitive impairment categories derived from the four ways of defining cognitive impairment separately. Pearson’s correlation coefficients were used to assess collinearity between covariates with likelihood ratios used to select between collinear covariate pairs. P values of <0.05 were considered significant and p values of <0.005 were considered highly significant.

To assess the performance of the MoCA for the diagnosis of cognitive impairment in SLE, sensitivity, specificity, positive predictive value and negative predictive value were assessed. Finally, to determine the diagnostic accuracy of various MoCA cut-off thresholds, receiver operating curves (ROCs) were plotted and the area under the curve (AUC) calculated. All analyses were performed using STATA software V.15.

Results

Participant characteristics

The SLE and HC groups were well matched demographically with no significant differences between the groups in terms of age, gender, ethnicity, premorbid IQ or education level (table 1). The median age was 45 years (range 22–64), 60% were Caucasian and the rest predominantly Asian, and all had good English proficiency. There were higher rates of unemployment, depression and anxiety in the SLE group (p=0.003–<0.0001, table 1). The SLE group had median disease duration of 15 years (range: 0.2–38.7), median disease activity (SLEDAI-2K) of 3 and median damage index (SLICC-SDI) of 1 (table 2). A history of cerebrovascular disease was present in 13%, seizures in 8% and cranial neuropathy in 5%. The prevalence of cognitive impairment in the SLE group varied from 19% to 52% depending on the definition used (table 1); the median MoCA score was 26 (range: 19–30). As expected, overlap between the definitions was observed (figure 1).

Table 1

Demographic characteristics and cognitive test results of study groups

Table 2

Clinical characteristics of patients with SLE

Figure 1

Venn diagram of number of cognitively impaired patients in SLE group using different cognitive impairment definitions (out of n=95 total in SLE group). Impairment defined by number of cognitive domains either 1.5 or 2 SD below healthy control (HC) group mean to form three definitions above as described in coloured text. N impaired by each definition in brackets. N in overlap between definitions indicated by numbers in black text. All three definitions pooled into fourth cognitive impairment category (n=49 impaired).

Association between MoCA and cognitive test battery results

MoCA scores significantly correlated with six out of the seven cognitive domains examined after adjusting for age (all p<0.001, coefficients 0.12–0.28; table 3). Premorbid IQ was highly collinear with MoCA scores, and MoCA scores were found to be more significantly associated with cognitive impairment on likelihood ratio testing (p<0.0001 vs 0.136), hence premorbid IQ was not included in the multivariate analyses. MoCA scores strongly correlated with cognitive impairment using all four definitions. An increase in MoCA score of 1 point was associated with reduced likelihood of meeting any definition of cognitive impairment by approximately 50% (OR 0.44–0.49) (table 3).

Table 3

Multivariate analysis of the Montreal Cognitive Assessment (MoCA), age and cognitive test results in SLE

We next examined the sensitivity of the MoCA for detecting cognitive impairment in SLE. Low MoCA scores, indicating more severe impairment, had high specificity but low sensitivity (table 4), and had high positive predictive value but only modest negative predictive value (table 5). Correspondingly, higher MoCA cut-offs improved sensitivity. Using a cut-off of <26, which is recommended for the detection of mild cognitive impairment or dementia,9 sensitivity ranged from 44.9% to 72.2% depending on the cognitive impairment definition used; the corresponding specificity was 84.4% to 93.8% (table 4) and had a negative predictive value of 61.4%–92.9% (table 5).

Table 4

Sensitivity and specificity of Montreal Cognitive Assessment (MoCA) for cognitive impairment in SLE

Table 5

Positive and negative predictive value of Montreal Cognitive Assessment (MoCA) for cognitive impairment in SLE

Finally, we used ROCs to assess diagnostic accuracy of different MoCA cut-off scores. Comparing AUC for various MoCA cut-offs against the three cognitive impairment definitions suggested that <27 was the cut-off with the maximum diagnostic accuracy (figure 2). This threshold had the highest AUC for three definitions of cognitive impairment: two domains >1.5 SD below the HC group mean, one cognitive domain >2 SD below and all three definitions pooled (AUC 0.78, 0.76 and 0.75, respectively). For the most severe definition of cognitive impairment, two domains >2 SD below the HC group mean, the AUCs for <26 and <27 were similar (0.783 and 0.776, respectively). A cut-off of <28 had a similar AUC of 0.70–0.74, but superior sensitivity (table 4) and positive predictive value (table 5).

Figure 2

Receiver operating curves (ROCs) for different Montreal Cognitive Assessment (MoCA) cut-offs by cognitive dysfunction (CD). Areas under the curve interpretation: 0.7–0.9=moderate accuracy, 0.5–0.7=low accuracy, ≤0.5=equal to chance. CD defined by number of cognitive domains either 1.5 or 2 SD below healthy control group mean, all three definitions pooled into fourth cognitive impairment category.

Discussion

Cognitive dysfunction is an important cause of functional impairment and disability in patients with SLE. Because formal diagnosis involves detailed neuropsychological testing that requires trained personnel, cognitive dysfunction in SLE is under-recognised. We sought here to assess the validity of a simple cognitive screening test in the identification of patients with SLE with cognitive impairment. We found a strong correlation between MoCA scores and formal cognitive test performance across a wide range of cognitive domains as defined by the ACR-recommended cognitive test battery, consistent with previous studies.10 The cognitive domain of psychomotor speed was the only cognitive endpoint where there was no correlation with the MoCA score, however, this domain was the least affected in the cohort studied. Using a variety of definitions for cognitive dysfunction in order to capture the broad spectrum of involvement, we found that MoCA score had significant associations with all four definitions of cognitive impairment.

The MoCA had good diagnostic accuracy and adequate sensitivity for use as a screening test for cognitive dysfunction in SLE. These findings are consistent with previous preliminary studies in this area.10–12 However, our findings suggest the optimal MoCA cut-off is <28, a higher threshold than previous studies in SLE.

The MoCA has established utility as a sensitive screening tool for detection of mild cognitive impairment in the elderly. For this population, the cut-off of 26 has a sensitivity of 90% and a specificity of 87%.9 In contrast, the sensitivity of the MoCA cut-off of <26 in our SLE cohort ranged from 44.5% to 72.2% depending on the definition of cognitive dysfunction, which is lower than those reported in previous studies in SLE.10 11 We saw a progressive decrease in sensitivity and corresponding increase in specificity as the MoCA threshold decreased, and this was observed across all definitions of cognitive impairment. The cut-off score of <27 was associated with the highest AUC across the four cognitive impairment definitions, in contrast to previous studies using ROC analyses which suggested <26 as having the highest AUC.10 12 Our study is the first to include patients with SLE with previous neuropsychiatric involvement and examine the performance of the MoCA screening tool against a range of cognitive impairment definitions.

There are multiple considerations when determining the optimal MoCA cut-off to use for screening purposes. For the purposes of screening, high sensitivity is favoured over high AUC or diagnostic accuracy, given that the primary aim is to detect all potential cases and follow these up with a more specific diagnostic test, namely conventional cognitive testing. However, higher sensitivity comes at the expense of specificity and therefore more false positives. Although in this context there are no high-risk follow-up tests such as biopsies, patients may be distressed by positive screening test results and the triggering of formal cognitive testing. In addition, access to neuropsychologists for formal testing may be limited in some settings, resulting in delay in moving from screening to formal result, contributing further to patient stress.

Keeping these considerations in mind, our findings suggest that the optimal cut-off MoCA score in screening for cognitive dysfunction in SLE population is <28, which had a sensitivity and specificity of 83%–94% and 45%–59%, respectively, and AUC of 0.70–0.74. For comparison, the most commonly used screening tool in the diagnosis of SLE is the ANA test, that has a similar sensitivity and specificity of 93% and 57%, respectively, and is considered to be an excellent screening tool.31

Screening tools such as MoCA serve in the initial assessment of patients, providing a quick and highly sensitive assessment for identifying potential cognitive dysfunction, but in clinical practice patient symptoms and functional impact are also important considerations in determining whether further cognitive assessment is required. Patient-reported outcome tools to measure cognitive symptoms such as the Cognitive Symptoms Inventory or Perceived Deficits Questionnaire2 should also potentially be incorporated into the preliminary assessment, before recommendation of appropriate follow-up which may include further investigations such as neuroimaging. In addition, MoCA score was highly collinear with premorbid IQ and hence the effects of education level and premorbid IQ must also be considered in its interpretation in individual patients. Therefore, in individual patients with above or below average education levels or premorbid IQ, the change in MoCA score over time and patient-reported functional impact are particularly important considerations.

There are several limitations to the current study. The optimal MoCA cut-off may vary according to age, ethnicity, premorbid IQ and educational level.32 Our sample was highly educated, with 66% having completed some form of tertiary education. This may limit the ability to extrapolate our findings to populations with lower education levels. However, this effect was mitigated by a well-matched control group and the use of standard scoring guidelines for the MoCA, which include a one-additional point adjustment for those who have completed less than 12 years of education.9 In addition, in this multiethnic cohort, some participants were not native English speakers, but cognitive assessments were performed in English hence participants were excluded if they had not completed at least part of their secondary schooling in English. The MoCA has been translated into 36 languages and it is unclear how using multiple language versions of the MoCA in the same cohort may affect its validity as a screening tool. Finally, in this cross-sectional study, we did not explore the utility of repeated MoCA testing for long-term monitoring in SLE. The utility of serial MoCA in SLE for monitoring is not yet determined, even though in other disease states there is a suggestion of a value for serial testing.33

In summary, the MoCA provides a brief, freely available, practical screening tool for cognitive dysfunction, which our findings confirm has adequate sensitivity and specificity for use in SLE with a cut-off of <28. Given that cognitive dysfunction is commonly reported by patients with SLE, screening will improve detection in clinical practice, enabling future research to focus on addressing unmet needs in this domain.

Data availability statement

All data relevant to the study are included in the article or uploaded as supplemental information.

Ethics statements

Patient consent for publication

Ethics approval

The study was approved by the Monash Health Human Research Ethics Committee (reference number: LNR/18/MonH/440).

Acknowledgments

We would like to acknowledge the participants in this study for volunteering their time.

References

Supplementary materials

  • Supplementary Data

    This web only file has been produced by the BMJ Publishing Group from an electronic file supplied by the author(s) and has not been edited for content.

Footnotes

  • Twitter @Sudha_Raghunath

  • Contributors SR performed data acquisition, data analysis and draft manuscript preparation. All authors contributed to study concept and design, data interpretation and manuscript editing. AH served as guarantor, and accepts full responsibility for the work and/or the conduct of the study, had access to the data, and controlled the decision to publish.

  • Funding The authors have not declared a specific grant for this research from any funding agency in the public, commercial or not-for-profit sectors.

  • Competing interests SR reports research grant from Lupus Victoria, SR also reports postgraduate scholarships from the National Health and Medical Research Council (NHMRC), Arthritis Australia and the Australian Rheumatology Association. YG-J is supported by Huntington’s Disease Society of America HD Human Biology Fellowship. EM reports grants from the NHMRC, Australia, during the conduct of the study; EM also reports grants and personal fees from AstraZeneca, BristolMyersSquibb, Eli Lilly, EMD Serono, Janssen, GlaxoSmithKline, as well as personal fees from Biogen, Servier, and Amgen which were outside the submitted work. JCS is a Director of Zindametrix, which provides cognitive assessment advice and assists in the implementation of cognitive testing in commercial and academic clinical trials in Huntington's disease. She also conducts professional consultation in neuropsychology via Stout Neuropsych. AH reports grants from AstraZeneca and EMD Serono outside the submitted work, sponsorship of the Australian Lupus Registry and Biobank (chaired by AH) is received from Janssen, BMS, AstraZeneca, and UCB; and honoraria from Abbvie.

  • Provenance and peer review Not commissioned; externally peer reviewed.

  • Supplemental material This content has been supplied by the author(s). It has not been vetted by BMJ Publishing Group Limited (BMJ) and may not have been peer-reviewed. Any opinions or recommendations discussed are solely those of the author(s) and are not endorsed by BMJ. BMJ disclaims all liability and responsibility arising from any reliance placed on the content. Where the content includes any translated material, BMJ does not warrant the accuracy and reliability of the translations (including but not limited to local regulations, clinical guidelines, terminology, drug names and drug dosages), and is not responsible for any error and/or omissions arising from translation and adaptation or otherwise.