Skip to content

Advertisement

  • Research article
  • Open Access
  • Open Peer Review

This article has Open Peer Review reports available.

How does Open Peer Review work?

A new device-aided cognitive function test, User eXperience-Trail Making Test (UX-TMT), sensitively detects neuropsychological performance in patients with dementia and Parkinson’s disease

BMC Psychiatry201818:220

https://doi.org/10.1186/s12888-018-1795-7

Received: 18 October 2017

Accepted: 18 June 2018

Published: 5 July 2018

Abstract

Background

A newer generation neuropsychological tests can take advantage of touch screen and mobile technology. We have developed a new Android application termed “User eXperience-Trail Making Test (UX-TMT)” for neurocognitive assessment and training. This study investigated the utility, including the reliability and the validity, of the UX-TMT as a screening test for cognitive decline in adults.

Methods

A total of 84 individuals aged 27–86 years were divided into three groups; healthy controls ([HC] n = 29), people with Parkinson’s disease (PD; n = 28), and people with mild cognitive impairment (MCI) and dementia (MCI&D; n = 27). We examined the distributions of the scores and the time required, and the effects of age and group on these distributions. We analyzed internal consistency and convergent validity in all samples and applied receiver operator characteristic (ROC) analysis to determine a cutoff score that could differentiate the MCI & D group from the HC group.

Results

97.6% of the participants completed all of the tasks, and the average total test time required for UX-TMT was 428.8 (± 109.1) s in the HC, 542.0 (± 168.7) s in the PD, and 777.5 (± 256.1) s in the MCI&D groups, respectively. The MCI&D group showed significantly lower UX-TMT scores and longer total time in completing the task than the HC group. In an ROC analysis, a score of 21 showed high sensitivity (.83) and specificity (.92), and the UX-TMT score plus age improved sensitivity to .96. Additionally, the UX-TMT scores showed significant correlation with the Mini-Mental State Examination (Japanese version) scores (r = .77, p = .001), and Cronbach’s alpha (.71–.83) indicated acceptable internal consistency.

Conclusion

The UX-TMT demonstrated high reliability and validity to detect cognitive decline in Japanese adults, highlighting its utility as a screening tool for epidemiological and clinical research.

Keywords

ApplicationCognitive functionDementiaParkinson’s diseaseScreening

Background

The number of individuals over 60 years of age worldwide is expected to increase from 901 million in 2015 to more than 2 billion by 2050 [1]. As populations age, age-related neurodegenerative diseases, such as Alzheimer’s disease (AD) and Parkinson’s disease (PD), are expected to increase as well [2].

AD is the most common neurodegenerative disease and the most prevalent form of dementia. The age-standardized prevalence of dementia for those aged ≥60 years varies only slightly across the globe (5–7%), and the prevalence is reported to double in about every 5 years. It has been estimated that 35.6 million people worldwide lived with dementia in 2010, with this number is expected to almost double every 20 years, to reach 115.4 million by 2050 [3]. Neuropathologically, AD is characterized by aggregation and deposition of beta amyloid (Aβ) peptide in the form of neuritic plaques, and hypophosphorylated tau protein in the form of neurofibrillary tangles. Clinically, it is characterized with a progressive loss of cognitive functions, primarily memory [4].

Mild cognitive impairment (MCI) is generally considered as an intermediate state between normal cognitive aging and dementia. Severe cognitive deterioration is often preceded by a preclinical stage with only subtle cognitive deficits that transit to full-blown presentation over time [5, 6]. The definition of MCI has evolved over the past decades. Originally, MCI was considered to show clear memory impairment and taken as prodromal AD. The definition of MCI does not focus sorely on memory impairment but also on impairment in other cognitive domains [7], to be currently classified into 4 subtypes: single-domain amnestic MCI, single-domain non-amnestic MCI, multiple-domain amnestic MCI, and multi-domain non-amnestic MCI [6]. According to the major population-based studies, the average prevalence of MCI is 18.9% based on the expanded Mayo Clinic criteria [7]. In a recent systematic review, the annual conversion rate of MCI to AD is reported to range from 5 to 17% depending on the study sample [8].

PD is the second common neurodegenerative disease after Alzheimer’s disease. In an epidemiological study, the projected number of individuals with PD over age 50 years in 15 of most populous nations was between 4.1 and 4.6 million in 2005, which is expected to double to 8.7–9.3 million by 2030 [9]. PD has been defined primarily as a movement disorder, with typical symptoms being resting tremor, rigidity, bradykinesia, and postural instability; it is pathologically characterized by degeneration of nigrostriatal dopaminergic neurons and the presence of Lewy bodies (misfolded α-synuclein). Recently, cognitive impairment has been increasingly recognized as one of the important non-motor symptoms in PD [10]. In the International Parkinson and Movement Disorders Society (MDS) task force’s first review, the mean prevalence of PD with MCI (PD-MCI) in non-demented patients is 27% (range: 19–38%), and it is associated with subsequent development of PD with dementia (PDD) [11, 12].

Furthermore, PD patients have a much higher cumulative risk of dementia than the age-referenced general population. The incidence of dementia in PDD is increased by 4 to 6 times, and the point prevalence of dementia in PD is close to 30% [13]. Several long-term longitudinal studies have indicated that the majority of patients with PD will develop dementia if they survive for more than 10 years after the diagnosis [10]. In a recent cohort study, the conversion rate from PD-MCI to dementia was 59% at 1 year, although a total of 28% of patients with baseline PD-MCI had reverted to normal cognition at 5-year follow-up [14]. The most common characteristic of PD-MCI is a more rapid decline in cases with non-amnestic and single-domain impairment. In particular, impairments in executive, attentional, and visuospatial domains are notable [10, 11]; behavioral symptoms, such as affective changes, hallucinations, and apathy are frequent as well [13].

Thus, in both AD and PD, cognitive impairments represent one of the major clinical challenges and strongly affect patients’ functioning, Quality of Life (QoL), health-related costs and caregiver’s burden. Although much effort has been made in the development of medication for dementia, the results have been somewhat futile thus far. Accordingly, the interest has been shifted to preventative measures in recent years. If an intervention could delay both disease onset and progression by a modest 1 year, there would be nearly 9.2 million fewer cases by 2050 [15]. In cases of AD, several risk factors (e.g., hypertension, obesity, diabetes, smoking, physical inactivity, and depression) are modifiable [16, 17]. Further, a systematic review and meta-analysis showed that cognitive training leads to measurable improvements in cognitive performance in PD patients, particularly in working memory, executive functioning, and processing speed, which are typically impaired in this disorder [18, 19].

Recently, a series of tablet- and smartphone-based applications for the assessment of cognitive function are appearing. In most previous studies, the utility of these applications has been tested against conventional methods, with a use of intra- and inter-person consistency [2023]. It has been unclear thought whether these assessments can be completely self-administered by any person [2024]. Some studies have investigated a novel model or isolated components of cognitive processes [25, 26]. Better assessment strategies would be beneficial both to patients and physicians, potentially minimizing sample size needed in randomized controlled trials (RCTs) by capturing subtler changes during asymptomatic disease stages [27].

We have previously developed a new Android application, the “User eXperience-Trail-Making Test (UX-TMT),” as an assessment and training tool for cognitive function [28]. In this study, we further incorporated 3 other tasks and 3 questions to the UX-TMT for dementia screening. User experience (UX) is one of the key concepts of user-centered approach and is critical in user interface (UI) development, as it comprehensively encompasses user needs, values, abilities, and limitations [29]. The main features of the UX-TMT were that (1) it employs a user-friendly device and design, (2) users are able to work on the tasks in a comfortable way, and (3) it allows clinicians and researchers to objectively and quantitatively evaluate cognitive functions.

We hereby report on discriminating performance of the UX-TMT as a screening test for cognitive decline in Japanese adults.

Methods

Participants

People with MCI and dementia (MCI&D), people with PD, and healthy controls (HC) were included in the present study. All participants were recruited through flyers from the National Center of Neurology and Psychiatry (NCNP) in Japan. The inclusion criteria for the MCI&D and PD groups were: a diagnosis of MCI&D or PD by a skilled psychiatrist or neurologist, and the ability to voluntarily understand the aim, risk, and benefit of this study. The exclusion criteria for both groups were: 1) comorbidity with other neurological and/ or psychiatric diseases, 2) noncompliance with the testing, 3) visual decline, hypoacusis, and motor deficits that disturb task performance. The inclusion criteria for the HC group were: over 20 years of age, being healthy, and voluntary participation in this study. The exclusion criteria for the HC group were same with the MCI&D and PD groups. We confirmed their visual, auditory, and motor ability with instructions before implementing the UX-TMT. Neuropsychological tests were individually administered by an experienced clinical psychologist (N.K.).

Measures

The user eXperience-trail making test (the UX-TMT)

The User eXperience-Trail Making Test (UX-TMT) was developed for neurocognitive assessment and training in 2015 [28]. It was inspired by the Advanced Trail Making Test (ATMT), Stroop test, and Reverse Stroop test [25, 30]. These tasks aim to evaluate the function of visuospatial working memory, executive function, and interference inhibition quantitatively. The novel feature of the UX-TMT is a modified trail making test, which includes targets flowing on the tablet screen. These conditions aim to evaluate function of visuospatial cognition and coordinated movement in a quantitative manner. The trail making test (TMT) is among the most commonly-used neuropsychological tests because of its sensitivity to detect structural and functional brain damage. The UX-TMT consists of 3 main components: 1) neurocognitive assessment, 2) cognitive training, and 3) lifelogging. In this paper, we focus on neurocognitive assessment unit, which originally included 6 variations of the TMT. We subsequently added further 4 tasks for comprehensive assessment of cognitive function in adults. The following items make up the revised UX-TMT; these are presented as text on tablet screen and/or by audio.

Item 1, demographic information

The participants enter their age, sex, and handedness. When the participant touches input field, a liquid crystal keyboard appears on the screen. After the participant hits enter key, screen transition occurs to fill information.

Item 2, mood

The participants enter their current mood in a Likert scale (1: extremely bad, 2: bad, 3: slightly bad, 4: neither bad nor good, 5: slightly good, 6: good, 7: extremely good).

Item 3, physical condition

The participants enter their current physical conditions in a Likert scale (1: extremely good, 2: good, 3: neither good nor bad, 4: bad, 5: extremely bad).

Item 4, sleepiness

The participants enter their current state of sleepiness in a Likert scale (1: feel smart and wide awake, 2: awake, 3: neither awake nor sleepy, 4: slightly sleepy, 5: extremely sleepy).

Item 5, Modified Trail making test

The trail making test (TMT) is a well-known for investigating cognitive performance that depends on executive function (EF), specifically, sustained attention and set-shifting. Recently, advanced trail making test (ATMT) has been developed for quantitative evaluation of visuo-spatial working memory [25]. In the ATMT, we use touch sensor screen instead of paper. The Numbers and Japanese characters “Hiragana” are arranged on the screen as touchable buttons. The ATMT consists of two conditions for both TMT-A and TMT-B. The first is “buttons that are fixed in location (Fixed)” and the second is “buttons that are random in location (Randomized)”. In this study, we add the third condition “buttons that are floating in location (Floating)” for both tests.

In this study, 10 numbers (1, 2, 3, 4, 5, 6, 7, 8, 9, and 10) or 5 numbers and 5 Japanese characters Hiragana ([a], [i], [u], [e], and [o]) are presented at random positions on the tablet screen as buttons (Fig. 1a). In the TMT-A, the participants are instructed to touch the numbers on the screen from 1 to 10 in a sequential order as quickly and accurately as possible. In TMT-B, the participants are instructed to touch alternately between the numbers and hiragana characters in a sequential order (1 - [a] then 2 - [i] through 5 - and [o]), as quickly and accurately as possible. In “Fixed” condition, the location of buttons is the same throughout a single trial, where the participants are able to memorize the position of buttons. When they take advantage of their visuospatial working memory, reaction time can be shortened. In “Randomized” condition, the location of all buttons is changed at random when the correct response has once been made; the participants should search for the following target each time. In “Floating” condition, all buttons are floating during the single trial. Here, the participants are required to make use of their eyes and dominant finger, thus aiming to investigate coordinated movement, sustained attention, and set-shifting. The score in this task ranges from 0 to 12.
Figure 1
Fig. 1

Schematic illustration of cognitive tasks on UX-TMT on 10.1-in. android tablet. a TMT-B, “Fixed”, “Randomized”, and “Floating” condition on item 5. 5 digits and 5 Japanese hiragana characters on the screen, b Single digit and a single button on item 6, c The name of color and 5 colored button on item 7, and d Instructing sentence on the top, a set of 6-choice words in the middle, and enter button on the bottom on item 8

Item 6, 1-back task

This follows the immediate recall session of the verbal memory task. Single numbers (randomly chosen from 0 to 9) and a single button are presented via text on the screen one at a time (Fig. 1b). Then, each digit is presented for 1.5 s and changed at a fixed inter-stimulus interval of 0.5 s. The participants are instructed to touch the button when the digit is the same as the previous digit, as quickly and accurately as possible. The target is available in a 20% frequency. The score in this task ranges from 0 to 2.

Item 7, Stroop and reverse-Stroop (R-Stroop) task

This follows a 1-back task. In this task, 1 word that is the name of a color and 5 colored buttons are displayed on the screen simultaneously (Fig. 1c). The participants are instructed to touch the colored button that matches the font color of the central word. Conversely, in the R-Stroop task, the participants are instructed to touch the colored button that matches the meaning of the central word. There are 2 conditions; one is color-meaning congruence, the “neutral” condition, and the other is color-meaning incongruence, the “interference” condition. In both the Stroop and R-Stroop tasks, the interference condition occurred in a 33.3–41.7% frequency, and each word is presented for 1.5 s and is changed with a fixed inter-stimulus interval of 0.5 s. The participants are instructed to perform the task as quickly and accurately as possible. The Stroop and R-Stroop tasks are a well-known for investigating selective attention to specific information and inhibition of prepotent responses during decision-making tasks involving stimuli and responses [30]. The score in this task ranges from 0 to 6.

Item 8, verbal memory task

This task is composed of immediate recall task and delayed recognition task. The former task follows a trail making test. Before the task starts, the participants are instructed to remember 3 words and recall them immediately after they disappear. After the participants enter start button, 3 words are presented on the screen, 1 at a time. If the participants succeed in the immediate recall task, they are asked to recall them later. After the participants finish the R-Stroop task, they are asked to select remembered words from a set of 6 choices on the tablet screen (Fig. 1d). The score in this task ranges from 0 to 6.

In this study, the total UX-TMT score was calculated by summing up the individual score in items 5, 6, 7, and 8, leaving a possible score range from 0 to 26; a higher score indicates a better cognitive performance.

The mini-mental state examination-Japanese (the MMSE-J)

The Mini-Mental State Examination is widely used for screening cognitive impairment and decline in about 10 min [31]. The MMSE Japanese version (MMSE-J) has been validated with a high sensitivity and specificity for dementia [32]. In order to avoid practice effects, it was done at least 2 weeks apart from the last examination. The scores in the MMSE-J and those in the UX-TMT were interpreted as an indicator for discriminating validity.

The Japanese version of the Montreal cognitive assessment (the MoCA-J)

The Montreal Cognitive Assessment (MoCA) is a brief, well-validated cognitive screening tool for detecting MCI with high sensitivity and specificity [33]. The Japanese version of the MoCA-J is a commonly used screening instrument that was designed to address some of the limitations in the MMSE [34]. In this study, 3 tasks from the MoCA-J were adopted for evaluating executive function and visuospatial abilities (Trail Making B task, a three-dimensional cube copying, and a crock-drawing task). The performance between Trail Making B task in the MoCA-J and the UX-TMT was compared.

The 12 item short-form healthy survey (SF-12)

The 12-item short-form health survey (SF-12) is a self-administered questionnaire, commonly utilized to evaluate health-related QOL. They evaluate 2 summary components (physical and mental health) and 8 profiles (physical functioning, role physical, bodily pain, general health, vitality, social functioning, role emotional, and mental health) [35]. In this paper, we focused on the score of vitality in the SF-12 as a comparator to the items 3 and 4 in the UX-TMT.

The Japanese version of the 20-item positive and negative affect schedule (PANAS20-J)

The Positive and Negative Affect Schedule (PANAS) is a self-report form and consists of 10 items each for positive affect (strong, inspired, active, enthusiastic, interested, excited, proud, alert, determined, and attentive) and negative affect (afraid, scared, upset, ashamed, guilty, nervous, distressed, irritable, jitter, and hostile) [36]. The Japanese version (PANAS20-J) has been validated and was found to be reliable [37]. We investigated differences across the groups as a reference to the items 2 in the UX-TMT.

A trained clinical psychologist administered the MMSE-J and MoCA-J. The SF-12 and PANAS20-J were self-rated. The UX-TMT is a semi self-administered test because its tutorials are displayed on the screen with the text as an instruction by the examiner. These instruments were administrated in a systematically counterbalanced manner to cancel order effect.

Statistical analysis

We used χ2 tests and two sample t-tests (Welch) to compare demographic and neuropsychological data among the groups. To confirm the effect of age and group on cognitive and other neuropsychological indices, we performed ANCOVA with age as a covariate. Furthermore, we performed receiver operating characteristic (ROC) curve analysis on the MMSE-J scores and UX-TMT scores between the MCI&D and HC groups, to investigate discriminating ability and to determine the cutoff score that is most appropriate. We also performed Spearman’s correlation analysis for the UX-TMT and performance in other neuropsychological tests in order to assess external validity. Finally, Cronbach’s alpha was obtained as an index for internal consistency of the scale.

Ethical considerations

Participants received verbal and written descriptions of the study in detail and provided their written informed consent. The study protocol was approved by the NCNP Institutional Review Board.

Results

In total, 84 participants were assigned either to the MCI&D (n = 27), PD (n = 28), or HC (n = 29) groups, respectively. Three participants in the MCI&D group were excluded from analysis, because the diagnosis was altered in one patient and two could not complete the UX-TMT. Demographic and neuropsychological characteristics of the participants are summarized in Table 1. There were no significant differences in sex and handedness among the 3 groups. The MCI&D group had significantly fewer years of education than the HC group. The HC group was significantly younger than the MCI&D and PD groups. Within the HC group, the lowest score in the MMSE-J was 24.
Table 1

Demographic characteristics of participants

 

HC

MCI&D

PD

Mean (SD)

Mean (SD)

Mean (SD)

Welch’s t-test

p-value

Age

55.8 (13.7)

79.4 (7.6)

68.6 (6.7)

< 0.001

Sex (Male:Female)

11:18

13:11

13:15

0.516

Handedness (Left:Right)

1:28

1:23

2:16

0.532

Education (y)

15.5 (3.8)

13.8 (2.3)

14.4 (2.5)

0.209

Mood (item 2)

4.9 (1.0)

4.7 (1.5)

4.9 (1.0)

0.883

Physical Condition (item 3)

2.3 (0.8)

3.0 (1.1)

2.8 (1.0)

0.026

Sleepiness (item 4)

2.5 (1.3)

2.5 (1.1)

3.4 (1.0)

0.004

PANAS20-J_P

32 (7.7)

29.4 (6.3)

29.29 (6.5)

0.155

PANAS20-J_N

19.5 (6.5)

22.7 (8.0)

24.8 (7.3)

0.005

SF12_Vitality

54.1 (11.1)

51.0 (11.3)

48.2 (11.8)

0.056

MMSE-J: Mini-Mental State Examination Japanese, MoCA-J: Japanese version of the Montreal Cognitive Assessment, EF: executive function, TMT: Trail Making Test, PANAS20-J_P: The Japanese version of 20-item Positive and Negative Affect Schedule positive affect score, PANAS20-J_N: The Japanese version of 20-item Positive and Negative Affect Schedule negative affect score, SF12_Vitality: SF-12 healthy survey, score on the subscale of vitality

Reliability

Cronbach’s alpha for the UX-TMT fell into a range of 0.71–0.83 in all 3 groups, indicating good internal consistency across the sample.

Validity

Discriminative validity

Figure 2 shows scatter plot of the UX-TMT scores (A), the total time required (B) and the MMSE-J score distributions among the 3 groups. Because of the significant differences in age between the 3 groups, we performed ANCOVA for the cognitive tests and the results are shown in Table 2. As expected, people in the MCI&D group performed worst in terms of cognitive assessments. Cognitive performances in the PD group fell within those in the MCI&D and HC groups, without statistical differences from the latter group.
Figure 2
Fig. 2

Distribution of UX-TMT score, total test time and MMSE-J score. a Distribution of UX-TMT score against MMSE-J score, and b distribution of total test time for UX-TMT and MMSE-J score. Triangle: Healthy Control (HC), cross: Parkinson’s disease (PD), circle: Mild Cognitive Impairment & Dementia (MCI&D), (n = 81)

Table 2

Cognitive outcomes

 

HC

MCI&D

PD

Mean (SD) [range]

Mean (SD) [range]

ANCOVA age-adjusted p-value

Mean (SD) [range]

ANCOVA age-adjusted p-value

UX-TMT score

23.2 (2.0)

17.6 (3.4)

0.003

21.0 (2.9)

0.260

UX-TMT_TMTB

1.8 (0.4)

1.5 (0.5)

0.217

1.8 (0.4)

0.458

UX-TMT_TRT (sec)

98.0 (38.2)

251.7 (112.2)

0.010

166.4 (106.5)

0.192

UX-TMT_TTT (sec)

428.8 [255.9–678.4]

777.5 [379.3–1658.5]

0.004

542.0 [248.0–1173.3]

0.520

UX-TMT_B-A (sec)

1.0 (1.3)

3.6 (3.2)

0.071

2.1(3.1)

0.365

MMSE-J

29.2 (1.5)

22.5 (3.9)

< 0.001

28.3 (2.3)

0.721

MoCA-J_EF

4.7 (0.5)

3.7 (1.3)

0.008

4.7 (0.6)

0.835

MoCA-J_TMT

1.0 (0.2)

0.5 (0.5)

0.004

0.9 (0.3)

0.881

Mood: The score of current mood on the UX-TMT, Physical Condition: The score of current physical condition on the UX-TMT, Sleepiness: The score of current state of sleepiness on the UX-TMT, UX-TMT score: Total score of the User experience-Trail Making Test, TMTB: The score of Trail Making Test-part B on UX-TMT, TRT: total response time, TTT: total test time, TMT_B-A: the difference of mean reaction time between TMT-part A and -part B on the UX-TMT

To verify discriminating ability of the UX-TMT, we performed an ROC analysis on the MMSE-J and UX-TMT scores in the MCI&D and HC groups (n = 53). Figure 3 indicates that, at a score of 21, the area under the curve was 0.93, and the sensitivity and specificity were 0.82 and 0.91, respectively. A total of 48 subjects, 4 in MCI&D group, 18 in PD group, and 26 in HC group, respectively, had a score of more than 21 in the UX-TMT. When the patients’ age was added as an explanatory variable on the UX-TMT score, the area under the curve improved to 0.97, and the sensitivity and specificity were 0.97 and 0.92, respectively. In reference, the MMSE-J score of 25 demonstrated the area under the curve of 0.902, with the sensitivity and specificity being 0.97 and 0.83, respectively. When the patients’ age was added as an explanatory variable on the MMSE-J score, the area under the curve, sensitivity and specificity were 0.98, 0.93 and 0.96, respectively.
Figure 3
Fig. 3

Receiver Operating Characteristic curve with UX-TMT or MMSE-J scores and age for differentiation. To differentiate HC from MCI&D groups (n = 53), a score of 21 in UX-TMT showed high sensitivity (.83) and specificity (.92), and UX-TMT score plus age improved sensitivity to .96. Long and short dashed line: MMSE-J score, long dashed line: MMSE-J score plus age, dotted line: UX-TMT score, solid line: UX-TMT score plus age

Convergent validity

The results of correlational analysis among the UX-TMT indices, age, and other neuropsychological tests are summarized in Table 3. The UX-TMT scores had a strong correlation with the MMSE-J scores (r = 0.77, p < .001). Furthermore, the Trail Making Test part B (TMT-B) score in the UX-TMT showed a moderate correlation with the TMT-B score in the MoCA-J (r = 0.56, p < .001) (Table 3).
Table 3

Correlation coefficients among variables

 

UX-TMT score

UX-TMT_ (table-based) TMT-B

UXTMT_Total Test Time: TTT

Age

Age

−0.68

−0.33

0.62

MMSE-J

0.77

0.49

−0.71

−0.62

MoCA-J_EF

0.51

0.52

−0.46

− 0.34

MoCA-J_TMT

0.64

0.56

−0.57

−0.40

MMSE-J: Mini-Mental State Examination Japanese, MoCA-J EF: Executive function (EF) aspects score on the Japanese version of the Montreal Cognitive Assessment, MoCA-J_TMT: Trail Making Test score on the Japanese version of the Montreal Cognitive Assessment, UX-TMT score: scores on the User experience-Trail Making Test, UX-TMT_(tablet-based) TMT-B: score on mode 2 of item 5 on the UX-TMT (TMT task-B)

Discussion

Recent studies have suggested that preventative intervention should start at the earliest opportunity. However, these subclinical individuals often pass currently available screening tests, such as the Mini-Mental State Examination (MMSE), and the Montreal Cognitive Assessment (MoCA), underscoring a need to develop new measures with high sensitivity and specificity to accurately detect preclinical status. In recent years, there has been an interest in identifying subjective cognitive decline (SCD) as an indication to start preventative interventions. SCD is a situation where cognitive impairments are identified by the patients, family members, or healthcare personnel, even though the patient’s cognitive test performance is in the “normal” range [8]. In the general population, SCD is associated with an increased risk of future cognitive decline, that is, progression to MCI or dementia including AD [10]. However, there have been few validated and reliable methods for capturing SCD. Therefore, a user-friendly neurocognitive and behavioral assessment tool is crucial that can be applied to high-frequency in-home monitoring [20].

To this end, we revised the UX-TMT for the purpose of screening cognitive impairment in adults in a cross-diagnostic manner. This report provides some evidence on its utility, validity, and reliability. The time required was short enough to be user-friendly; the average total test time was 428 to 778 s across the groups, with modest standard deviations. Some major screening tests for cognitive impairment, e.g., the MMSE-J, the MoCA-J, and Hasegawa’s Dementia Scale-Revised require from 6 to 10 min to complete [38]. One of the main features of the UX-TMT is that we can record detailed time required for each of the procedure.

In this study, this semi self-administrated instrument indicated good acceptability/tolerability in that 97.6% of the participants completed all of the tasks. In the SMART study, 65% of the cognitively intact participants and merely 29% of the MCI participants were able to complete a computer-based assessment for cognitive function at home [39]. In the ADNI study, Maruff et al. compared between in-clinic and unsupervised assessments for completion rates, and examined age and gender effects, effect of diagnoses as well as stability over time. They reported that 98% of a total 100 participants [cognitive normal = 51, mild cognitive impairment (MCI) = 41, Alzheimer’s disease dementia (AD) = 6] completed baseline in-clinic assessment. The completion rate of second unsupervised assessment was down to 78%, with 92% requiring one attempt [40]. These results call for a user-friendly assessment scales that are cross-diagnostic and can be used in unsupervised settings for the elderly people whose tolerability to tests may well be compromised.

The UX-TMT scores and the total time required for the UX-TMT were significantly different between the MCI&D and HC groups. The ROC analysis (n = 53) indicated that the UX-TMT showed a high sensitivity (0.83) and specificity (0.92) at a cutoff score of 21. When age was added as an explanatory variable, the predictive performance improved. The MMSE-J demonstrated almost the same sensitivity (0.97) and specificity (0.83) between the HC and MCI&D groups. It is known that the MMSE scores are affected by age not only in HCs, but also in dementia and PD patients [4144], and we also added age as an explanatory variable in subsequent analyses; the performance was similar to the UX-TMT. Pedraza et al. reported that age- and education-adjusted MMSE score of 23/24 provided excellent classification accuracy for dementia (0.96), with a high sensitivity (0.92) and specificity (0.98), together with the accuracy (0.96) of unadjusted cutoff score of 22/23 in African American older adults [41]. Onoda et al. has reported that the scores of the cognitive assessment for dementia, iPad version: CADi2, had a high sensitivity (0.85) and specificity (0.81) in participants with moderate AD and age-matched healthy controls [27]. Thus, the UX-TMT was found to be at least equivalent to, or better than, the currently available alternatives.

The PD patients tended to report lower mood and functions in conventional self-rating questionnaires, corroborating the previous study [13]. It is well known that the sleepiness is deeply associated with the cognitive function. Some studies suggest that an electronic diaries or single-item measures are useful for evaluating subjective and objective function [45, 46]. The results suggest that the items in the UX-TMT (i.e., mood, physical condition, and sleepiness) are expected to aid in evaluation of subjective cognitive issues in PD patients briefly. In this study, the UX-TMT did not statistically differentiate PD patients from controls; this may have stemmed from a selected (cognitively preserved) population, limited sample size or a possibility that our scoring method is not sensitive enough.

The UX-TMT showed significant correlations with the MMSE-J (r = .77) and the trail making test part B of the MoCA-J (r = .56). Fellows et al. reported that the digital trail making test (dTMT) showed significant correlation with paper version of trail making test part A (rs = .541) and part B (rs = .799), respectively, in 68 participants comprised of the elderly controls and patients including PD and MCI [26]. They administrated the test by using a stylus instead of pencil and used 26 encircled numbers and letters in paper version and 20 or 19 encircled numbers and letters in tablet version TMT, respectively. In our study, we used 10 encircled numbers and Japanese characters Hiragana. Further research may be needed to decide the optimal methodology to deliver TMT. Cronbach’s alpha (0.71–0.83) showed acceptable consistency for the UX-TMT.

The applications of cognitive assessments can be combined with other technologies, such as smartphone and video conferencing, and the feasibility of remote assessment has been verified [24]. Recently, the next generation applications take advantage of touch sensors and mobile technology [25, 30]. They have been applied to exploring behavioral indices and quantitative measurements of perturbed cognitive processes [25, 26, 30, 47]. Other possibility includes real-time and remote behavioral monitoring of cognitive and motor function in the elderly [26, 47]. Larger studies using age- and education-matched sample are required to further test the usefulness of the UX-TMT in senior populations. Possible diurnal variation may be addressed with the event- or time-driven methods. Test-retest reliability of the UX-TMT should be addressed in the subsequent studies. Nonetheless, it is possible that this instrument may serve to patient-centered design for e-healthcare in the future.

Conclusion

In this novel revision of the UX-TMT, 3 tasks were added to enable more comprehensive assessment for cognitive function. We found that the UX-TMT had high feasibility, sensitivity, and specificity for discriminating between people with MCI and dementia from controls, suggesting adequate construct validity. An optimal cutoff score of 21 was identified to be useful in clinical settings. The UX-TMT scores were significantly correlated with those in the MMSE-J, demonstrating its concurrent validity. The UX-TMT had moderate Cronbach’s alpha values, indicating good internal consistency. The UX-TMT is now only available in Japanese but may be of use as a mass-screening and to monitor disease progression of dementing disorders.

Abbreviations

AD: 

Alzheimer’s Disease

ANCOVA: 

Analysis of covariance

ATMT: 

Advanced Trail Making Test

dTMT: 

digital Trail Making Test

EF: 

Executive Function

HC: 

Healthy Controls

MCI: 

Mild Cognitive Impairment

MCI&D: 

Mild Cognitive Impairment & Dementia

MDS: 

the international parkinson and Movement Disorder Society

MMSE-J: 

Mini-Mental State Examination Japanese version

MoCA-J: 

Japanese version of the Montreal Cognitive Assessment

NCNP: 

National Center of Neurology and Psychiatry

PANAS20-J: 

The Japanese version of 20-item Positive and Negative Affect Schedule

PD: 

Parkinson’s Disease

PDD: 

Parkinson’s Disease with Dementia

QoL: 

Quality of Life

RCTs: 

Randomized Controlled Trials

ROC: 

Reciever Operating Characteristic

R-Stroop task: 

Reverse-Stroop task

SCD: 

Subjective Cognitive Decline

SF-12: 

The 12 item short-form Healthy Survey

TMT: 

Trail Making Test

TRT: 

Total Reaction Time

TTT: 

Total Test Time

UI: 

User Interface

UX: 

User eXperience

UX-TMT: 

User experience?Trail-Making Test

WM: 

Working Memory

Declarations

Acknowledgements

This study was supported by research grants from the National Center of Cognitive Behavior Therapy and Research, National Center of Neurology and Psychiatry (to Dr. Mizusawa). We would like to thank all participants. We would also like to thank Ryosuke Atsumi, Rihito Matsuoka, Drs. Yuji Takahashi, Tomoko Okamoto, Tadashi Tsukamoto, Toshiyuki Yamamoto, Takashi Sakamoto, Manabu Araki, Youwei Lin, Yohei Mukai, Tomoya Kawazoe, Taiji Mukai, Masayuki Miyazaki, Asami Tasaki, Takashi Odo, Takuma Inagawa, Seido Shimizu, and Ayumi Tsuru for data collection. The authors thank Dr. Akihiko Nunomura for the study oversight. The authors have no conflicts of interest to declare in relation with this article. The final version of the manuscript was made possible with significant intellectual input from Professor Takefumi Suzuki, M.D., Ph.D.

Funding

This study was supported by research grants from the National Center of Cognitive Behavior Therapy and Research, National Center of Neurology and Psychiatry (to Dr. Mizusawa and Dr. Horikoshi).

Availability of data and materials

The datasets generated and/or analyzed during the current study are not publicly available due to a concern on participant privacy, but can be considered for availability by the corresponding author upon reasonable request.

Authors’ contributions

All authors initiated this study. NK was responsible for conception and design of this study, developing the application, data collection and analysis, and writing the draft of the manuscript. NK, YY, YS, MM, YT, IS and MH made the protocol. NK, YS, and YY collected data. NK, KM, and YT performed statistical analysis. NK and SY have developed the application. MM and MH oversaw the study. All authors read and approved to submit the final version of the manuscript.

Ethics approval and consent to participate

This study was approved by the Institutional Review Committee of the National Center of Neurology and Psychiatry (approval number A2016–062). All Participants received verbal and written descriptions of the study and provided their written informed consent.

Consent for publication

Not applicable.

Competing interests

The authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Open AccessThis article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Authors’ Affiliations

(1)
Translational Medical Center, National Center of Neurology and Psychiatry, Tokyo, Japan
(2)
National Center for Cognitive Behavior Therapy and Research, National Center of Neurology and Psychiatry, Tokyo, Japan
(3)
Interdisciplinary Graduate School of Medicine and Engineering, University of Yamanashi, Yamanashi, Japan
(4)
Department of Psychiatry, National Center Hospital, National Center of Neurology and Psychiatry, Tokyo, Japan
(5)
Department of Neurology, National Center Hospital, National Center of Neurology and Psychiatry, Tokyo, Japan
(6)
Faculty of Medicine, University of Tsukuba, Ibaraki, Japan
(7)
School of Medicine, Fukushima Medical University, Fukushima, Japan
(8)
National Institute of Technology, Tokyo College, Tokyo, Japan

References

  1. United Nations: World population aging 2015. ISBN: 978-92-1-057854-7.Google Scholar
  2. Johnson IP. Age-related neurodegenerative disease research needs aging models. Front Aging Neurosci. 2015;7:168. https://doi.org/10.3389/fnagi.2015.00168.View ArticlePubMedPubMed CentralGoogle Scholar
  3. Prince M, Bryce R, Albanese E, Wimo A, Ribeiro W, Ferri CP. The global prevalence of dementia: a systematic review and meta-analysis. Alzheimers Dement. 2013;9:63–75.View ArticlePubMedGoogle Scholar
  4. Ferencz B, Gerristen L. Genetics and underlying pathology of dementia. Neuropsychol Rev. 2015;25:113–24.View ArticlePubMedGoogle Scholar
  5. Adams HHH, Bruijn RFAG, Hofman A, Uitterlinden AG, Duijn CM, Vernooij MW, Koudstaal PJ, Ikram MA. Genetic risk of neurodegenerative diseases is associated with mild cognitive impairment and conversion to dementia. Alzheimers Dement. 2015;11:1277–85.View ArticlePubMedGoogle Scholar
  6. Aerts L, Heffernan M, Kochan NA, Crawford JD, Draper B, Trollor JN, Sachdev PS, Brodaty H. Effects of MCI subtype and reversion on progression to dementia in a community sample. Neurology. 2017;88:1–8.View ArticleGoogle Scholar
  7. Petersen RS, Caraccio B, Brayne C, Gauthier S, Jelic V, Fratiglioni L. Mild cognitive impairment: a concept in evolution. J Intern Med. 2014;275(3):214–28.View ArticlePubMedPubMed CentralGoogle Scholar
  8. Cheng YW, Chen TF, Chiu MJ. From mild cognitive impairment to subjective cognitive decline: conceptual and methodological evolution. Neuropsychiatr Dis Treat. 2017;13:491–8.View ArticlePubMedPubMed CentralGoogle Scholar
  9. Dorsey ER, Constantinescu R, Thompson JP, Biglan KM, Holloway RG, Kieburtz K, Marshall FJ, Ravina BM, Schifitto G, Siderowf A, Tanner CM. Projected number of people with parkinson disease in the most populous nations, 2005 through 2030. Neurology. 2007;68:384–6.View ArticlePubMedGoogle Scholar
  10. Aarsland D, Creese B, Politis M, Chaudhuri KR, Ffytche DH, Weintraub D, et al. Cognitive decline in Parkinson disease. Nat Rev Neurol. 2017;13(4):217–31.View ArticlePubMedPubMed CentralGoogle Scholar
  11. Litvan I, Goldman JG, Troster AI, Schmand BA, Weintraub D, Petersen RC, Mollenhauer B, Aarsland D, Kulisevsky J, Rodriguez-Oroz MC, Burn DJ, Barker RA, Murat E. Diagnostic criteria for mild cognitive impairment in Parkinson’s disease: movement disorder society task force guidelines. Mov Disord. 2012;27(3):349–56.View ArticlePubMedPubMed CentralGoogle Scholar
  12. Hoogland J, Boel JA, de Bie RMA, Geskus RB, Schmand BA, Dalrymple-Alford JC, Marras C, Adler CH, Goldman JG, Troster AI, Burn DJ, Litvan I, Geurtsen GJ. Mild Cognitive Impairment as a risk factor for parkinson’s disease dementia. 2017;32(7):1056–1065.Google Scholar
  13. Emre M, Arsland D, Brown R, Burn DJ, Duyckaerts C, Mizuno Y, Broe GA, Cummings J, Dickson DW, Gauthier S, Goldman J, Goetz C, Korczyn A, Lass A, Levy R, Litvan I, McKeith I, Olanow W, Poewe W, Quinn N, Sampaio C, Tolosa E, Dubois B. Clinical diagnostic criteria for dementia associated with Parkinson’s disease. Mov Disord. 2007;22(12):1689–707.View ArticlePubMedGoogle Scholar
  14. Pedersen KF, Larsen JP, Tysnes OB, Alves G. Natural course of mild cognitive impairment in Parkinson disease. Neurology. 2017;88:767–74.View ArticlePubMedGoogle Scholar
  15. Brookmeyer R, Johnson E, Graham KZ, Arrighi HM. Forecasting the global burden of Alzheimer’s disease. Alzheimers Dement. 2007;3:186–91.View ArticlePubMedGoogle Scholar
  16. Baumgart M, Snyder HM, Carrillo MC, Sam F, Kim H, Johns H. Summary of the evidence on risk factors for cognitive decline and dementia: a population-based perspective. Alzheimers Dement. 2015;11(6):718–26.View ArticlePubMedGoogle Scholar
  17. Ngandu T, Lehtisalo J, Solomon A, Levalahti E, Ahtiluoto S, Antikainen R, Backman L, Hanninen T, Jula A, Laatikainen T, Lindstorm J, Mangialasche F, Paajanen T, Pajala S, Peltonen M, Rauramaa R, Stigsdotter-Neely A, Stranberg T, Tuomilehto J, Soininen H, Kivipelto M. A 2 year multidomain intervention of diet, exercise, cognitive training, and vascular risk monitoring versus control to prevent cognitive decline in at-risk elderly people (FINGER): a randomised controlled trial. Lancet. 2015;385(9984):2255–63.View ArticlePubMedGoogle Scholar
  18. Svenningsson P, Westman E, Bllard C, Aarsland D. Cognitive impairment in patients with Parkinson's disease: diagnosis, biomarkers, and treatment. Lancet Neurol. 2012;11:697–707.View ArticlePubMedGoogle Scholar
  19. Leung IHK, Walton CC, Lewis SJG, Valenzuela M, Lampit A. Cognitive training in Parkinson disease: a systematic review and meta-analysis. Neurology. 2015;85(21):1843–51.View ArticlePubMedPubMed CentralGoogle Scholar
  20. Honda M, Hashimoto K, Miyamura K, Goto H, Abo M. Validity and reliability of a computerized cognitive assessment tool ‘higher brain functional balancer’ for healthy elderly people. Jpn. J Cogn Neurosci. 2010;12(3–4):191–7.Google Scholar
  21. Onoda K, Hamano T, Nabika Y, Aoyama A, Takayoshi H, Nakagawa T, Ishihara M, Mitaki S, Yamaguchi T, Oguro H, Shiwaku K, Yamaguchi S. Validation of a new mass screening tool for cognitive impairment: cognitive assessment for dementia, iPad version. Clin Interv Aging. 2013;8:353–60.View ArticlePubMedPubMed CentralGoogle Scholar
  22. Brouillette RM, Foil H, Fontenot S, Correro A, Allen R, Martin KC, Bruce-Keller AJ, Keller JN. Feasibility, reliability, and validity of a smartphone based application for the assessment of cognitive function in elderly. PLoS One. 2013;8(6):e65925. https://doi.org/10.1371/journal.pone.0065925.View ArticlePubMedPubMed CentralGoogle Scholar
  23. Onoda K, Yamaguchi S. Revision of the cognitive assessment for dementia, iPad version (CADi2). PLoS One. 2014;9(10):e109931. https://doi.org/10.1371/journal.pone.0109931.View ArticlePubMedPubMed CentralGoogle Scholar
  24. Park HY, Jeon SS, Lee JY, Cho AR, Park JH. Korean version of the mini-mental state examination using smartphone: a validation study. Telemed JE Health. 2017;23(10):815–21. https://doi.org/10.1089/tmj.2016.0281.View ArticleGoogle Scholar
  25. Kokubo N, Inagaki M, Gunji A, Kobayashi T, Ohta H, Kajimoto O, Kaga M. Developmental change of visuo-spatial working memory in children: quantitative evaluation through an advanced trail making test. Brain and Development. 2012;34(10):799–805. https://doi.org/10.1016/j.braindev.2012.02.001.View ArticlePubMedGoogle Scholar
  26. Fellows RP, Dahmen J, Cook D, Edgecombe MS. Multicomponent analysis of a digital trail making test. Clin Neuropsychol. 2017;31(1):154–67. https://doi.org/10.1080/13854046.2016.1238510.View ArticlePubMedGoogle Scholar
  27. Dodge HH, Zhu J, Mattek NC, Austin D, Kornfeld J, Kaye AJ. Use of high-frequency in home-monitoring data may reduce sample size needed in clinical trials. PLoS One. 2015;10(9):e0138095. https://doi.org/10.1371/journal.pone.0138095.View ArticlePubMedPubMed CentralGoogle Scholar
  28. Kokubo N, Atsumi R, Kawakubo R, Goto K, Shibata N, Taira M, Morohoshi S, Yoshimoto S Asano K, Aoki H. Development of android application “user eXperience-trail making test: UX-TMT” for the neurocognitive assessment and training through medicine engineering collaboration. In: The 1st annual meeting for cognitive enhancement in psychiatric disorders; 2015. in Japanese.Google Scholar
  29. Melnick ER, Hess EP, Guo G, Breslin M, Lopez K, Pavlo AJ, Abujarad F, Powsner SM, Post LA. Patient-centered decision support: formative usability evaluation of integrated clinical decision support with a patient decision aid for minor head injury in the emergency department. J Med Internet Res. 2017;19(5):e174.View ArticlePubMedPubMed CentralGoogle Scholar
  30. Yasumura A, Kokubo N, Yamamoto H, Yasumura Y, Nakagawa E, Kaga M, Hiraki K, Inagaki M. Neurobehavioral and hemodynamic evaluation of Stroop and reverse Stroop interference in children with attention-deficit/hyperactivity disorder. Brain and Development. 2014;36(2):97–106. https://doi.org/10.1016/j.braindev.2013.01.005.View ArticlePubMedGoogle Scholar
  31. Baker E, Iqbal E, Johnston C, Broadbent M, Shetty H, Stewart R, Howard R, Newhouse S, Khondoker M, Dobson RJ. Trajectories of dementia-related cognitive decline in a large mental health records derived patient cohort. PLoS One. 2017;12(6):e0178562. https://doi.org/10.1371/journal.pone.0178562.View ArticlePubMedPubMed CentralGoogle Scholar
  32. Sugishita M, Hemmi I, Takeuchi T. Reexamination of the validity and reliability of the Japanese version of the mini-mental state examination (MMSE-J). Japanese Journal of Cognitive Neuroscience. 2016;18:168–83. (in Japanese)Google Scholar
  33. Nasreddine ZS, Phillips NA, Bedirian V, Charbonneau S, Whitehead V, Collin I, Cummings JL, Chertkow H. The Montreal cognitive assessment, MoCA: a brief screening tool for mild cognitive impairment. J Am Geriatr Soc. 2005;53:695–9.View ArticlePubMedGoogle Scholar
  34. Suzuki H, Fujiwara Y. Development of the Japanese version of Montreal cognitive assessment (MoCA) and the utility. Japanese Journal of Geriatric Psychiatry. 2010;21(2):198–202. (in Japanese)Google Scholar
  35. Ware J Jr, Kosinski M, Keller SD. A 12-item short form health survey: construction of scales and preliminary tests of reliability and validity. Med Care. 1996;34(3):220–33.View ArticlePubMedGoogle Scholar
  36. Watson D, Clark LA, Tellegen A. Development and validation of brief measures of positive and negative affect: the PANAS scales. J Pers Soc Psychol. 1988;54:1063–70.View ArticlePubMedGoogle Scholar
  37. Kawahito J, Ohtsuka Y, Kaida K, Nakata A. Reliability and validity of the Japanese version of 20-item positive and negative affect schedule. Hiroshima psychological research. 2011;11:225–40. (in Japanese)Google Scholar
  38. The Japan Geriatric Society: https://www.jpn-geriat-soc.or.jp/tool/tool_02.html. Accessed 6 Sept 2017.
  39. Seelye A, Mattek N, Reynolds C, Sharma N, Wild K, Kaye J. The Survey for Memory, Attention, and Reaction Time (SMART): A Brief Online Personal Computing Based Cognitive Assessment for Healthy Aging and Mild Cognitive Impairment. Abstract ID: a18921. London: Alzheimer’s Association International Conference; 2017. https://www.alz.org/aaic/about/london.asp Google Scholar
  40. Maruff P, Petersen RC, Aisen PS, Weiner M, Albala B. Acceptability, Usability, Validity and Sensitivity of Cognition Assessed in Unsupervised Settings. Abstract ID: a16958. London: Alzheimer’s Association International Conference; 2017. https://www.alz.org/aaic/about/london.asp Google Scholar
  41. Pedraza O, Clark JH, O’Bryant SE, Smith GE, Ivnik RJ, Graff-Radforf NR, Willis FB, Petersen RC, Lucas JA. Diagnostic validity of age- and education-corrections for the mini-mental state examination (MMSE) in African American elders. J Am Geriatr Soc. 2012;60(2):328–31.View ArticlePubMedGoogle Scholar
  42. Piccinin AM, Terrera GM, Clouston S, Reynolds CA, Thorvaldsson V, Deary IJ, Deeg DJH, Johansson B, Mackinnon A, Spiro A III, Starr JM, Skoog I, Hofer SM. Coordinated analysis of age, sex, and education effects on change in MMSE scores. Psychological Sciences and Social Sciences. 2013;68(3):374–90.View ArticlePubMedGoogle Scholar
  43. Freitas S, Simone MR, Alves L, Santana I. The relevance of sociodemographic and health variables on MMSE normative data. Applied Neuropsychology: Adult. 2015;22:311–9.View ArticleGoogle Scholar
  44. Zawadka-Kunikowska M, Zalewski P, Klawe JJ, Pawlak J, Tafil-Klawe M, Kedziora-Kornatowska K, Newton JL. Age-related changes in cognitive function and postural control in Parkinson's disease. Aging Clin Exp Res. 2014;26:505–10.View ArticlePubMedGoogle Scholar
  45. Hauser RA, Stocchi F, Rascol O, Huyck SB, Capece R, Ho TW, Sklar P, Lines C, Michelson D, Hewitt D. Preladenant as an adjunctive therapy with levodopa in parkinson disease. Two randomized clinical trials and lessons learned. JAMA neurol. 2015;72(12):1491–500.View ArticlePubMedGoogle Scholar
  46. Ahmad F, Jhajj AK, Stewart DE, Burghardt M, Bierman AS. Single item measures of self-rated mental health: a scoping review. BMC Health Serv Res. 2014;14:398.View ArticlePubMedPubMed CentralGoogle Scholar
  47. Jenkins A, Lindsay S, Eslambolchilar P, Thornton IM, Tales A. Administering cognitive tests through touch screen tablet devices: potential issues. J Alzheimers Dis. 2016;54(3):1169–82.View ArticlePubMedGoogle Scholar

Copyright

© The Author(s). 2018

Advertisement