Computerized Neurocognitive Testing

By: 

C Thomas Gualtieri MD, North Carolina Neuropsychiatry Clinics

Neuropsychological testing is essential to the evaluation of patients with brain injury, but brain injuries do not always occur when neuropsychologists are nearby. The sheer numbers of brain injuries, especially concussions, have inclined clinicians and policymakers towards the use of computerized neurocognitive testing (CNT) .  Professional sports teams, university athletic programs and the armed forces of many nations have begun to use CNT as a way to manage the extraordinary number of people with brain injuries.

New technology in the service of improved care for more people, and at a drastically lower price, is something we should all be happy about. CNT has extraordinary potential in brain injury, just as it does in other fields.  However, this new health technology was not born fully formed; it needs to be nurtured to attain its full potential. 

CNT began in the early 1980s, with the old Commodore microcomputers. The technology was, for many years, quite obscure.  Sophisticated testing batteries were used in occupational health and in aerospace and military medicine; then, later, in clinical trials for new drug development. It is only in the past five years, however, that new, commercially available CNT’s have begun to have an impact on clinical practice.   ImPact1, for example, is a CNT that was specifically developed for sports concussion, and is used by several professional teams. Neurotrax2 is a CNT marketed to neurologists for early dementia detection. CNS Vital Signs, a CNT  which the author helped develop, is widely used by neuropsychologists and physicians, and is a broad spectrum battery appropriate for many different clinical applications.3

It is likely that CNT will be used more frequently in coming years, and will be increasingly important in the evaluation and management of patients with brain injury. As that happens, however, we all would do well to remain cognizant of certain problems that are intrinsic to, and perhaps even unique to this new technology.

CNT’s are extremely sensitive to virtually all of the clinical conditions associated with cognitive dysfunction.  They are capable of calculating reaction times with millisecond accuracy, and can generate massive amounts of precise data.  The technology, however, can be a mixed blessing.  Data can be misinterpreted or misused by poorly trained clinicians.  In our communications with physicians and even psychologists who have used CNT’s in their practices, we have not always been impressed by their facility at judging exactly what the tests mean and how to respond appropriately to the results they generate.

On the other hand, if only highly qualified specialists were allowed to interpret the results of CNT’s, that would defeat the purpose of computerization, which is to make cognitive evaluation widely available to all the people who need it. Concussions in particular and brain injuries in general, are prevalent.  So are other medical conditions associated with mild cognitive dysfunction: attention deficit disorder and learning disabilities, mild cognitive impairment and early dementia, cognitive dysfunction associated with cancer, chemotherapy, cardiovascular disease, obesity and diabetes, hepatic encephalopathy, epilepsy, stroke, migraine, alcoholism and substance abuse.  In all of these areas, CNT is the most practical way to evaluate the cognitive components of the condition.  This is not simply an academic matter.  Cognitive dysfunction is an important predictor of medical morbidity, even in patients who do not have an overt cognitive disorder.4

CNT should not be the exclusive precinct of an elite group of doctors. The evaluation of cognitive impairment is simply too important to be left to the experts.  However, clinicians who will use CNT’s need to be aware of some basic principles of test theory, psychometrics and appropriate clinical application.

Is A CNT A “Standardized” Test?

The older CNT’s were used as research instruments, where a subject’s performance would simply be compared to his own baseline.  That was suitable for certain kinds of research but not for a clinical instrument.  Some of the newer CNT’s have been standardized on the basis of hundreds or thousands of normal subjects.5 

Why is this important if all one plans to do is compare a patient to his or her own baseline?  It has to do with the validity of a test.  A test is not valid if scores do not improve with maturation and decline with ageing.  If a test battery is sound, normal females should score higher on tests of verbal memory and normal males should score higher on tests of visual memory; males are faster at finger tapping and females are better at coding.  A normative database does not only give one an idea of where a particular subject stands with respect to norms, it assures you that the structure of the test is valid.  A “standardized” test is one that generates scores in reference to a well-constructed normative database. 

The most useful CNT’s have large numbers of clinical comparison groups – patients with a wide range of psychiatric and neurological disorders.  This is one aspect of the discriminant validity of a test.  Scores should improve when ADHD patients are treated with stimulants and they should go down if normal subjects are treated with scopolamine or benzodiazepines.  Patients with depression should perform better after they are treated.   Patients with multiple sclerosis should show evidence of slow processing speed and patients with early Alzheimer’s disease should have deficits in memory, attention and perceptual ability.  If a test doesn’t generate data such as these, then its use in brain injury is reasonably suspect.

A CNT should also have concurrent validity – that is, it should generate results that are similar to established neuropsychological tests.  Because many CNT’s are “idiosyncratic” – that is, tests that are unique to that particular test battery, and quite different from conventional tests – concurrent validity cannot be assessed.  The computer format is given to new ways of testing cognitive function, and many of the idiosyncratic test batteries are quite useful, at least in research.  For clinical purposes, however, it is better to choose a CNT that is comparable to tests that clinicians are familiar with.  In clinical practice, obscurity is never a virtue.

How Reliable Is CNT?

When we reviewed the test-retest reliability of a number of different CNT’s, we found that they were just as reliable as conventional paper-and-pencil neuropsychological tests.3   However, the reliability of three commonly used concussion management batteries was found to be only “low-to-moderate” when they were evaluated by independent researchers.6   Different concussion batteries do not always generate the same results.7  Concussion history is not necessarily manifest in the results of these CNT’s.8

Can A Computerized Test Be Diagnostic?

No test in medicine or psychology is “diagnostic.”  Diagnosis is a clinical exercise that demands the integration of data from multiple sources.  The results of a test, whether it’s a blood count or a formal neuropsychological battery, may incline the clinician’s thinking in one direction or another.  But test results do not, by themselves, constitute a medical diagnosis.

Our research with CNT’s, beginning with the old NES2, have shown quite clearly that CNT’s do not have the specificity necessary to qualify as diagnostic instruments.910  They are highly sensitive to mild cognitive dysfunction, though, and that makes them suitable to be used as screening instruments.  But a sensitive test will also generate a high percentage of false positives.  In our clinics, every new patient is administered a comprehensive CNT.  It is not uncommon to encounter patients with one or more cognitive domain score that is more than two standard deviations below the population mean.  But that is the equivalent of discovering a mild abnormality on an MRI scan or an EEG.  “Clinical correlation is needed,” as they say.  Absent a clinically meaningful explanation, the best thing to do is to repeat the test at some later date, as the patient is seen in follow-up.

Should Athletes Be Baselined For Concussion Management?

Because such a high proportion of the “normal” population have one or more areas of cognitive weakness – subtle learning disabilities, for example, that have never been identified11 – it is essential to have a baseline before the patient engages in any endeavor where concussion is likely to occur. 

Sports medicine is only one area where concussion is at issue.  Military medicine is another.  In fact, in any activity where physicians are required to attest to a patient’s safety to return to work following an injury, baseline and serial cognitive testing ought to be helpful.  Police, firefighters, commercial drivers, people who operate heavy machinery or who work at heights – sooner or later they will all be baselined.  The cost of CNT – a few dollars a test – is more than outweighed by the savings in insurance premiums and the benefit to public safety.

A medical record that includes neurocognitive testing when the patient is in a healthy state is probably a good thing.  Relying on computerized neurocognitive testing to determine whether a young athlete is safe to return to play is something else again.  The value of CNT’s for this purpose may well have been overstated in the literature.  Taking American football as an example, a recent review concluded that “serious short-term consequences of sport-related concussion are extremely rare and unlikely to be significantly modified via management strategies that rely on baseline testing.”12

Some athletes have learned to “game” the system; to deliberately do poorly at baseline in order to protect their playing status if they happen to be concussed.  Very short test batteries have a certain appeal for an athletic evaluation, but they seem to have problems with reliability.6   Few, if any, achieve the test-retest criterion of 0.90 that is desirable for making decisions about individual change.13,  14

Nor is CNT sufficient for the purpose of concussion management; symptom severity, neuropsychological function, and postural stability are not related nor are they affected to the same degree after concussion.15  The ability of a gifted athlete to improve his or her performance in a test that requires psychomotor speed and accuracy far exceeds that of an ordinary person, so improvement in performance with serial testing does not necessarily indicate full recovery from the effects of mild brain injury.  The danger of CNT’s in sports medicine is that complex psychophysical data are interpreted by people whose qualifications are less than well-suited for that purpose (e.g., athletic trainers and high school coaches), and that a false sense of security would thus be engendered. 

Perhaps computerized concussion management is an improvement over doing nothing.  It is no substitute, though, for a medical examination, and is not the only criterion that should determine an athlete’s fitness for play. 

Is It Possible To Fake The Results?  How Do I Know If A Patient Is Faking It?

How to know a test score is invalid?  That subject is amply dealt with in the neuropsychological literature; there are several tests specifically designed to detect sub-optimal effort and malingering.  It is also possible to “embed” validity indicators into a standard battery of tests.  For example, a memory test can be a “forced choice” test, where random responding will generate a score of 50%, and less than 50% indicates willful exaggeration.  Another way to detect potentially non-organic presentations is when a patient’s test performance is inconsistent with his clinical presentation and history as well as the nature of the complaint.  Athletes who are being baselined, or pseudo-patients who are pretending to have had a concussion, should never score as poorly as patients with early dementia or severe brain injury.  The most useful CNT’s for clinical practice contain built-in algorithms to detect invalid test results.3

What Is A Clinically Meaningful Change In Test Scores?

A good deal of mental energy has been put forth on the issue of “reliable change,” that is, whether a change in test scores is meaningful or not.16 In other words, if an athlete scores lower than baseline after he or she is concussed, is the change meaningful, or is it simply the result of test variance?  Determining the probability of score changes of different magnitudes over different time intervals is critical to the interpretation of neurocognitive retest data, and no NP test battery can be considered complete without providing this information.12  Alternative methods to calculate a “reliable change index” (RCI) have been proposed, based on standard error calculations, test reliability, etc.1718   Although RCI’s are essential for the purpose of sports concussion management, the CNT’s most commonly used for that purpose have yet to report rigorous data.   

CNT Summary

The advantages of CNT are practical and scientific.  The practical advantages have to do with efficiency and accuracy.  They are inexpensive and quick to administer.  There is no possibility of scoring error or tester bias.  The scientific advantage is the wealth of data that can be generated that are simply not possible in conventional testing, for example, reaction times in milliseconds and response variability. 

CNT’s have disadvantages as well.  Computerized tests are run on a tight clock, so reaction time and processing speed are central to someone’s performance.  In computerized testing, there is no tester present to maintain the patient’s focus, or to ease off when he or she grows weary; thus, effortful attention is also a key element of performance.  This makes them suitable for use in brain injury patients because processing speed and attention are key deficits in patients who have had a brain injury.  But it is also a potential source of bias in the interpretation of tests of executive function, for example.

Like guns and automobiles, however, the only dangerous thing about a CNT is the man or woman who is using it.  Like every new technology, there is the risk of over-use and inappropriate application.  CNT’s have great potential, but they are not likely to drive neurologists and neuropsychologists out of business.

 

REFERENCES

1 Lovell MR, Collins MW. ImPACT: Immediate Post-Concussion Assessment and Cognitive Testing. Pittsburgh, PA: Neurohealth Systems, LLC; 1998

3 Gualtieri, CT & Johnson, LG, Reliability and Validity of a Computerized Neurocognitive Test Battery, CNS Vital Signs.  Archives of Clinical Neuropsychology, 21, 623-643, 2006.

4 Ehlenbach, WJ, Hough, CL, Crane, PK, et al.  Association between acute care and critical illness hospitalization and cognitive function in older adults.  JAMA 303, 763-770, 2010.

6 Broglio SPFerrara MSMacciocchi SNBaumgartner TAElliott R.  Test-retest reliability of computerized concussion assessment programs. J Athl Train. 2007 Oct-Dec;42(4):509-14.

7 Schatz PPutz BO.  Cross-validation of measures used for computer-based assessment of concussion. Appl Neuropsychol. 2006;13(3):151-9.

8 Broglio SPFerrara MSPiland SGAnderson RBCollie A.  Concussion history is not a predictor of computerised neurocognitive performance.  Br J Sports Med. 2006 Sep;40(9):802-5; discussion 802-5.

9 Arcia, E., and C.T. Gualtieri. (1994). Neurobehavioral performance of patients with closed head injury, attention deficit, and controls. Brain Injury, 8, 395-404.

10 Arcia, E., and C.T. Gualtieri. (1993). Association between patient report of symptoms after mild head injury and neurobehavioral performance. Brain Injury 7, 481-489.

11 Culbertson, JL, Edmonds, JE.  Learning disabilities.  In Neuropsychology for Clinical Practice: Etiology, Assessment & Treatment of Common Neurological Disorders.  Eds, Adams, RL, Parsons, OA, Culbertson, JL & Noxon, SJ.  1996.

12 Randolph CKirkwood MW. What are the real risks of sport-related concussion, and are they modifiable?  J Int Neuropsychol Soc. 2009 Jul;15(4):512-20.

13 Nunnally JC, Bernstein IH. Psychometric Theory. 3rd ed. New York, NY: McGraw-Hill, Inc; 1994.

14Randolph C,McCrea M, Barr WB.  Is Neuropsychological Testing Useful in the Management of Sport-Related Concussion?  J Athl Train. 2005 Jul–Sep; 40(3): 139–152.

15 Ellemberg D, Henry LC, Macciocchi SN, Guskiewicz KM, Broglio SP.  Advances in sport concussion assessment: from behavioral to brain imaging measures.  J Neurotrauma. 2009 Dec;26(12):2365-82

16 Jacobson NS, Traux P. Clinical significance: a statistical approach to defining meaningful change in psychotherapy research. J Consult Clin Psychol 1991;59:12–19.

17 A CollieP MaruffM McStephenD Darby.  Are Reliable Change (RC) calculations appropriate for determining the extent of cognitive change in concussed athletes?  Br J Sports Med 2003;37:370-372

18 N R Temkin.  Standard Error In The Jacobson And Truax Reliable Change Index: The “Classical Approach” Leads To Poor Estimates.  Journal of the International Neuropsychological Society (2004), 10:6:899-901