Innovative Community Based Assessment of Executive Function – The I Can Way

By: 

Leslie Birkett, BSc. OT, OT Reg. (Ont.), Deidre Sperry, M.Sc., S-LP(C) Reg. CASLPO

Independence in daily functioning requires the navigation of complex environments and increasingly complex information systems, making intact executive functioning paramount to survival.  The devastating consequences of Traumatic Brain Injury (TBI) are well known to rehabilitation professionals and it is important that clinicians gain a comprehensive understanding of the person’s cognitive strengths and limitations.  A comprehensive assessment requires that rehabilitation professionals obtain information regarding the individual’s ability to organize his/her activities, stay on task, sort out problems that arise in an efficient and effective manner, prioritize work that needs to be done and keep a watchful eye on him/herself to self-monitor.  This real-world information is vital in returning individuals to their life roles and activities.  When assessments of executive functions are completed in the quiet confines of clinic or office, there is a risk in missing the important dynamic interplay and unpredictability real life offers.  Because of the limitations of clinic based executive function assessments, a clinical “leap of faith” is required to extrapolate a client’s ability to function in community and workplace environments based solely on impairment level test findings.  It is widely agreed that greater attention needs to be paid to evaluating executive functions in everyday life.  However, the question is “how?”  The I CAN is a promising method of conducting executive functioning assessment in natural settings to bridge the gap from clinic to life. 

It is not uncommon for clinicians to be faced with the clinical puzzle involving a person who tests well but does not function well.  This gap may be attributed, at least in part, to the considerable impact of the environment on an individual’s functioning.  Certainly clinical experience and intuition may guide, but that is not adequate to determine best-practice interventions.  Similarly, without a strong understanding of the complex interplay of the various components of executive functioning the picture is also incomplete.

There are numerous definitions of executive functions.  Brown (2006) stated: “Although the definition of executive function is still evolving, most researchers agree that the term should be used to refer to brain circuits that prioritize, integrate, and regulate other cognitive functions”.   This broad-based definition demonstrates the over-arching regulatory role that executive functions provide for effective daily living.

The I CAN is based on the framework of the World Health Organization (WHO).  In order to fully understand the I CAN, it is imperative to be aware of the WHO’s International Classification that includes “Impairment”, “Limitations of Activity”, and “Restrictions of Participation”.  As per the WHO, impairment is a problem in body function or structure; an activity limitation is a difficulty encountered by an individual in executing a task or action while a participation restriction is a problem experienced by an individual with involvement in life situations.   Although this framework is widely referred to in the rehabilitation literature further steps must be taken to develop assessment procedures to address each level of functioning.  Assessments by health care professionals are dominated by information related to impairment.  This phenomenon is puzzling because although the primary focus of neurorehabilitation is to return an individual to daily functioning in real environments, appropriate assessment measures are lacking.  There is a need for assessment procedures that are based on the individual’s specific roles and activities. 

Over the years, a growing number of assessment tools have been developed offering the clinician a method to include assessment of Activity Limitations, but even these forward-thinking tools do not include tasks that are specific to the roles and activities of an individual person with a brain injury.  The I CAN provides clinicians with a means to evaluate a person’s ability to perform activities of everyday living in naturalistic community environments.  It can be tailored to an individual’s specific roles.  It is structured enough to be repeated, yet flexible enough to be person centered.

There are a number of other assessments that were developed to address the important issues related to the ecological validity of assessment of executive functions including the Behaviour Rating Inventory of Executive Functions (BRIEF), the Behavioural Assessment of the Dysexecutive Syndrome (BADS), the Hamburger Turning Task, the Tinker Toy Test, and the Barkley Deficits in Executive Functioning Scale (BDEFS).  However none offers the dynamic, unpredictable, observable, and hierarchical components of the I CAN.  As questionnaires, The BRIEF and BDEFS depend highly on the insight of the self-rater or the accuracy of the informant. The BADS and Tinker Toy Test are typically administered clinic-based assessment tools utilizing pencil and paper and/or task objects.  The Hamburger Turning Task involves a simulated real-world vocational task. These tests lack the dynamic nature of real life and hence must still make a prediction as to how a person will function in the real world of everyday living.

For individuals with a TBI, the assessment process generally focuses on standardized testing at the impairment level whether conducted by a Neuropsychologist, Occupational Therapist or Speech Language Pathologist.  An appointment is made to spend, in some cases, several hours in a quiet room answering questions, doing pencil and paper tasks, and being observed.  Mark Ylvisaker and Tim Feeney encouraged professionals to understand that assessment of executive functions requires contextualized exploration beyond the use of formal tests.  “Traditionally executive system assessment includes the attempt to understand how effectively an individual can size up a possibly confusing situation, decide what to do to achieve meaningful goals, plan how to do it, do it and stick to it until it is done, pay attention to how effectively it was done and identify a clever way to do better if necessary.  By contrast, when we are assessing executive functions, using formal testing, the tasks are presented clearly in a sterile environment: the goal and ground rules are set by the evaluator, the evaluator provides the motivation, initiation and encouragement and takes responsibility for evaluating performance and rarely is there an opportunity for the tested individual to try the task again, having learned from the experience of the first attempt” Ylvisaker, Szekeres and Feeney (1998).  They also stress, “It is necessary to systematically explore the executive dimensions of behavior in real-world contexts and with real-world tasks and stressors”. 

Reliability and validity are the hallmarks of a clinically sound assessment tool.  Reliability is a measure of reproducibility of results whereas validity is a measure of whether a test or tool measures what it purports to measure.  In the development of the I CAN, the emphasis to date has been on developing a tool with strong face validity.  The challenges inherent in determining the reliability of the I CAN lie with the very nature of the tool itself.  Specifically, the I CAN was developed to measure a person’s functioning in the real world.  Hence the individual’s performance is not reproducible as the community is not a controlled environment.  Other clinically sound assessment tools, such as the Canadian Occupational Performance Measure (COPM), which is a measure of a client’s self-perceived changes in occupational performance problems, focused their research on evaluation of specific constructs relative to their tool.  The COPM identified test-retest reliability and the responsiveness of the measure to be research priorities. 

The literature questioning the ecological validity of standardized assessment of executive functioning is rich.  Sbordone & Guilmette (1999) provided an opinion regarding the ecological validity of the assessment of executive functioning. They stated “It is well known that the demands placed on the patient with brain injury within the highly structured test environment are too restrictive to capture many of the difficulties the patient may have in his or her daily life.”  Sbordone furthered this discussion by identifying that the behavioural impairments of a brain-injured person may not be apparent when tested in a quiet and structured environment.  He noted that it was therefore also possible that incorrect conclusions could be drawn.  In 2004, Manchester, Priestly and Jackson noted that often there is only a “weak to moderate relationship between performance on such tests and everyday behaviour”.  As well, they note, “Whilst many patients with frontal lesions and problems with executive functioning in everyday life do perform badly on tests thought to be sensitive to the executive functions, many do not.”  They identified that the testing environment itself may be problematic in that it may be a “poorly conducive arena for eliciting these deficits”. Later Barkley (2010), opined that the use of a battery of neuropsychological tests to determine executive functioning had “low or no relationship to impairments that were evident in various domains in major life activities or to ratings of executive functioning symptoms in daily life in those adults (low ecological validity)”.   

Clinicians are often confronted with this very conundrum.  A review of the literature confirmed our clinical instinct that we must look beyond the information gained using traditional assessment protocols.  Because our clinical practices follow the “Participate to Learn” approach to community rehabilitation, we knew that the answer could be found if we could bring the assessment out of a predictable and manageable environment and into the community and real life activities of everyday living.  Carlson et al. (2006) stated that participation in valued life roles “would provide both the opportunity and the context for any necessary skill learning and attitude change”.  The “Participate to Learn” approach has 3 central tenets (1) roles can serve as rehabilitation goals; (2) learning through experience in real-life activities is a successful rehabilitation strategy; (3) both personal and environmental support can be used to enable participation in valued life roles.  The I CAN incorporates each of these tenets.

The I CAN was developed in an effort to help build our understanding of a client’s ability to formulate realistic goals, plan how to achieve them and carry out those plans effectively in everyday activities.  It was critical to include the opportunity to learn from experience, and for the testing environment to be community based therefore ensuring interruptions, interactions and unexpected events.  The I CAN is a hierarchal, systematic, community based evaluation of executive functioning with tasks inspired by a reality television scavenger hunt.  The I CAN engages clients in three novel, yet familiar, experiences designed to give opportunity for observation of executive functioning. One of the unique features of the I CAN is the interdisciplinary clinical collaboration inherent in designing appropriate challenges.  The complementary perspectives that a Speech Pathologist and an Occupational Therapist bring to the development of the challenges ensure that a wide spectrum of observational opportunities is present. 

Imperative to the success of the I CAN is the inclusion of a specially trained Therapy Support Worker (TSW).  Specific knowledge the TSW requires includes: a strong understanding of executive functions and knowledge of the I CAN assessment process. One of the most critical skills that the TSW needs to possess is the ability tonot intervene unless safety was at risk.  The trained TSW observes how a person manages the various challenges developed by the Regulated Health Professionals (RHP) and reports back using the I CAN check list.

CLINICAL EXAMPLE

The example of M.G., a 29 year old man with a severe TBI, will illustrate how the challenges are developed.

When M.G. was 26 years old he was involved in a motor vehicle collision while riding an All Terrain Vehicle (ATV).  He was not wearing a helmet, was thrown from the vehicle and struck his head.  His Glasgow Coma Scale was 5/15 with CT scan confirming multiple intra parenchymal hemorrhagic foci, predominantly in the left frontal lobe, with small hemorrhages in the right frontal lobe.  He spent one month in an acute care hospital before being discharged to his parents’ home. 

Prior to injury, M.G. worked as a licensed trade person.  His leisure time pursuits included numerous outdoor activities as well as attending cultural events in a nearby large urban centre. 

After his injury he presented with cognitive impairment including: difficulty with concentration, task initiation, problems planning and organizing and impaired multi-tasking.  Clinically his team was aware of the significant impact of fatigue on his functioning.  It was noted in the neuropsychological report that overall he had “relatively good executive cognitive skills”.  Also, as can be typical of the TBI population he demonstrated a lack of awareness of his deficits.  M.G. was hoping to return to his demanding career and independent living.  Despite his “relatively good executive cognitive skills”, M.G.’s rehabilitation team and his family had observed occasions where his functioning in real life situations appeared contrary to the neuropsychological test findings.  The I CAN assessment was requested.

As with all I CAN assessments, the Regulated Healthcare Professionals (OT and SLP) gathered a detailed history.  Together they developed the challenges in a manner that would target key areas of concern.  They also worked closely with a TSW who was unknown to M.G. prior to involvement with the I CAN and as such basic information regarding the client was provided.  They explained to the TSW the challenges, assessment hypotheses and additional safety and behavioural considerations relevant to this case. The TSW accompanied M.G. on all challenges and provided feedback on a structured I CAN assessment form regarding the client’s performance.

The first challenge (Tent Task) required M.G. to pitch a 6-person tent in a local conservation area in time to host a snack with visitors.  To do this he needed to telephone the conservation authority to secure a permit for the tent.  He needed to predict the time to begin pitching the tent to ensure it would be erected in time to prepare the snack for the visitors.  Two days later, he was allotted 5 hours to complete the second challenge (City Trip).  This involved taking a train to a nearby city.  Once there he needed to locate a specific destination in order to obtain three unique items.  Once he had obtained the items, he was informed that the train was no longer running and he needed to find an alternate way home.  The third challenge (Comparison Shopping) was scheduled for the following morning.  M.G. received a telephone call from a “customer” requesting a recommendation for the purchase of an appliance.  A number of specifications were provided.  M.G. needed to comparison-shop and develop a response for his customer by the end of that day. 

The Tent Task was chosen as a first task because of M.G.’s high level of familiarity and enjoyment regarding outdoors activities.  The novelty of the task for M.G. was that he had never erected the model of tent provided.  So while the task was highly familiar, the specifics were not.  This task required him to initiate, predict, plan, problem solve and co-operate, as he had to work with the TSW to erect the tent.  The City Trip involved a more complex set of tasks.  The trip allowed for the observation of his ability to manage time, adjust plans, problem solve and make decisions. While M.G. was familiar with this city and the initial method of transit, the complexity of the task involved sending him to an ethnic neighbourhood in the city with which he was not familiar and then requiring him to find an alternate means of transportation home.   The Comparison Shopping task was scheduled for the morning after his trip to the city in order to allow the opportunity to observe the impact of fatigue.  This task also challenged his ability to manage time, but was the most complex task due to the significant need for self-direction.  It also had a considerable cognitive-communication component, as it was imperative that he process the instructions from the customer correctly and be able to interact with sales staff in a busy store to gather the information necessary to complete the assignment.  He also needed to be able to develop a plan of action that would address those specific concerns and prepare a response in a timely manner.

The I CAN utilizes a four-point rating scale and each skill is rated on this scale.  The scale rating is as follows: 1 – no evidence of skill; 2 – skill is emerging; 3 – skill is present but not yet consistent; 4 – skill is evident and consolidated.

The first, and perhaps the most important aspect of the I CAN experience was the high degree of client engagement throughout the process.  Participation in the I CAN allowed M.G. to gain increased insight into the impact of his executive cognitive functioning difficulties.  He now, more fully, understands how the impairments identified in his neuropsychological evaluation impact his participation in daily activities.  M.G.’s increased understanding evolved through his participation in the specific tasks chosen and his self-review of his performance elicited through a guided interview with the evaluator.

The information gained from the I CAN provided M.G.’s treatment team with important information to contextualize his intervention with the goal of improving function in everyday activities. A broader understanding was gained regarding the challenges M.G. faced to return to work and independent living and the supports necessary to achieve success.

CLINICAL UTILITY

The literature supports employing more ecologically valid means of assessment protocols to evaluate the executive functioning of people with TBI (Barkley 2010, and Ylvisaker, Szekeres,  & Feeney, 1998).   Moving away from direct cognitive remediation retraining is another strong theme (Coelho, Ylvisaker, & Turkstra, 2005).  Also, In 2006 Mark Ylvisaker advised clinicians of the limited successes yielded by restorative cognitive exercises.  He recommended that instead of the provision of cognitive exercises to the individual with a brain injury, that provision of rehabilitation services through supported everyday routines would be more efficacious.  The evidence to date is that the I CAN follows best practice principles by providing treatment teams with clinically relevant information that promotes client participation in valued life roles.  This has been validated by client feedback regarding their experience with participation in the I CAN and the resultant changes to their rehabilitation program.  Moreover, respected rehabilitation professionals including treating psychologists, occupational therapists and speech-language pathologists have found the information captured through use of the I CAN assessment to be clinically valuable and contextualized.  In specific, the real-life examples of the client’s executive functioning have provided the treating teams with important clinical data. 

While the I CAN appears to be a promising new assessment tool, it requires more rigorous psychometric analysis.  This must be done with great care so as to not sacrifice face validity.  It is not possible to standardize real life.  We cannot control for the spontaneous problems, stressors and responses to stressors one finds in different scenarios of everyday living.  We will however strive to develop sets of prescribed scenarios that will have a core of predictable activities within.  Inter-rather reliability will be addressed through the use of videotaped scenarios of real people participating in various I CAN challenges.

FUTURE DIRECTIONS

The I CAN is proving to be a valuable clinical tool.  Our current efforts are directed towards increasing our clinical sample size to increase our understanding of the use and face validity provided by this tool. Long-term plans include the development of a training program and manual for clinicians wishing to utilize this exciting tool.

References:

  • Barkley, A. (2011). Barkley Deficits in Executive Functioning Scale. New York, NY: Guilford Press.
  • Barkley, A. (2010). Evaluating Executive Functioning Deficits in Everyday Life. The ADHD Report, 18, 9-10.
  • Brain Injury Association of New York State. http://www.bianys.org. Learnet tutorials. Accessed October 2009.
  • Brown, T. E. (2006). Executive Functions and Attention Deficit Hyperactivity Disorder: Implications of two conflicting views. International Journal of Disability, Development and Education, 53(1), 35–46.
  • Carlson, P. M., Boudreau, M., Davis, J., Johnson, J., Lemsky, S., McColl, M. et al. (2006). “Participate to Learn”: A promising practice for community ABI rehabilitation.  Brain Injury, 20(11), 1111-1117.
  • Coelho, C., Ylvisaker, M., & Turkstra, L. (2005). Non-Standardized Assessment Approaches for Individuals with Traumatic Brain Injuries. Seminars in Speech and Language, 26(4), 223-241.
  • Gan, C., Campbell, K. A., Snider, A., et al. (2008). Giving Youth a Voice (GYV): A measure of youth’s perceptions of the client-centeredness of rehabilitation services. Canadian Journal of Occupational Therapy, 75(2), 96-104.
  • Gordon, M., Barkley, R. A., & Lovett, B. J. (1998). Tests and observational measures. In: R. A. Barkley (Ed.),Attention deficit hyperactivity disorder: A handbook for the diagnosis and treatment (pp. 294-311). New York, NY: Guilford Press.
  • Koehler, R., Wilhelm, E., & Shoulson, I. (2011). Cognitive Rehabilitation Therapy for Traumatic Brain Injury: Evaluating the Evidence. Washington, DC: National Academy of Sciences.
  • Larkins, B. (2007).  The application of the ICF in cognitive-communication disorders following traumatic brain injury. Seminars in speech and Language, 28(4), 334-342.
  • Law, M., Baptiste, S., Carswell, A., McColl, M., Polatajko, H., & Pollock, N. (1994).  Canadian Occupational Performance Measure, Second Edition. Toronto, Ontario: Canadian Association of Occupational Therapists.
  • Lezak, L., Howieson, D., & Loring, D. (2004). Neuropsychological Assessment 4th Ed. New York, NY: Oxford University Press.
  • MacDonald, S., & Johnson, C. (2005). Assessment of subtle cognitive-communication deficits following traumatic brain injury. The normative study of the functional assessment of verbal reasoning and executive strategies. Brain Injury. 19(11), 895-902.
  • Manchester, D., Priestly, N., & Jackson, H. (2004). The assessment of executive functions: coming out of the office. Brain Injury, 18(11), 1067-1081.
  • Roth, M., Isquith, P., & Gioia, G. (2000). Behaviour Rating Inventory of Executive Function – Adult Version (BRIEF®-A). Lutz, Florida; PAR.
  • Sbordone, R. J., & Guilmette, T. J. (1999). Ecological Validity: Prediction of Everyday and Vocational Functioning From Neuropsychological Test Data. In J. J. Sweet (Ed.), Forensic Neuropsychology Fundamentals and Practice (pp. 227-254). Lisse, The Netherlands: Swets & Zeitlinger.
  • Sbordone, R. J. (2000). The Executive Functions of the Brain.  In: G. Groth-Marnat (Ed.),Neuropsychological Assessment in Clinical Practice (pp. 437-456). New York, NY: John Wiley & Sons Inc.
  • Sbordone, R. J. (2001). Limitations of neuropsychological testing to predict the cognitive and behavioral functioning of persons with brain injury in real world settings. NeuroRehabilitation, 16, 199-201.
  • Shugars, S. (2007). A Functional Assessment of Executive Functioning: The Hamburger Turning Task.Pittsburgh, PA: University of Pittsburgh.
  • Silver C. (2000). Ecological Validity of Neuropsychological Assessment in Childhood Traumatic Brain Injury. Journal of Head Trauma Rehabilitation, 4(15), 1022-1040.
  • Szekeres, M., Ylvisaker, M., & Holland, A. L. (1985). Cognitive Rehabilitation Therapy: A framework for intervention.  In: M. Ylvisaker (Ed.), Head Injury Rehabilitation: Children and Adolescents (pp. 219-246). Austin, TX: Pro-Ed Inc.
  • Wilson, B., Alderman, N., Burgess, P., Emslie, H., & Evans, J. (1996). Behavioural Assessment of the Dysexecutive Syndrome. London, England: Pearson Assessment.
  • Ylvisaker, M., Szekeres, S., & Feeney, T. (1998). Cognitive Rehabilitation: Executive Functions.  In: M. Ylvisaker (Ed.), Traumatic Brain Injury Rehabilitation: Children and Adolescents, Second Edition (pp. 222-269).  Boston, MA: Butterworth-Heinemann.
  • Ylvisaker, M. (2008). Self-Regulatory Conference. Toronto.

Authors

Leslie Birkett, BSc. OT, OT Reg. (Ont.)
Occupational Therapist in Private Practice
Burlington, Ontario

Deidre Sperry, M.Sc., S-LP(C) Reg. CASLPO
Speech-Language Pathologist in Private Practice
Dundas, Ontario