Needs Assessment and Stakeholders in Medical Simulation Curriculum Development

Function

A needs assessment is the process of assessing the training and educational needs of individuals, groups, or organizations and aligning their needs with the curriculum.[4] Arguably the most crucial stage in training and curriculum development needs assessment informs the learning objectives, design, delivery, and evaluation of training and is the foundation of training success.[4][5] However, needs assessment is often largely overlooked, as training and curriculum developers often are not aware of its importance or are unaware of how to conduct a needs assessment.[4] This is problematic because research has shown that curricula developed based on results from a needs assessment improve learning and transfer of learning to the job environment more than curricula developed without a needs assessment.[6] Therefore, the goal of this article is to provide guidance on how to properly conduct a needs assessment, particularly in the context of developing a simulation curriculum, and to explicate the role of various stakeholders who can affect, are affected by, or are involved in curriculum development.

Curriculum Development

There are commonly considered to be three components of needs assessment: organizational analysis, job or task analysis, and person analysis. Below, we discuss the description of each component, stakeholders involved, and methods to conduct the analysis.

Organizational Analysis

Organizational analysis is the process of assessing the broader contexts, groups, and organizations (e.g., hospital, specialty program, and medical school) in which the curriculum is to be integrated.[4] More specifically, organizational analysis can inform the development of a curriculum that aligns with the goals, needs, and strategies of the organization that it is to be implemented within. This helps ensure that the curriculum will receive buy-in from leaders and stakeholders. For example, if a primary goal of a hospital is to achieve healthcare equity, simulations intended to teach providers cultural competency skills will likely receive stakeholder buy-in, and these simulations would thereby be well-received as part of a curriculum. In addition to assessing the goals of stakeholders within an organization, the organizational analysis also entails identifying the requirements of relevant external groups or stakeholders (e.g., Accreditation Council for Graduate Medical Education when designing a curriculum for medical residents, Accreditation Commission for Education in Nursing when developing a curriculum for nursing students), when relevant. This helps ensure that the curriculum will be aligned with the needs of its learners, as defined by these external agencies.

Furthermore, the organizational analysis assesses whether the organization has the resources necessary for curriculum development and sustainability. For instance, virtual reality (VR) simulations have been increasingly used to train healthcare professionals, both technical and non-technical skills.[7] However, VR can be expensive to develop and maintain; organizational analysis can inform whether an organization has (or can obtain) the resources to support the inclusion of VR-based simulations in a curriculum. In a similar vein, organizational analysis can help determine the fidelity and/or type of simulations that can be included within a given curriculum. That is, if the organization neither has—nor can acquire— an advanced human patient simulator, then a simulation wherein this equipment is essential would not be merited to include in a curriculum.

Finally, organizational analysis entails assessing the work environment by determining aspects of the organization that might facilitate or interfere with the implementation and sustainability (e.g., support from supervisors and peers) of the curriculum, as well as the transfer of learning to the job environment. Take, for example, a surgery resident who has completed robotic surgery simulation training. A supervisor (e.g., attending surgeon) who is open to providing sufficient opportunities for the resident to perform robotic surgery in the operating room will facilitate the resident’s transfer of learning from the simulation-based robotic training to the job environment. However, if an attending is hesitant to allow residents to perform the robotic operations regardless of the number of hours practiced on the simulator, less transfer of training will occur, and the simulation will not be as value-added in the curriculum. In other words, supervisor support, which is assessed through organizational analysis, facilitates the transfer of learning and, ultimately, the success of the curriculum.

To conduct an organizational analysis, one-on-one semi-structured interviews and/or focus groups can be conducted with parties such as organizational leaders, faculty, managers/supervisors, curriculum directors/coordinators, peers, and patients. Through guided conversations with these stakeholders, the issues mentioned above (i.e., the broader context of the curriculum, available resources, and barriers and facilitators) can be uncovered. However, some training may be required to collect and analyze the interview and focus group data appropriately, and multiple individuals are involved in these processes. Thus, sufficient time and resources should be allocated to conduct an effective organizational analysis. An alternative, less resource-intensive source of data that can help discover the goals, needs, and issues of the organization includes a review of documents (e.g., reports, newsletters, and websites). However, document review provides a more limited view and thus should serve as a supplementary method rather than the sole method of organizational analysis. 

Task Analysis

Results from an organizational analysis can serve as a benchmark in the beginning to identify what learners need to learn (informing curriculum development). More specifics of the knowledge, skills, attitudes, and other characteristics (KSAOs) required to perform job tasks can then be assessed through task analysis. Task analysis is the process of studying activities or tasks performed in a given job, the conditions under which the tasks are performed, and the KSAOs required to perform these tasks.[4] The purpose of task analysis in curriculum development is two-fold: First, results from a task analysis inform the learning objectives of a curriculum by identifying KSAOs that learners need to learn and acquire to perform the tasks on the job. For example, in performing robotic surgery, surgeons need to know not only the anatomy of a human body but also the different functions of robotic arms and how to use them. Therefore, one of the objectives in this simulation may be for learners to become proficient in using all robotic arms. Second, task analysis is beneficial in developing the content of the simulations. For instance, understanding how a decision is made to perform an emergency cesarean section can help create the conditions of a realistic simulation scenario.

To conduct a useful task analysis, a preliminary list of tasks and KSAOs specific to the learner group of interest (e.g., intensive care nurses, orthopedic surgeons, outpatient pharmacists) is prepared by working with subject-matter experts (i.e., incumbents [current employees on the job and targeted learners of the curriculum], their supervisors and/or expert panels comprised of representatives of the learner group). Below we describe these methods for working with incumbents and supervisors and then for working with experts.

Task Analysis with Job Incumbents

First, incumbents can be interviewed and/or observed to develop a preliminary list of tasks and KSAOs. Then, the list should be confirmed with incumbents and their supervisors in one-on-one meetings, focus groups, or surveys. During this process, both incumbents and supervisors are asked to rate the tasks performed on the job in terms of (a) how frequently they are performed, (b) their importance, (c) how much time is spent performing them, and (d) their difficulty. Based on these ratings, the list of tasks and KSAOs are refined. KSAOs can be prioritized, and as it is likely, there will be more KSAOs than what can be feasibly learned and practiced in a given curriculum.

Further, not all of the KSAOs will be amenable to learning via simulation. For example, positive personality traits (e.g., conscientiousness) and individual characteristics (e.g., strong work ethic), although important and desirable, cannot be trained in simulations. Thus, decisions should be made to prioritize those KSAOs that can be best learned in simulations, as well as based on results from the organizational analysis on resource availability and work environment. Finally, if the curriculum intent is to span across an extended period (e.g., over four years of medical school or three years of residency), tasks and KSAOs targeted in the curriculum can be ordered based on specific periods wherein they are most appropriate (e.g., year one vs. year two) by considering the difficulty and required prerequisite KSAOs for each.

Task Analysis with Expert Panels

An alternative (or supplementary) approach to conducting task analysis is via working with a panel of expert stakeholders in the specific learner group (e.g., stakeholders who have extensive experience and/or are national leaders, such as department chairs, program directors, residency directors, and professors of the given content area). In this method, the expert group can follow the Delphi method using surveys to identify and refine tasks learners need to perform on the job.[8] Following this process, the experts generate a preliminary list of tasks that can then be content-analyzed to remove duplicate and irrelevant tasks. After the content analysis, the experts rate the tasks based on (a) how frequently they are performed, (b) numbers of individuals in the learner group of interest who should be able to perform them, and (c) their impact on patients. Curriculum developers can then rate the feasibility of incorporating the skills needed to perform these tasks into simulations. Descriptive statistics are then used to prioritize skills based on all of these data. Finally, the experts review the prioritized skills in terms of their fit and priority for a simulation curriculum. From here, curriculum developers can finalize the list of prioritized skills to then integrate them into the curriculum.

As task analysis takes place through interviews, observations, and surveys, curriculum developers need to be aware of the advantages and disadvantages of these methods. Similar to the advantages and disadvantages of using interviews mentioned in conducting organizational analysis, observation methods allow for collecting rich data in a natural job environment without interrupting the flow of work. However, it can require trained personnel and intensive resources to collect and analyze such data. Compared to interview and observation methods, survey methods are easier to administer, can require fewer resources, and can facilitate the collection of data from a large sample. However, surveys can be subject to bias, and particularly in conducting task analysis, incumbents might provide inflated ratings so that their tasks and KSAOs seem essential to make their job seem more vital and desirable.[9] Therefore, ratings from incumbents’ supervisors should ideally be included in conducting task analysis to balance out these biases.

Person Analysis

Person analysis is the process of assessing (a) who needs training and (b) if learners are ready for training.[4] That is, the primary goal of the person analysis is to ensure that the curriculum design meets the specific needs of the targeted learners. This goes beyond task analysis, wherein job-relevant KSAOs are determined and prioritized for inclusion in a curriculum based on gaining an understanding of the tasks performed on a given job and KSAOs needed to complete the tasks but without much consideration of learners' current or average performance within each KSAO. Person analysis takes this one step further to develop a curriculum that is more tailored to the specific learners in an organization, program, or setting. In other words, task analysis informs KSAOs that should and must be included in a curriculum, whereas person analysis assesses current KSAOs and diagnoses where performance deficiencies are. Consider the example of a community hospital that is considering incorporating various KSAOs identified via task analysis as relevant and essential for emergency medicine into a simulation curriculum program for emergency medicine providers. A person analysis may reveal that emergency medicine providers at this small community hospital are knowledgeable and skillful in some of the KSAOs identified as necessary, such as asthma management, but not in others that were also highly rated, such as pediatric transfusion practices. Thus, prioritizing the management of asthma in the curriculum and developing simulations around it might be superfluous, whereas emphasizing pediatric transfusion KSAOs would be more value-added for these learners.

To conduct a person analysis, interviews or surveys can be used to collect data from incumbents, supervisors, and peers. These stakeholders can be asked their assessment of incumbents’ current performance, as well as proficiency and confidence in KASOs required to perform their job. As incumbents might provide inflated self-ratings, it is important to include multiple perspectives from their supervisors and peers on incumbents’ current performance and developmental goals.[9] In addition to interviews and surveys, educators and curriculum developers can review performance appraisals, if available and accessible, in the organization. Performance appraisals can provide information on incumbents’ current performance as well as their developmental areas and goals based on their self-ratings and supervisors’ ratings. Moreover, although person analysis may find that specific incumbents need training, this does not necessarily ensure they are ready for the training. In other words, if learners do not have minimum experiences and pre-requisite KSAOs, they will not be prepared and competent to take a particular simulation module in a curriculum. For instance, a simulation module designed for a resident might not be appropriate for a medical student. On the other hand, some learners might be much further along in the curriculum than their peers, and they might be ready for advanced training. Therefore, results from a person analysis will help educators and curriculum developers determine who needs and is prepared for training.

Clinical Significance

The need for continued learning is ever-growing in clinical education and practice, and training is one of the most commonly used and effective methods to improve learning.[10] Simulation-based training, in particular, is an important part of curricula to deliver and acquire new knowledge and skills. Educators and curriculum developers need first to conduct a needs assessment to assess what clinical knowledge and skills learners need to acquire and/or improve, who the targeted learners are, and how to design and develop simulations. Without a thorough needs assessment, there can be a disconnect between what is offered in a curriculum and what the learners need. This can lead to clinicians who have significant gaps in essential areas of clinical knowledge and skill. A robust needs assessment ensures that there is no such gap and helps educators and curriculum developers make informed decisions in the simulation curriculum development process.

Enhancing Healthcare Team Outcomes

Decades of research have established that interprofessional teamwork is one of the most important fundamental drivers of the quality and safety of healthcare delivery, as well as patient, staff, healthcare team, and organizational outcomes.[11] Training is widely used and evidenced to improve teamwork KSAOs such as communication and conflict resolution.[12] In addition to informing educators about the key clinical KSAOs that learners need, needs assessment can illuminate learners’ gaps in essential teamwork KSAOs that can be trained in a simulation-based training curriculum. For instance, in the event of a multi-patient trauma, in addition to the clinical skills needed to care for this patient, first responders need teamwork skills to work together to respond to the emergency effectively and efficiently. Needs assessment can uncover the teamwork KSAOs required in such a situation (e.g., team briefing and communication), as well as who requires training to improve these teamwork KSAOs. The process of needs assessment also helps develop, deliver, and evaluate simulations based on the types of emergencies that these first responders are likely to encounter in the real world and how they can apply needed teamwork KSAOs in these emergencies. As such, needs assessment can lead to improved safety, quality, teamwork, and patient outcomes by allowing educators to close the gap learners may have in the KSAOs that have been evidenced to improve such outcomes.