Optimized SRT assessment process establishment through DMAIC cycle
This study followed the lean six sigma DMAIC methodology  to optimize the process of resident assessment using an App-based e-training platform, a mobile app called “XUEYIKU” developed by Peking University Third Hospital (Fig. 1). DMAIC refers to the five sequential phases of process improvement: Define, Measure, Analyze, Improve, and Control . The DMAIC tool of SigmaGuide (http://www.sigmalogic.de) was used to help implement the DMAIC cycle. It supports collecting, processing and integrating the necessary information along the five phases of DMAIC and provide tools and charts along the DMAIC phases.
Panels hearing from students and teachers about resident assessments were held every month. Complaints and opinions were filtered and translated into requirements. It was confirmed that the key need shared by both clinical students and educators was to have no unplanned delay or cancellation (UDC) of resident assessment. To integrate and clarify the scope of this study, the core process was mapped to the Supplier–Input–Process–Output–Customer (SIPOC) diagram (Fig. 2). A study team of multidisciplinary experts including administrators, clinical teachers, and evaluation experts was established. The main issue or defect was defined as UDC, which results in unnecessary or additional work. The definition of the rate of UDC is cases of UDC/total planned resident assessments per semester. After comprehensive consideration based on the baseline data, the target of this study was set as reducing the incidence of UDC to zero by March 2022.
In this project, the pre-study process started with an exam and ended with the filing of examination materials (Fig. 2). Factors causing delay or cancellation were analyzed across the process using the wastes framework (Table 1). In total, 18 sub-wastes were identified through brainstorming of Input–Analyze, with actual examples after 26 field observations and 16 team meetings since June 2021. Online surveys developed by the study team were conducted using random sampling in 36 clinical departments to identify the negative factors influencing resident assessment. We then compared the rate of UDC, satisfaction with the whole process of resident assessment (using a five-point Likert-type scale), and time savings before and after the application of an online app developed using informational technology (See Improve).
The collected data on factors causing UDC were evaluated graphically. Eleven main root causes were responsible for 87.5% of the issues encountered. These were examiner time conflict, student time conflict, insufficient number of examiners, supervisor time conflict, grade statistics, insufficient number of exam assistants, reporting of results, material archiving, unfamiliarity with the process, uncooperative patients, and feedback (Fig. 3).
To find solutions for the root causes, we reviewed and analyzed the process map in a brainstorming session, to find intelligence technology (IT)-supported ways to improve activities (Fig. 4). We established an App-based e-training platform using the lean thinking in order to improve resident training assessment. To be specific, we used natural language processing (NLP) technology [23,24,25,26] to extract features and convert them to a normalized data structure. The department of education provided training on how to organize the resident assessment.
As we previously reported , NLP algorithm in the current study is mainly used after assessment and serves the feedback stage (Fig. 4). Taking a resident’s inadequate grasp of the differential diagnosis of systemic lupus erythematosus in the exam as an example, the XUEYIKU app divided the text of a teacher’s evaluation into words in a Chinese word segmentation process, and extracted the keywords “systemic lupus erythematosus” and “differential diagnosis” by using the semantic analysis of NLP. Specifically, abbreviation substitution, synonym replacement, and text similarity calculation may also be used in information retrieval, machine translation, text mining etc., according to the teacher’s actual evaluation. The student side of the app matches training tasks with the same theme in the test question bank according to the keywords extracted and sends them to students for intensive learning.
The measured main root causes related to multiple aspects in the wastes framework (Table 1). Time conflicts of examiners, students, and supervisors together accounted for 40.8% (281/688) of the main causes. These causes led to waiting, over-processing and motion. The new App-based e-training platform supports one-click notification, educator database memory and automatic student rotation schedule-matching. Functions to support grade statistics, reporting and feedback were developed to save waiting, and have solved 25.4% (175/688) of the main causes. Text conversion of voice input has also been developed to save time and provide more feedback.
Insufficient examiners and exam assistants together accounted for 18.5% (127/688) of the main causes. These causes showed under-used talent in resident assessment. The IT-supported online platform included both information and technology talents, and it has optimized the whole process to assist the organization of assessment.
Archiving accounted for 5.7% (39/688) of the main causes. The app was designed to file all trace and assessment forms, grades and feedback. It has therefore helped to solve the inventory problem of material archiving. Standard operation procedure (SOP) and communication skills were both highlighted in the training to help reduce the problems of process unfamiliarity (5.2%, 36/688) and uncooperative patients (4.4%, 30/688). Step-by-step activities are also embedded in the app to aid the completion of the whole process.
The study team verified the improvements and benefits of the newly developed app continuously. They also organized feedback panels and discussions over an interval of 2–4 weeks. An SOP was developed, including a process control, and this was installed and revised to take into account field observations. Control charts were used to monitor the process, and departments with high rates of UDC were prompted about SOP use by the educational management committee.
Evaluation of the process established
An online survey was conducted to investigate the effect of introducing the optimized SRT assessment process. We compared the rate of UDC and the satisfaction before and after the use of the App-based e-training platform using the Wilcoxon matched-pairs signed-ranks test. All data was analyzed using Stata version 15.0. All tests were two-sided and were considered statistically significant if P < 0.05.
This study was reviewed and approved by the Peking University Third Hospital Medical Science Research Ethics Committee (No. IRB00006761-M2022063). All methods were carried out in accordance with relevant guidelines and regulations. Informed consent was obtained from all participants.