Document text
AD
Award Number: W81XWH-07-C-01 15
TITLE: Department of Combat Medic Training - Technology Enhancement
PRINCIPAL INVESTIGATOR: PAWAN SINGH
CONTRACTING ORGANIZATION: Deloitte Consulting LLP
Alexandria, VA 22314
REPORT DATE: April 2011
TYPE OF REPORT: Final
PREPARED FOR: U.S. Army Medical Research and Materiel Command
Fort Detrick, Maryland 21702-5012
DISTRIBUTION STATEMENT: Approved for public release; distribution unlimited
The views, opinions and/or findings contained in this report are those of the author(s) and
should not be construed as an official Department of the Army position, policy or decision
unless so designated by other documentation.
REPORT DOCUMENTATION PAGE
Form Approved
OMB No. 0704-0188
Public reporting burden for this collection of information is estimated to average 1 hour per response, including the time for reviewing instructions, searching existing data sources, gathering and maintaining the
data needed, and completing and reviewing this collection of information. Send comments regarding this burden estimate or any other aspect of this collection of information, including suggestions for reducing
this burden to Department of Defense, Washington Headquarters Services, Directorate for Information Operations and Reports (0704-0188), 1215 Jefferson Davis Highway, Suite 1204, Arlington, VA 22202-
4302. Respondents should be aware that notwithstanding any other provision of law, no person shall be subject to any penalty for failing to comply with a collection of information if it does not display a currently
valid OMB control number. PLEASE DO NOT RETURN YOUR FORM TO THE ABOVE ADDRESS.
1 . REPORT DATE (DD-MM-YYYY) 2. REPORT TYPE
01-04-2011 Final
3. DATES COVERED (From - To)
25 SEP 2007 -31 MAR 2011
4. TITLE AND SUBTITLE
Department of Combat Medic Training - Technology Enhancement
5a. CONTRACT NUMBER
W81XWH-07-C-01 15
5b. GRANT NUMBER
5c. PROGRAM ELEMENT NUMBER
6. AUTHOR(S)
PAWAN SINGH
E-Mail: [email protected]
5d. PROJECT NUMBER
5e. TASK NUMBER
5f. WORK UNIT NUMBER
7. PERFORMING ORGANIZATION NAME(S) AND ADDRESS(ES)
Deloitte Consulting LLP
Alexandria, VA 22314
8. PERFORMING ORGANIZATION REPORT
NUMBER
9. SPONSORING / MONITORING AGENCY NAME(S) AND ADDRESS(ES)
U.S. Army Medical Research and Materiel Command
Fort Detrick, Maryland 21702-5012
10. SPONSOR/MONITOR’S ACRONYM(S)
11. SPONSOR/MONITOR’S REPORT
NUMBER(S)
12. DISTRIBUTION / AVAILABILITY STATEMENT
Approved for Public Release; Distribution Unlimited
13. SUPPLEMENTARY NOTES
14. ABSTRACT
This final report delivers the CONOPS in compliance with the statement of work. Deloitte Consulting LLP, in support of
Department of Combat Medic Training (DCMT), conducted research, designed and developed a technical solution to digitize and
integrate forms-based testing and training documents into the 68W Training Program over a three year period. The CONOPS
document enclosed in this report fulfills the requirements for submitting documentation outlined in the final statement of work for
this research and pilot program.
15. SUBJECT TERMS
Automate Training Process, e-Forms, Paper Reduction, Tablet EMT PC, Automated Roll Call/Attendance, Real-time Exam
Scoring, wireless, indexed forms
16. SECURITY CLASSIFICATION OF:
17. LIMITATION
OF ABSTRACT
18. NUMBER
OF PAGES
19a. NAME OF RESPONSIBLE PERSON
USAMRMC
a. REPORT
u
b. ABSTRACT
u
c. THIS PAGE
u
uu
57
19b. TELEPHONE NUMBER (include area
code)
Standard Form 298 (Rev. 8-98)
Prescribed by ANSI Std. Z39.18
Final Report / Concept of Operations ( CONOPS )
DCMT TE W81 XWH-07-C-01 1 5
TABLE OF CONTENTS
INTRODUCTION . 3
BODY . 4
REPORTABLE OUTCOMES . 56
CONCLUSION . 56
REPORT INTRODUCTION
Deloitte is under contract W81XWH-07-C-01 15 Department of Combat Medic Training -
Technology Enhancement to the U.S. Army Medical Research & Materiel Command's
(USAMRMC) Telemedicine & Advanced Technology Research Center (TATRC). During the
contract period Deloitte was purchased by Deloitte. There were no project impacts due to this
action.
The Department of Combat Medicine Training Technology Enhancement (DCMT) team is
tasked with conducting the research, design, and analysis of a technical solution to digitize and
integrate forms-based testing and training documents into the Army’s 68 W Training
Program). Specific areas of interest were the digital conversion of paper to electronic forms, and
two-way secure wireless synchronization of the tablets and central servers managing the data.
Deloitte-Federal Healthcare
3 of 57
Final Report / Concept of Operations ( CONOPS )
DCMT TE W81 XWH-07-C-01 1 5
BODY
TATRC
Department of Combat Medic Training -
Technology Enhancement
Final Report
CONCEPT OF OPERATIONS
W8 1 XWH-07 -C-0 1 1 5
Final Phase II Report
15 April 2011
Deloitte.
Deloitte-Federal Healthcare
4 of 57
Final Report / Concept of Operations ( CONOPS )
DCMT TE W81 XWH-07-C-01 1 5
TABLE OF CONTENTS
1 EXECUTIVE SUMMARY . 6
1 . 1 Approach . 6
1 .2 Exemption from Research Protocol . 6
1 .3 What this report says: . 6
2 INTRODUCTION . 8
2. 1 Document Overview . 8
2.2 Project Overview . 9
3 CURRENT STATE . 11
3.1 Background . 11
3.2 Description of Current Process . 11
3 . 3 Operational Environment . 12
4 FUTURE STATE STRATEGY . 12
4. 1 Problem definition . 12
4.2 Methodology . 12
4.3 Prototype Design . 1 3
4.4 Projected Benefits . 14
5 WORKPLAN . 16
5.1 PHASE 1 . 16
5.2 Phase II . 16
6 CONCEPTS OF THE PILOT SYSTEM . 17
6. 1 DCMT Activity Diagram . 17
6.2 DCMT USE CASES . 18
6.3 DCMT Technology Layers . 18
6.4 Detailed Use Cases Aligned with Requirements . 20
6.5 Evaluate Exams . 22
6.6 Systems Administration Use Case . 23
6.7 Web Portal Use Case . 25
6.8 Application and Data Security Requirements . 27
7 SUMMARY OF THE PILOT BUILD OUT . 28
7 . 1 Pilot Hardware . 28
7.2 Software Specifications . 29
8 PILOT OPERATION AND TEST RESULTS . 30
8.1 Preliminary Testing Results . 30
8.2 December 2009 Field Testing . 45
8.3 Final Testing January 2010 . 48
9 CONCLUSION . 51
9 . 1 Strategic Outcomes . 51
9.2 Performance Results . 52
9 . 3 General Comments . 53
9 .4 Recommendations to Expand Utilizations into Other Areas . 53
10 APPENDIX A . 57
Deloitte-Federal Healthcare
5 of 57
Final Report / Concept of Operations ( CONOPS )
DCMT TE W81 XWH-07-C-01 1 5
1 Executive Summary
1.1 Approach
Deloitte is under contract W81XWH-07-C-0115 Department of Combat Medic Training -
Technology Enhancement to the U.S. Army Medical Research & Materiel Command's
(USAMRMC) Telemedicine & Advanced Technology Research Center (TATRC). The DCMT
project team created this Concept of Operations (CONOPS) document to details the entire
project’s research, design, testing and pilot efforts for integrating an automated testing solution
into the training program for combat medics.
1.2 Exemption from Research Protocol
This project was determined to be exempt from IRB protocol per Appendix
1.3 What this report says:
Section 1 - Executive Summary:
(this section)
Section 2 - Introduction:
This section of the CONOPS provides a high-level overview of the project including problem
statements and setting.
The Department of Combat Medicine Training Technology Enhancement (DCMT) team is
tasked with conducting the research, design, and analysis of a technical solution to digitize and
integrate forms-based testing and training documents into the Army’s 68 W Training
Program). The specific area of interest was the digital conversion of paper to electronic forms,
and two-way secure wireless synchronization of the tablets and central servers managing the
data.
Section 3 - Current State:
This section provides detail of the current process including the manual and automated system(s)
in use, description of environment, definitions, modes of operation, user classes and roles, and a
description of the current support environment. Analysis of these current process and system
details is described for the purpose of determining future alternatives.
The current process (at initiation of the project) was a tedious, slow manual process involving
paper-based checklists filled out by an instructor as each student performs lifesaving procedures
in a number of clinical scenarios. The completed checklist was hand-delivered by runners to a
group of evaluators in a separate location, where the exam was scored. There were several
disadvantages identified to be addressed by the project including delays in the process and the
inability for course managers to receive real-time reports of student performance. The manual
process also required a large group of soldiers to be used as runners.
Section 4 - Future State Strategy:
Deloitte-Federal Healthcare
6 of 57
Final Report / Concept of Operations ( CONOPS )
DCMT TE W81 XWH-07-C-01 1 5
This section of the DCMT CONOPS describes the case for change; provides a recommended
plan of action based on the findings of the analysis of current system. The analysis of current
system details is merged with research into the potential alternative solutions. Recommendations
were developed from this analysis and are described in this section
The specific area of interest identified was the digital conversion of paper to electronic forms,
and two-way secure wireless synchronization of a tablet device, and central servers managing the
data. Applied research and analysis of the department’s business processes and functional
requirements will be more thoroughly understood for the purpose of influencing possible new
designs of functionality that can be developed within the 68W program.
Recommendations for a tablet-based wireless solution were developed based on the following
projected benefits:
1 . Increased efficiency:
a. reduced paper use and associated storage and management cost
b. free up manpower used as runners
c. instant scoring/feedback including details for corrective assistance
d. decreased time required for overall testing process
2. Increased accuracy
a. legible capture of instructor comments
b. validation of input prior to submission
c. import of student demographics
3. Administrative simplification
a. enable automated management of performance indicators
b. web-enabled access
c. automated synchronization of TEMTPC and EMT Web Portal
d. digital data sharing capability
4. Enhanced data security
a. encryption during testing process during transfer between instructor to evaluator
b. encrypted storage
c. encryption for transfer
Section 5 - Work Plan:
This section contains a timeline showing the milestones and deliverables in sequential fashion
for both phase I and phase II.
Section 6 - Concepts of the Pilot System;
This section displays and explains UML activity diagrams and use case descriptions.
Section 7 - Summary of the Pilot Build out:
This section provides details of the components used in the pilot project. The pilot was built
around three components and their installed software:
• Tablet PC
• Laptop Console
• Central Server
Section 8 - Pilot Operation and Test Results
Deloitte-Federal Healthcare
7 of 57
Final Report / Concept of Operations ( CONOPS )
DCMT TE W81 XWH-07-C-01 1 5
This section contains findings from the pilot’s operational use gathered during the testing phase
to include a comprehensive listing of fixes, changes and enhancements identified or requested
during interaction with staff on site at the pilot.
Section 9 - Conclusion
This section outlines the measured impact of transitioning from the current state to the new
concept piloted by providing the strategy map, listing of performance results and general
comments and recommendations to address both future operations in the 68W training program
and expanding the use of this or similar technology into other areas.
The performance results measured include a significant reduction in paper documents being
stored and processed (paper forms for 10,000 exams provided per year), reduction of over 1900
man-hours per year, and a 30% reduction in overall time for testing).
2 Introduction
The “68 Whiskey” (68W - Combat Medic) military occupational specialty is the second-largest
specialty in the Army, with nearly 38,000 medics spread across the active and reserve
components. The Department of Combat Medic Training (DCMT) trains 8,000 new medics a
year, with class sizes that stretch to nearly 500 students. A new iteration of training starts every
two weeks, and at any one time, as many as 2,500 students are working their way through the
program.
This project was approved to focus on improving a specific element of the 68 W training
program- the All Skills Practical Exam (PE) required by the National Registry of Emergency
Medical Technicians (NREMT).
2.1 Document Overview
2.1 .1 DCMT CONOPS Objective
The DCMT project team created this Concept of Operations (CONOPS) document to detail the
entire project’s research, design, testing and pilot efforts for integrating an automated testing
solution into the training program for combat medics. It serves as the final report for the project,
and as a source of recommendations for application of the solution evaluated and tested by the
project team.
2.1.2 Audience
• MHS end users
• Potential partners and stakeholders for follow-on phases
• Developers
• Medical Treatment Facilities
• Training Site Cadre'
Deloitte-Federal Healthcare
8 of 57
Final Report / Concept of Operations ( CONOPS )
DCMT TE W81 XWH-07-C-01 1 5
2.1.3 Expected evolution of the CONOPS
The DCMT CONOPS will provide the background and fundamental concepts as basis for next
steps for MHS implementation/sustainment activities. This is the final report for the project, and
as such, this document is not scheduled for future revision by the current project team, however
it is possible that changes could take place during the eventual system development process that
will require subsequent revisions, including potential change(s) in policy or overall system
requirements.
2.1.4 Revision History
Version
Description of Change
Author
Date
1
Final Report
Raman Singh
4/15/2011
2.2 Project Overview
The DCMT technology enhancement project was tasked with research, design, and development
of a technical solution that provides digitized forms and integrated testing and training
documents for the 68W training environment. Solution design resulted in a pilot, which was
adopted by DCMT and continues to be utilized. Hardware purchased was set up in the
operational environment and will remain on site for use after project termination.
2.2.1 Problem Summary
An inefficient manual paper-based process was being used to evaluate performance of students
taking a practical exam. Paper forms required storage space and the storage and retrieval process
was labor intensive. Transport of the forms during testing was also manual, introducing potential
for loss or breach of data security. The process required manual entry of demographic data,
leaving room for error in reading hand-written entries, and requiring time to validate
completeness of data entry. Management of performance indicators for the instructors
themselves was also manual.
2.2.2 Approach
This project included:
• Performance of current state analysis and industry research to identify industry best
practices and trends to determine suitable solution set to provide increased efficiency for
the evaluation process
• A pilot of the solution including design, development, testing and deployment of the
solution set in an operational setting
• Compilation of recommended options for reaching the desired future state including
recommended situations that could benefit from the project’s findings
• Action on findings within scope and budget
• Final report
Deloitte-Federal Healthcare
9 of 57
Final Report / Concept of Operations ( CONOPS )
DCMT TE W81 XWH-07-C-01 1 5
2.2.3 DCMT Technical Enhancement Project Objective
During this study, the Deloitte team, working with the Department of Combat Medic Training
(DCMT), conducted research necessary to design and develop a technical solution to digitize and
integrate forms-based testing and training documents into the 68W Training Program.
This research was focused on a sample group of students and instructor(s), not the service
members themselves (This is not a human study). The population was determined from students
and instructors who participate in 68W exercises. The specific areas of interest were the digital
conversion of paper to electronic forms, and two-way secure wireless synchronization of the
tablet, and central servers managing the data. Applied research and analysis of the department’s
business processes and functional requirements will be more thoroughly understood for the
purpose of influencing possible new designs of functionality that can be developed within the
68W program or in similar environments.
2.2.4 DCMT Technical Enhancement Project Scope
The scope of phase I of this project was to conduct a study to define and test forms automation
and process re-design. Phase II expanded the initial findings to demonstrate the concept in a
pilot that includes providing hardware and training necessary to automate practical exams for the
68W training program.
The project:
• Provided 70 programmed tablets to DCMT to be used by instructor/evaluators to enter
notes on student progress as they work through a simulated event
• Designed, loaded and tested customized forms software on the tablets to emulate forms
being used for testing to allow simplified data entry
• Provided 1 laptop with the console application for use by the course coordinator to view
and manage test results, reports, manage testing process to include instructor performance
measures
• Provided 2 laptops with server/storage capacity for automated synchronized back-up and
additional data management capability
• Designed and implemented a web portal allowing the data to flow between elements
within the network established
• Provided training to staff for use of the components
• Evaluated the use of the components in a test environment, then an operational
environment
• Collected and acted upon feedback from participants in the pilot to maximize the utility
of the system
Deloitte-Federal Healthcare
10 of 57
Final Report / Concept of Operations ( CONOPS )
DCMT TE W81 XWH-07-C-01 1 5
3 Current State
3.1 Background
The training provided at DMT includes the All Skills Practical Exam (PE) required by the
National Registry of Emergency Medical Technicians (NREMT). The EMT PE was established
to ensure each soldier at the Department of Combat Medic Training is competent in thirteen
separate skills trained from the 68W EMT-Basic curriculum. The PE consists of six skill
stations; five mandatory stations and one random skill station.
3.2 Description of Current Process
The current process for the PE was a tedious, slow manual process:
Each student was required to perform lifesaving procedures in a
number of clinical scenarios. Using paper-based checklists, an
instructor evaluated each student's performance in a scenario. The
instructor begins the PE by reading the scenario to each student, and
then each student performs timed simulated lifesaving procedures on a
fellow student who is acting as the casualty.
At the end of the scenario the instructor handed the graded checklist to
a runner who in turn walked this checklist to a group of evaluators in a
separate location. If the student has passed, all is well. If the student
has not passed that station, he must return later and redo that particular
clinical scenario. The student who has failed the scenario is not
provided feedback as to why he failed. He simply has to repeat the
entire scenario. The use case diagram below documents the current process.
Deloitte-Federal Healthcare
11 of 57
Final Report / Concept of Operations ( CONOPS )
DCMT TE W81 XWH-07-C-01 1 5
3.3 Operational Environment
The EMT PE skill stations are remote from the normal classroom training environment - at a
location that simulates disaster and wartime scenarios. The course evaluators that are monitoring
student progress are limited to recording progress based on a check-sheet with an optional free-
text feedback section to document notes on student behavior related to the score given. Students
are not provided feedback during the exam.
4 Future State Strategy
4. 1 Problem definition
As hundreds of students queue for the Practical Exam, there are many delays in the process and
students spend a great deal of time just waiting for their turn in the next station. The manual
method did not allow course managers to receive real-time reports of students’ performance.
Also, a large group of soldiers were used as runners to support the event, taking them away from
more meaningful work for an entire day.
4.2 Methodology
During this study, Deloitte - in support of Department of Combat Medic Training (DCMT) -
researched, designed and developed a technical solution to digitize and integrate forms-based
testing and training documents into the 68W Training Program. This research focuses on the
process flow used by instructor(s) to evaluate the performance of a sample group of students, not
the service members themselves. It is not a human
study. The population was determined from
students and instructors who participated in 68W
PEs. The specific area of interest was the digital
conversion of paper to electronic forms, and two-
way secure wireless synchronization of the Tablet
PC, and central servers managing the data.
After defining requirements and designing and
testing a prototype in phase I, a pilot was conducted
to prove the concept, with the goal of producing an
understanding of the department’s business
processes and functional requirements and to ensure
the design would support the mission in an
operational environment.
This final report and supporting document will
serve to provide a record of the results for use in
improving the efficiency and accuracy of the 6W
program and potentially apply lessons learned to
other programs as well. The projected benefits
included specific gains in efficiency, accuracy, data
security and enhanced administration capability.
Deloitte-Federal Healthcare
12 of 57
Final Report / Concept of Operations ( CONOPS )
DCMT TE W81 XWH-07-C-01 1 5
4.3 Prototype Design
Based on needs identified by DCMT, a technical architecture was conceived, and requirements
for the solution design were developed.
EMT SfTE
]
EMT SERVER
The diagram above shows the functional design of the technical architecture. The design
selected includes:
1- Tablet PCs used for data entry and transmission using secure wireless transmission
2- The Tablet EMT Program (TEMTP) - a software program developed for this testing
automation solution - resides on each of the 75 Tablet PCs programmed
3- EMT Coordinator - an application residing on a laptop station used by the course coordinator
4- EMT Server/Web Portal - web based portal providing access to data stored on the server
The instructors evaluating students taking the exam use Tablet PCs with TEMT (the automated
testing program) and transfer the data using the Tablet PCs wireless transmission capability
across a secure wireless closed network to the EMT Coordinator. A course coordinator reviews
any test failures and monitors results and performance of both students and instructors using the
EMT Coordinator processing center software, and the EMT Coordinator synchs the data with the
EMT Web Portal for storage. Once stored in the EMT Web Portal, the data can be retrieved
using an offline web portal access protocol, making the data secure and allowing re-creation of
the form as originally entered during the exam. The data can also be processed for reports and
analysis or for transmission to authorized requestors either through the internal web portal or by
downloading encrypted files to any digital storage media device.
Deloitte-Federal Healthcare
13 of 57
Final Report / Concept of Operations ( CONOPS )
DCMT TE W81 XWH-07-C-01 1 5
4.4 Projected Benefits
During the assessment of the current state, we identified several benefits to the current processes
listed in the left column and the benefit categories listed in the top row of the table below.
- ttml
Efficiency
Accuracy
Simplification
Security
Administer Exam
Evaluate Exam
Management
Reporting
Detailed benefits are described below.
4.4.1 Increased efficiency:
a. Reduced paper use and associated storage and management cost:
Storage and management of paper forms in filing systems is a major cost that is mitigated by use
of electronic systems. In the pilot the form is never printed, is stored as a digital file in a central
database which allows dynamic recreation of the instructor’s entry.
b. Free up manpower used as runners and processors:
Use of the soldiers as runners during the testing is not required for transfer of data that is now in
digital format from start to finish. This allows the soldiers to be utilized for other duties.
Processors that took the paper from the runners, scored exams and processed results to be
returned by runner to instructors are also reduced or eliminated by the automation of the process.
c. Instant scoring/feedback including details for corrective assistance:
Scoring will be tabulated instantly upon submission by the instructor evaluating the student’s
performance. The results will display instantly for the instructor to pass on the initial pass/fail
status to the student. In the event of a failure, the course training coordinator has the authority to
over-ride the results, and will utilize the automated report including any notes captured during
the evaluation in combination with other factors to make the final decision. The coordinator can
utilize the “fail report” feature to generate a list of students who have failed a portion of the
course in order to perform a review at any time - allowing more efficient processing compared to
reviewing paper forms brought by runners to a collection point for manual processing. A
coordinator can provide the student more comprehensive rationale for their failure at this point to
help them prepare to re-test.
d. Decreased time required for overall testing process:
The original process of evaluating students included two days - totaling approximately 20 hours
of time - to run the simulation and evaluate two shifts of students each day. The process
Deloitte-Federal Healthcare
14 of 57
Final Report / Concept of Operations ( CONOPS )
DCMT TE W81 XWH-07-C-01 1 5
efficiencies gained from automating testing and eliminating some of the wait times was projected
to be significant.
4.4.2 Increased accuracy:
a. Legible capture of instructor comments:
Paper forms included hand-written entry, which is eliminated with the tablet based solution. The
instructor has an option to enter notes in free text with automatic conversion to digital text
(conversion of handwriting to text) or use pre-scripted comments from a drop-down menu.
b. Validation of input prior to submission:
The tablet will not allow submission of an incomplete evaluation, and requires the instructor to
validate the content prior to submission.
c. Import of student demographics:
The automated form’s population of student demographic information for identification reduces
the potential for error in filling out the form, preventing potential loss of results due to
incomplete fields.
4.4.3 Administrative simplification:
a. Enable automated management of performance indicators:
Statistical analysis normally used to monitor the performance of both students and instructors is
automated and can be accessed using pre-loaded report formats. This allows analysis of patterns
including pass/fail rates, multiple failures at same site, relating the failures to specific factors has
potential to improve the quality of the training program. Reports are produced with a click rather
than having to transcribe and analyze paper documents into computable format.
b. Web-enabled access:
Ability to access various reports and features of the system using any computer with web access
allows flexibility for application use.
c. Automated synchronization of EMT Coordinator and EMT Web Portal:
As digital data must be protected from loss either by hardware failure or theft, The EMT
Coordinator will automatically transfer or synch all records to a central repository residing on a
local secure web server - the EMT Web Portal. This will not only free up space on the device -
allowing it to work more efficiently - but will provide a failsafe location to store all records for
all exam locations. In the event of hardware failure where data would normally be loss, a
coordinator or leadership can easily retrieve (i.e. view or print) that data from the server by
logging into a web portal. The web service will encrypt and transfer the records from TEMTPC
application to EMT Web Portal from the click of a button. Once all data has been successfully
synchronized EMT will notify the coordinator of its successful completion. This action can be
instantly verified by login into the EMT Web Portal and viewing a log of all synchronized
records.
d. Digital data sharing capability:
When data requests are received, such as requests from the National Registry, there was
extensive manual effort required to prepare and share paper-based forms. The solution designed
would allow access to be granted through EMT Web Portal if approved, or data can be easily
exported by selected criteria to a portable medium (i.e. CD or Tape). This feature can save
shipping costs as well. The data stored will allow dynamic recreation of the exact format and
content of the data entered on the screen of the tablet by the instructor.
Deloitte-Federal Healthcare
15 of 57
Final Report / Concept of Operations ( CONOPS )
DCMT TE W81 XWH-07-C-01 1 5
4.4.4 Enhanced data security:
It is essential to maintain confidentiality of each candidate’s records. All components will
provide 128-bit encryption. As data will be transmitted from component to component it will by
encrypted using a 128-bit algorithm to ensure privacy and prevent theft. Encryption applies to
three states of the data - during the initialization process, when the data is entered by instructors
and processed to include providing results, storage, and transfer from storage to a requestor.
a. Encryption during testing process during transfer between instructor to evaluator:
Data is encrypted as soon as it is submitted in the tool for transmission during evaluation.
b. Encrypted storage:
Once entered, the storage of data includes encryption to ensure that data cannot be used if stolen
or shared without approval.
c. Encryption for transfer:
The data can be transferred via the web portal or copying to storage media - remaining encrypted
for secure transfer.
5 Work Plan
The timeline below is divided into three phases: the first two were performed by Deloitte and the
last one reflects the ongoing field use of the tools and methods that were provided. It shows
major milestones, with expanded milestones/deliverables listed in the key below.
►
2007
2011
5.1 PHASE I
1. Project Start
2. Define Project Scope
3. Functional Requirements - What were the requirements needed?
4. System Requirement Specifications - How would the system meet these needs?
5. System Design
6. Prototype build and testing - Working in an office environment, test the devices.
7. Validation of system functionality
8. Phase I end
5.2 Phase II
8. Project Start
9. System Build - Build the applications and wireless/server environment
10. Certification and Accreditation
Deloitte-Federal Healthcare
16 of 57
Final Report / Concept of Operations ( CONOPS )
DCMT TE W81 XWH-07-C-01 1 5
11. System Testing & Training
12. System Delivery
13. Technical Support Plan & Documentation
14. Final Report
15. Project End
Operational Use
16. Implement Recommendations
6 Concepts of the Pilot System
The system transforms the current paper-based Practical Exam into an electronic format residing
on a Tablet PC based secure wireless network. It digitizes the forms and the flow of the data
from the Tablet PCs to the testing coordinator and finally to the DCMT central servers using a
web portal application.
6. 1 DCMT Activity Diagram
This system digitizes the testing forms, and helps to improve the efficiency of the instructional
learning and examination process as depicted in the UML Activity Diagram below.
DMCT Activity Diagram for Practical Examinations
Coordinator
Examiner
Server
Log on to EMT
Console
Import
Candidates from
Spreadsheet
Review Exam
Performance
Make Changes as
Necessary
Log on to EMT
Tablet
Examiners
Recieve
Candidate Lists
Adminster Exams
Data
synchronized
Data View ed
on Web Portal
Deloitte-Federal Healthcare
17 of 57
Final Report / Concept of Operations ( CONOPS )
DCMT TE W81 XWH-07-C-01 1 5
6.2 DCMT Use Cases
The system supports the following objectives using mobile wireless environment:
1 . Collect metrics in real-time,
2. Analyze the collected metrics,
3. Provide feedback based on the analysis, and
4. Generate reports based on the feedback to aid further improvement of the training and
testing process.
In this research, we documented the Use Cases using a standard object based UML modeling
technology IBM Rhapsody 7,5. Information and models are contained in Appendix A and B.
The Use Case models identify the interaction between the System (DCMT) and the various
Actors using that system. Each use case is then broken down into a series of system
requirements.
These system requirements will then be housed around a three tiered set of Layers
6.3 DCMT Technology Layers
The system contains three layers of technology for collection, processing and reporting.
• Layer 1: Tablet PC Forms Application to collect and review data
• Layer 2: Local Laptop Server Application to process and analyze secure digital data
• Layer 3: EMT Web Portal and Central Database Server for real time reporting and
verification
Deloitte-Federal Healthcare
18 of 57
Final Report / Concept of Operations ( CONOPS )
DCMT TE W81 XWH-07-C-01 1 5
6.3.1 Layer 1 : Tablet PC Forms Application
EMT Tablet PC Application - The EMT Tablet PC application is used by the instructors to
digitally capture data entered into an electronic version of the EP exam. The application
replicates an exact electronic copy of the 13 EP forms to provide the comfort and recognition to
the instructor of the actual paper form. The application provides validation controls so that
necessary information is collected and entered. The electronic form will automatically display
the candidate information and examiner’s name.
6.3.2 Layer 2: Local Laptop Server Application
This application will be used by the site Coordinators to manage instructors and candidates as
well as facilitate data gathering, reporting and analysis. The data submitted by instructors at the
site is stored on the local laptop server and managed by the application. The application scores
the form and inserts data into a secure database. This database contains the forms’ information,
which is used for real-time exam activity and real-time reporting. The application allows the
EMT coordinator to synchronize data stored in the local, central repository of the EMT central
server, viewed via a secure web portal.
6.3.3 Layer 3: EMT Web Portal and Central Research Data Collection
System
The EMT Coordinators and Leadership use the EMT Web Portal application to view and analyze
the data submitted by EMT sites. Data submitted by the EMT sites is synchronized from the
local EMT sites into the central EMT database server. This database server acts as the permanent
repository to store EMT related data. The EMT Web Portal application accesses the EMT central
database server to generate reports and acts as a data archiving tool as well. The EMT web portal
restricts access to those individuals who are authorized to view data.
Deloitte-Federal Healthcare
19 of 57
Final Report / Concept of Operations ( CONOPS )
DCMT TE W81 XWH-07-C-01 1 5
6.4 Detailed Use Cases Aligned with Requirements
The requirements are presented in the format I Req X.X where X reflects the tag number
assigned to each requirement..
6.4.1 Practical Exam Administration
This Use Case scope entails the actions performed by the instructor or evaluator to document the
practical exam. It consists of the two use cases of
• Select Student and Administer the Exam
• Score the Exam
There are five high level requirements that support these two cases and are shown in the diagram
below.
6.4.2 Req 1.3.SELECTION OF CANDIDATE AND EXAMINATION
1.3.1 - The system will fetch the list of scheduled candidates from the local database every day.
1.3.2 - The system will display a list of candidates scheduled for that day alone and nobody else.
1.3.3 - The evaluator will select a candidate from the list.
1.3.4 - The system will then display the list of 13 exams for the candidate, allowing access only
to those forms that were not already graded by the evaluator earlier on the same day.
1.3.5 - The system will not allow access to any forms for candidates who have failed four exams
or the same exam three times.
6.4.3 Req 1.4.SCORING OF EXAMS
1.4.1 - The system will allow the evaluator to check or uncheck responses to the questions based
on the actions performed by the candidate.
1.4.2 - The system will allow the evaluator to check or uncheck certain critical criteria based on
whether or not the candidate performed some important actions.
Deloitte-Federal Healthcare
20 of 57
Final Report / Concept of Operations ( CONOPS )
DCMT TE W81 XWH-07-C-01 1 5
6.4.4 Req 1.5.COMMENTS
1.5.1 The system will allow the evaluator to note down remarks and comments about the
candidate’s performance and a list of the actions performed by the candidate
6.4.5 Req 1.7.FORM ACCESS
1.7.1 The system will not allow a candidate to test for any particular exam with a particular
evaluator more than once.
1.7.2 The system will not allow a candidate who has failed the same test thrice or four tests on
the whole to take any further tests on the same day.
6.4.6 Req 1.8.DATA VALIDATION AND FORM SUBMISSION
1.8.1 The system will not allow an incomplete form (i.e. One where all fields have not been
filled) to be submitted.
1.8.2 The system will accept a signature of the evaluator and only then will it let the form to be
submitted.
1.8.3 The system will authenticate the evaluator’s signature with the last four digits of the
evaluator’s Social Security Number, and only then will the signature be accepted and the form
allowed to be submitted.
1.8.4 The system will allow evaluators to make changes even after the form has been signed, but
the form has to be resigned before being submitted.
1.8.5 The system will insert all submitted records into the local server’s database immediately
via a wireless connection.
1.8.6 The system will not allow evaluators to make changes to, or view submitted forms.
Deloitte-Federal Healthcare
21 of 57
Final Report / Concept of Operations ( CONOPS )
DCMT TE W81 XWH-07-C-01 1 5
6.5 Evaluate Exams
The diagram below shows the high level requirements that support the evaluation of the exam
results.
Layer 1
1.6 Pass Fail
Criteria
7f
6.5.1 Req 1.6 PASS/FAIL CRITERIA
1.6.1 - The system will consider a candidate who has a score greater than the minimum score
required to pass the skill and no checked critical criteria, to have passed the skill.
1.6.2 - Any candidate, not meeting any of the above stated requirements, is considered to have
failed the skill.
6.5.2 Req 2.3 VIEW EXAMS
2.3.1 - The application will allow the coordinator to view submitted exams (both passed and
failed) for review.
2.3.2 - The application will allow the coordinator to view the exams based on groups and teams.
2.3.3 - The application will allow the coordinator to view graded individual skills.
6.5.3 Req 2.4 OVERRIDE
2.4.1 - The application will allow the coordinator to rescore submitted exams should there be a
need.
2.4.2 - The application will allow the coordinator to submit the rescored exams only if the
candidate receives a passing grade after rescoring.
Deloitte-Federal Healthcare
22 of 57
Final Report / Concept of Operations ( CONOPS )
DCMT TE W81 XWH-07-C-01 1 5
6.5.4 Req 2.6 MANAGE APPLICATION FUNCTIONALITY
2.6.1 - The application will allow the coordinator to delete submitted forms should there be a
need.
2.6.2 The application will allow the coordinator to delete all or particular synchronized records.
6.6 Systems Administration Use Case
6.6.1 Req 1.1 AUTHENTICATION
1.1.1 - The users will be created on the local EMT Console server by the site coordinator.
1.1.2 The system will allow only authorized users (i.e. the evaluators), to log in.
1.1.3 The system will authenticate the users by verifying the username and password entered by
1.1.4 the user against the ones set in the EMT local database server.
1.1.5 The system will lock the user account if the wrong username or password is entered thrice.
1.1.6 The system will require the users to reset their passwords, every 90 days.
6.6.2 Req 1.2 APPLICATION CONFIGURATION
1.2.1 The users will be required to configure the system, the first time they login.
Deloitte-Federal Healthcare
23 of 57
Final Report / Concept of Operations ( CONOPS )
DCMT TE W81 XWH-07-C-01 1 5
1.2.2 They will need to provide the name or IP Address of the local laptop server which hosts the
local database.
6.6.3 Req 2.1 AUTHENTICATION
2.1.1 The application will have an administrator username and password, which will be provided
only to the site coordinator.
2.1.2 The system will lock the user account if the wrong username or password is entered thrice.
2.1.3 The system will require the users to reset their passwords, every 90 days.
6.6.4 Req 2.2 MANAGEMENT OF USERS AND CANDIDATES
2.2.1 The application will allow the coordinator to prepare the list of scheduled candidates for
the day, either by adding them manually or by importing from an Excel spreadsheet.
2.2.3 The application will allow the coordinator to edit the profiles of existing candidates.
2.2.4 The application will allow the coordinator to add evaluators, create usernames and
passwords for them, and edit and manage their profiles.
2.2.5 The application will allow the coordinator to add groups and teams and create and delete
mappings between them.
2.2.6 The application will allow the coordinator to assign candidates to particular groups and
teams.
6.6.5 Req 2.7 SECURE DATA
2.7.1 All data, instructors’, candidates’ and exams’, will be stored in a secure MS SQL Server
Database.
2.7.2 All wireless data flow between the tablets and the local server will use the DoD certified
encryption software “AirFortress”.
2.7.3 All data flowing between the local server and the central server will adhere to DoD
network policy.
6.6.6 Req 2.8 DATA SYNCHRONIZATION
2.8.1 The application will allow the coordinator to synchronize the records with the central
database using a web service.
2.8.2 The application will authenticate the user before synchronizing, by getting them to enter a
username and password, which was created when they registered as a user on the web portal.
2.8.3 The application will run the web service over a Secure Socket Layer (SSL) on port
443 (https).
6.6.7 Req 3.1 AUTHENTICATION
3.1.1 The web service will use the same username and password that was created by the user
when he/she registered on the web portal.
3.1.2 This username and password will be required to be provided when synchronizing records
from the local laptop server, which is the only way to access the web service.
Deloitte-Federal Healthcare
24 of 57
Final Report / Concept of Operations ( CONOPS )
DCMT TE W81 XWH-07-C-01 1 5
6.6.8 Req 3.2 DATA INSERTION
3.2.1 The web service will insert the data it receives from the local laptop server during the
synchronization process, into the central database.
3.2.2 The data in the central database will be accessed via the web portal alone and by registered
portal users alone.
6.6.9 Req 3.3 DATA SECURITY
3.3.1 The web service will be run over a Secure Socket Layer (SSL) on port 443 (https).
3.3.2 The data in the central database will be accessed via the web portal alone and by registered
portal users alone.
6.7 Web Portal Use Case
6.7.1 Req 4.1. REGISTRATION
4.1.1 The portal will allow potential users to register for accounts by providing their Name,
Social security Number and e-mail address.
4.1.2 The users will also specify the kind of user group they would like to register for like
Coordinator, Administrator or Evaluator.
4.1.3 The administrator will then review the details provided and approve or disapprove the
users.
4.1.4 The portal will send out an e-mail to the requestors, informing them about the status of
their request.
4.1.5 The e-mail will also contain the username and password, which will be auto-generated by
the portal, should the request be accepted.
Deloitte-Federal Healthcare
25 of 57
Final Report / Concept of Operations ( CONOPS )
DCMT TE W81 XWH-07-C-01 1 5
4.1.6 The portal will ask the users to change their password on the first login and later
periodically.
6.7.2 Req 4.2 USER AUTHENTICATION
4.2.1 The portal application will verify the username, password and account type, before letting
the users log in to the application.
4.2.2 The portal will allow only three types of users: Administrators, Coordinators and
Evaluators.
4.2.3 The portal will allow the administrator to manage users and their profiles, process requests
for accounts and nothing more.
4.2.4 The portal will allow coordinators to view all records, synchronized from their site alone.
4.2.5 The portal will allow evaluators to view records submitted by them alone and nothing
more.
4.2.6 The system will lock the user account if the wrong username or password is entered thrice.
4.2.7 The system will require the users to reset their password, every 90 days.
6.7.3 Req 4.3 REPORTS AND RECORDS
4.3.1 The portal will allow the coordinator to view individual exam-based reports.
4.3.2 The portal will allow the coordinator to view rescored exams-based reports.
4.3.3 The portal will allow the coordinator to view evaluator-based reports.
4.3.4 The portal will allow the coordinators to view individual exams submitted from their site
and also exams that were rescored by them.
4.3.5 The portal will allow the evaluators to view exams and reports pertaining to exams
submitted by them alone, and evaluator-based reports pertaining to them alone.
4.3.6 The portal will not allow the evaluators to view any rescored exam or rescored exams
report.
6.7.4 Req 4.4 DATA ARCHIVING
4.4.1 The portal will serve as a central repository for all data collected using the Local Laptop
Server application and the Tablet PC application.
4.4.2 The portal will allow for coordinators or leadership to retrieve any exam, transfer exam
data to a portable media or print a replica of the exam.
6.7.5 Req 4.5 DATA ACCESS:
4.5.1 The portal will allow the coordinators to view individual exams submitted from their site
and also exams that were rescored by them.
4.5.2 The portal will allow the evaluators to view exams and reports pertaining to exams
submitted by them alone, and evaluator-based reports pertaining to them alone.
4.5.3 The portal will not allow the evaluators to view any rescored exam or rescored exams
report.
Deloitte-Federal Healthcare
26 of 57
Final Report / Concept of Operations ( CONOPS )
DCMT TE W81 XWH-07-C-01 1 5
6.8 Application and Data Security Requirements
6.8.1 Req 5.1 SECURE AUTHENTICATION
5.1.1 The portal will automatically generate the username and password for the users by using
their first name, last name and social security number.
5.1.2 The portal will create a password based on the following rules:
5. 1.2.1 Password will have a minimum length of 10 characters
5. 1.2.2 Password will contain at least 2 each of upper-case alphabets, lower-case alphabets,
numbers and special characters.
5.1.3 The portal will request the user to change the password on the first login and again
periodically.
5.1.4 The access to Tablet PC and EMT Console will be managed by the DoD Network Policy.
5.1.5 The users for the Tablet PC and EMT Console applications will be created on the EMT
Console server.
5.1.6 The same password rules will apply to users created for the local EMT Console server
application and the Tablet PC application.
6.8.2 Req 5.2 DATABASE AUTHENTICATION
5.2.1 Any access to the database will be authenticated by a username and password which will be
provided by the application once the user has been authenticated.
6.8.3 Req 5.3 DATA TRANSMISSION SECURITY
5.3.1 The data being transmitted by the wireless network will be protected by the DoD approved
encryption software “AirFortress”, available at the site of deployment.
5.3.2 The data being transmitted by the wired network will be protected by the security
mechanisms available on the DoD network.
Deloitte-Federal Healthcare
27 of 57
Final Report / Concept of Operations ( CONOPS )
DCMT TE W81 XWH-07-C-01 1 5
7 Summary of the Pilot Build Out
The Pilot project was built around three components and their installed software:
• Tablet PC
• Laptop Console
• Central Server
7.1 Pilot Hardware
7.1.1 TABLET PC Specifications
•Brand: Fujitsu
•Model Number: Tablet PC
•Processor: INTEL CORE2 DUO
Ultra Low Voltage U7600
•Screen: 12. 1" XGA TFT with
indoor/outdoor display with wide
viewing angles
40 GB S-ATA 150,
•Hard Drive
5400 rpm hard drive
•Memory:
memory (1 GB x 1)
•Main battery:
•Other:
multinationals 56K4 V.90 modem and
Gigabit Ethernet LAN Integrated Intel®
PRO/Wireless 3945 ABG Network Connection
(Tri-mode 802.11a/b/g) Integrated Fingerprint
Sensor, embedded TPM, dedicated Smart Card slot, Fujitsu Security Application Panel
1 GB DDR2 667 MHz
Lithium ion (9-cell)
Integrated
7.1 .2 Local Laptop Server Specifications
Local Laptop Server
•Brand: Any
•Model Number: Any
•Processor: Intel® Core™ 2 Duo
processor (2.4GHz 800MHz 3MBL2)
•Operating System: Windows XP Professional
•Screen: 14.1 SXGA+ TFT
•Hard Drive 200 GB Hard Disk Drive,
5400rpm with Disk Encryption
•Memory: 2 GB RAM
•Battery: 9 cell Li-Ion Battery
•Keyboard: Keyboard US English
• Optical device: DVD Recordable 8x Max Dual Layer, Ultrabay Slim
• Pointing Device: UltraNav (TrackPoint and TouchPad)
• Expansion slots: PC Card Slot & Smart Card Slot
Deloitte-Federal Healthcare
28 of 57
Final Report / Concept of Operations ( CONOPS )
DCMT TE W81 XWH-07-C-01 1 5
Wireless Card: 802. 1 la/b/g/n Wi-Fi wireless LAN card
Power cord: Country Pack North America
Optical device:
Pointing Device:
Expansion slots:
Power cord:
7.1.3 Central Server Specifications
Central Server
•Brand: Any
•Model Number: Any
•Processor: Intel® Core™ 2 Duo
processor T8300 (2.4GHz 800MHz
3MBL2)
•Operating System: Windows Server 2003
•Hard Drive 200GB Hard Disk Drive
•Memory: 2 GB * 2
DVD Recordable 8x Max Dual Layer
UltraNav (TrackPoint and TouchPad)
N/A
Country Pack North America
7.2 Software Specifications
7.2.1 TABLET PC Software
• The application will require Microsoft .Net® Framework v 3.5.
7.2.2 Local Laptop Server Software
• The application will require Microsoft .Net® Framework v 3.5.
• The application will require Microsoft SQL Server® 2005 with reporting services tools.
• The application will require Microsoft Internet Information Services® (IIS®) v 6.0.
• The application will require Microsoft Web Services Enhancements® (WSE®) v 3.0.
• The application will require Microsoft Office® 2003 or 2007 (preferably 2007) to be
installed on the local laptop server and the systems running the web portal to be able to export
reports to an MS Excel® spreadsheet.
7.2.3 Central Server Software
• The application will require Microsoft Internet Explorer® (IE®) v 6.0 or later.
Deloitte-Federal Healthcare
29 of 57
Final Report / Concept of Operations ( CONOPS )
DCMT TE W81 XWH-07-C-01 1 5
8 Pilot Operation and Test Results
Using the requirements from Section 7&8, Deloitte built a pilot DCMT system. Many additions
to functionality were added during this phase with a compiled functionality list in the subsections
that follow.
Pilot systems testing for compliance with the requirements were performed in the laboratory in
April 2008. Work continued on the project until January 2010 where final field testing was
conducted using actual subjects. We will describe the high level testing results in this section
with complete testing results provided in Appendix C.
8. 1 Preliminary Testing Results
System preliminary testing occurred over several days in late March 2008.
Second test March 2009
Final testing January 2010
8.1 .1 Evaluator Tablet PC Functionality
The table below shows the eventual requirements created, the testing action and the expected
system responses.
Requirements
Description
Action
Expected Results
user enters his
credentials to login
into the tablet PC
user enters his/her correct
User Id and password
The system authenticates the user based on
his/her credentials against the EMT database
and displays the pending Candidate list for the
current set of exam days
user enters incorrect
credentials
user enters incorrect User Id
and password
The system prompts the user with message
"Please enter correct user name and
password"
List of pending
candidates is
displayed
Once the user credentials
are verified and he/she is
logged in, the system
displays the list of pending
candidates who are
scheduled for the current set
of exam dates.
System displays the scheduled candidates for
the current set of exam dates. This list is sorted
alphabetically based on the last name of the
candidates.
user clicks on the
Update Profile button
to be able to change
his/her name, SSN,
Password and PIN
The user clicks on the
Update Profile button
Another window opens up with the First and
Last names and the Username and fields to
update the password and PIN.
User changes name,
password and/or PIN
The user changes his/her
First and/or Last name
and/or enters correct old
password and new
password and/or correct old
PIN and new PIN, which
have not been used in any of
the last 10 password/PIN
changes, twice(once as
System displays a message
"Name/Password/PIN has been successfully
updated"
Deloitte-Federal Healthcare
30 of 57
Final Report / Concept of Operations ( CONOPS )
DCMT TE W81 XWH-07-C-01 1 5
confirmation) and clicks on
the Update Profile button
User enters incorrect
old Password/PIN
The user enters an incorrect
old Password and/or PIN
System prompts the user with message
"Incorrect old password/PIN entered"
User enters used
new Password/PIN
The user enters a Password
and/or PIN that has been
used in any of the 10
previous password/PIN
changes
System prompts the user with message "New
password/PIN cannot have been used in any of
the previous 10 attempts"
User enters incorrect
format for new
password
The user enters a password
that contains less than 10
characters or does not
contain at least two of:
special characters, numbers,
uppercase letters, lowercase
letters
System prompts the user with message
"Incorrect password format"
User enters the name
or part of it in the
Search by Name box
and clicks on the
Search button
User enters the soldier's
name or part of it in the
search by name text box and
clicks on the Search By
Name button
System filters the candidate list to show only
those patients whose names either partly or
completely match the search string
User enters the
required SSN using
the number pad and
clicks the Search
button
User selects the candidate's
SSN using the number pad
that pops-up when the user
clicks on the Search by SSN
field. He/She then clicks on
the Search by SSN button.
System filters the candidate list to show only
the patient whose SSN matches the entered
string.
User selects the
required candidate
from the candidate
list and clicks on
LOAD FORM
User selects the required
candidate's name from the
Candidate list and clicks on
the VIEW EXAM button
System displays a confirmation page with the
candidate's name in big, bold letters and a
YES and NO button.
User clicks YES on
the confirmation
page
User confirms that the name
of the candidate and if it
matches, clicks on the YES
button
System loads the exams list for the candidate
with the passed & failed exams marked
appropriately
User clicks NO on
the confirmation
page
User confirms that the name
of the candidate and if it
does not match, clicks on
the NO button
System loads the candidates list for the current
set of exam dates again
List of all 13 exams
is displayed
Once the user verifies the
name of the candidate, a list
of 13 exams is displayed
The list shows all 13 exams with updated
statuses for each exam. The passed exams
are grayed out and disabled with a tick mark
before the exam name. The failed exams are
displayed with a red cross mark before the
name of the exam.
Selection of an exam
user select an exam for the
candidate and presses the
exam button/link
An exam is selected and the form for the exam
is displayed
selection of a failed
exam by the same
examiner
user selects a failed exam
for a candidate who was
earlier failed in the same
exam by the same examiner
System displays a message "Please ask the
candidate to go to a different examiner for this
exam"
selection of a failed
exam by a different
user selects a failed exam
for a candidate who was
System displays a message "Candidate has
failed this exam <X> times". When the user
Deloitte-Federal Healthcare
31 of 57
Final Report / Concept of Operations ( CONOPS )
DCMT TE W81 XWH-07-C-01 1 5
examiner
earlier failed in the same
exam by a different
examiner
clicks OK, it opens the exam form.
selection of an un¬
counseled failed
exam by a different
examiner
user selects a failed exam
for a candidate who was
earlier failed in the same
exam by a different
examiner and the candidate
has not yet been counseled
by the coordinator for the
failed exam
System displays a message "Candidate has
not yet been counseled for the exam. Please
ask the candidate to go to the coordinator to
complete counseling". The user clicks on OK
and the System loads the list of candidates
again.
exam form is loaded
The chosen exam form is
loaded
System loads the corresponding exam form.
The name of the exam is displayed on top of
the form. The name of the candidate, the name
of the evaluator and the exam date are
displayed too. The timer is displayed with the
start time button alone enabled. Also based on
the attempt number, a label saying RETEST
for the 2nd attempt and RETEACH for the 3rd
and final attempt is also displayed on top. The
total score is displayed at the bottom of the
questions list. The instructions buttons, the
scenario buttons (where applicable) and the
comments buttons are also enabled. There
should be a signature button at the bottom(on
the 2nd page on two page exams)
The user chooses to
view the instructions
for examiner
The user clicks on the
"Instructions for examiner"
button
The system displays the instructions for the
examiner in a full screen window with a close
button on top. This window cannot be resized.
The user chooses to
view the instructions
for candidate
The user clicks on the
"Instructions for candidate
button
The system displays the instructions for the
candidate in a full screen window with a close
button on top. This window cannot be resized.
the user chooses to
view a scenario for
the exam
The user chooses a scenario
and clicks on the button with
the corresponding number.
The scenario is loaded in a pop-up window.
The window always stays on top and it is
possible to move it around and resize it. The
corresponding scenario button remains
highlighted on either page. The scenario
window is hidden whenever a message is
popped-up for the user, until the user responds
to the message.
The user chooses to
cancel out of the
form
The user chooses to cancel
the form if he has selected
the wrong form by pressing
the cancel; button on the
top.
The system prompts the user to confirm "Do
you want to exit the exam?” When the user
clicks "yes" the form is closed and the user is
returned to the exams list for the candidate.
user is able to start
the timer
The user choose to start the
timer manually by pressing
the "start time" button
The timer starts to track the time taken to
complete the examination. The start time
button is disabled. The pause time and stop
time buttons are enabled.
user is able to start
the timer on either
page
The user choose to start the
timer manually by pressing
the "start time" button on
either page
The timer starts to track the time taken to
complete the examination. The start time
button is disabled. The pause time and stop
time buttons are enabled. This is reflected on
the other page too.
user is able to pause
The user clicks on the
The timer stops. The PAUSE TIME button is
Deloitte-Federal Healthcare
32 of 57
Final Report / Concept of Operations ( CONOPS )
DCMT TE W81 XWH-07-C-01 1 5
the timer
PAUSE TIME button and the
timer stops ticking
hidden and the RESUME TIME is visible. The
STOP TIME is still enabled.
user is able to pause
the timer on either
page
The user clicks on the
PAUSE TIME button on
either page and the timer
stops ticking
The timer stops. The PAUSE TIME button is
hidden and the RESUME TIME is visible. The
STOP TIME is still enabled. This is reflected on
the other page too.
user is able to
resume the timer
The user clicks on the
RESUME TIME button and
the timer stops ticking
The timer starts again where it was paused.
The RESUME TIME button is hidden and the
PAUSETIME is visible. The STOP TIME is still
enabled.
user is able to
resume the timer on
either page
The user clicks on the
RESUME TIME button on
either page and the timer
stops ticking
The timer starts again where it was paused.
The RESUME TIME button is hidden and the
PAUSETIME is visible. The STOP TIME is still
enabled. This is reflected on the other page
too.
user is able to stop
the timer manually
The user presses the "Stop
Timer" button when exam is
done.
The timer stops. All timer controls are disabled
user is able to stop
the timer manually
on either page
The user presses the "Stop
Timer" button on either page
when exam is done.
The timer stops. All timer controls are disabled.
This is reflected on the other page too.
user grades the
candidate for an
exam
The user starts grading the
candidate for the selected
exam.
The user is able to select the radio buttons to
mark the candidate’s response to a question.
Automatic
calculation of
candidates score on
an exam
As user keeps on marking
the Y/N answers. The
system automatically adds a
1 mark for each question
answered as Y. The system
subtracts 1 mark for each
question answered as N.
The marks for the candidate are automatically
calculated and displayed. Refer "Quantitative
Statistics" tab to see the minimum passing
score required to PASS the examination.
user selects the
check box for a
"Critical criteria"
user select the check box for
a critical criteria question
The critical criteria check box gets checked
For the forms which
span over multiple
pages, a "Next"
button is available on
the first page
If exam got multiple pages,
the evaluator clicks the
"next" button to move to the
next page of the form
The next page of the form is displayed for the
user to grade the candidate on. The previously
graded questions are correctly populated.
For the forms which
span over multiple
pages, a "Back"
button is available on
the 2nd page
If exam got multiple pages,
the evaluator clicks the
"back" button to move to the
previous page of the form
The previous page of the form is displayed for
the user to grade the candidate on. The
previously graded questions are correctly
populated.
user is allowed to
enter/see his
comments
user clicks on VIEW/ADD
Comments and a pop up text
box opens
The user is able to enter in his remarks by
either appending to the existing comments or
replacing the existing comments or by adding
new comments. He/she can also view
comments he/she entered previously for the
same exam.
user is able to enter
long comments
user enters really long
comments and these are
accepted
System allows users to enter comments up to
1 000 characters
user tries to submit
an incomplete form
user clicks on submit button
to submit an incomplete form
The system prompts the user to complete the
form before submitting it. The non-completed
Deloitte-Federal Healthcare
33 of 57
Final Report / Concept of Operations ( CONOPS )
DCMT TE W81 XWH-07-C-01 1 5
question shall displayed in RED
user tries to submit
an form without
signing
user clicks on submit button
without signing the form
The system prompts the user for signing the
form before submitting it.
User enters the
wrong PIN while
signing the form
The user signs the form and
clicks submit.
The system prompts the user to enter the
correct PIN
User enters the
correct to sign the
form
The user signs the form and
clicks submit.
The following message is displayed, "Digitally
Signed By: " followed by the name of the user
followed by the time and date of signing, in the
Signature Box.
user completes the
grading of the exam
by signing the form
but needs to revise
the grading
The user signs the form but
decides to re-grade the form.
So he clicks on any of the
questions to re-grade
The system clears the Digital Signature and
form is available for re-grading
user completes the
grading of the exam
by signing the form
and tries to submit
after digitally signing
the form with his/her
PIN
The user signs the form by
entering his/her PIN and
clicks submit.
The form is submitted and the following
message is displayed - "Exam Details
successfully submitted". Once the user clicks
on the OK button, the form is closed and the
user is returned to the candidate list. Any open
scenario pop-ups are also closed. The user is
returned to the candidate list.
user completes the
grading of the exam
by signing the form
and tries to submit
after digitally signing
the form with his/her
PIN
The user signs the form by
entering his/her PIN and
clicks submit.
If the wireless connection is lost, the same is
displayed as a message and the form is not
submitted or closed. Scenario windows are not
closed.
User clicks on sign
or submit
User clicks on the sign or
submit form buttons.
System prompts the user with the message -
"Timer will automatically be stopped now".
Once the user clicks OK, the timer is stopped
and all timer controls are disabled.
All candidates who
have either passed
or failed the entire
ALL SKILLS
EXAMINATION will
be removed from the
candidate list
Any candidate who has
passed 5 mandatory and 1
random exams or failed the
same exam thrice or failed 4
different exams is removed
from the candidate list. The
candidate list only contains
Pending candidates
System displays correct data on the list.
User enters the
wrong PIN while
signing the form
The user signs the form and
clicks submit.
The system prompts the user to enter the
correct PIN
User enters the
correct to sign the
form
The user signs the form and
clicks submit.
The following message is displayed, "Digitally
Signed By: " followed by the name of the user
followed by the time and date of signing, in the
Signature Box.
user completes the
grading of the exam
by signing the form
but needs to revise
the grading
The user signs the form but
decides to re-grade the form.
So he clicks on any of the
questions to re-grade
The system clears the Digital Signature and
form is available for re-grading
user completes the
The user signs the form by
The form is submitted and the following
Deloitte-Federal Healthcare
34 of 57
Final Report / Concept of Operations ( CONOPS )
DCMT TE W81 XWH-07-C-01 1 5
grading of the exam
by signing the form
and tries to submit
after digitally signing
the form with his/her
PIN
entering his/her PIN and
clicks submit.
message is displayed - "Exam Details
successfully submitted". Once the user clicks
on the OK button, the form is closed and the
user is returned to the candidate list. Any open
scenario pop-ups are also closed. The user is
returned to the candidate list.
user completes the
grading of the exam
by signing the form
and tries to submit
after digitally signing
the form with his/her
PIN
The user signs the form by
entering his/her PIN and
clicks submit.
If the wireless connection is lost, the same is
displayed as a message and the form is not
submitted or closed. Scenario windows are not
closed.
User clicks on sign
or submit
User clicks on the sign or
submit form buttons.
System prompts the user with the message -
"Timer will automatically be stopped now".
Once the user clicks OK, the timer is stopped
and all timer controls are disabled.
All candidates who
have either passed
or failed the entire
ALL SKILLS
EXAMINATION will
be removed from the
candidate list
Any candidate who has
passed 5 mandatory and 1
random exams or failed the
same exam thrice or failed 4
different exams is removed
from the candidate list. The
candidate list only contains
Pending candidates
System displays correct data on the list.
8.1.2 Examiner Portal Functionality
The table below shows the eventual requirements created, the testing action and the expected
system responses.
Requirements
Description
Action
Expected Results
Examiner Login
Enter the User Id and password and hit
submit tab
Gives Examiner access to the
portal and brings to the "Home"
screen of the portal
View Records by
Date Range
In From and To tabs select the dates as
desired to view the records and hit "Submit"
Lists all the exams reviewed in
the specified time line
Export the records to
Excel
Select the "Export to Excel" link in the top
right corner
Open the records in an excel file
View and Print exam
given by the student
Select the exam intended to view and click
on “Print Exam" icon. When the exam
opens select File>Print
Prints the selected exam
View Records by
Month and Year
In Month tab select the month desired for
viewing. In Year tab select the year
Lists all the exams reviewed in
the specified month and year
Export the records to
Excel
Select the "Export to Excel" link in the top
right corner
Open the records in an excel file
View and Print exam
given by the student
Select the exam intended to view and click
on “Print Exam" icon. When the exam
opens select File>Print
Prints the selected exam
View Records by
Date
Select the specific date in the "Date" tab as
required
Lists all the exams reviewed by
the examiner on that specific
date
Deloitte-Federal Healthcare
35 of 57
Final Report / Concept of Operations ( CONOPS )
DCMT TE W81 XWH-07-C-01 1 5
Export the records to
Excel
Select the "Export to Excel" link in the top
right corner
Open the records in an excel file
View and Print exam
given by the student
Select the exam intended to view and click
on “Print Exam" icon. When the exam
opens select File>Print
Prints the selected exam
Search Record by
SSN
Enter the SSN # searching for in the "SSN”
tab and Last Name of the student in the
"Last Name" tab
Lists the records
Export the records to
Excel
Select the "Export to Excel" link in the top
right corner
Open the records in an excel file
View and Print exam
given by the student
Select the exam intended to view and click
on “Print Exam" icon. When the exam
opens select File>Print
Prints the selected exam
Sort Records by
exams passed
The green "Passed" radio button is selected
Lists all the exams that were
passed by the students
administered by the instructor
Sort Records by
exams failed
The red "Failed" radio button is selected
Lists all the exams that were
failed by the students
administered by the instructor
Sort Records by
Exam Name
In the drop down menu select the exam
name for which want to see the desired
results
Lists all the exams given by the
instructor with the selected exam
name
Export records into
Excel
Select the "Export to Excel" link in the top
right corner and select "Open"
Open the records in an excel
Exam Centers Report
based on date range
Select Reports from left menu and then
select Exam Centers Report on the right. In
From and To tabs select the dates as
desired to view the reports for. Then hit
Submit and select the exam for which need
to view the results
Displays the 3D graph with the
exam center report value
Export the Report to
Excel
Select Export to Excel link on top right
Opens the report in Excel file
Export the Report to
PDF
Select Export to PDF link on top right
Opens the report in .pdf file
The coordinator adds
a new examiner
The coordinator selects Manage Examiner
icon and selects Add Examiner link and fills
in the credentials Username, Password,
First Name, Middle Name, and Last Name
New examiner is added to the
portal
The coordinator
deletes examiner
The coordinator selects the examiner he
wishes to delete and clicks on the red
Delete User Account icon
The examiner account is deleted
from the portal
The coordinator adds
a commander
The coordinator selects Manage
Commander and selects Add Commander
and fills in the credentials Username,
Password, First Name, Middle Name, and
Last Name
New commander is added to the
portal
The coordinator
deletes commander
user account
The coordinator selects the commander
from the displayed list he needs to delete
and clicks on the red Delete User Account
icon. The system prompts for logging
password. Enter admin password
The commander account is
deleted
The coordinator adds
a site coordinator
The coordinator selects Manage Site
Coordinators and selects Add Site
Coordinator and fills in Username,
Password, First Name, Middle Name, and
New site coordinator is added
Deloitte-Federal Healthcare
36 of 57
Final Report / Concept of Operations ( CONOPS )
DCMT TE W81 XWH-07-C-01 1 5
Last Name
The coordinator
deletes the site
coordinator user
account
The coordinator selects the site coordinator
from the displayed list he needs to delete
and clicks on the red Delete User Account
icon
The site coordinator user profile
is deleted from the portal
The coordinator adds
a admin to the portal
The coordinator selects Manage
Administrators and selects Add Admin and
fills in Username, Password, First Name,
Middle Name, and Last Name
New administrator is added to
the portal
The coordinator
deletes the
administrator deletes
user account from
the portal
The coordinator selects the administrator
from the displayed list he needs to delete
and clicks on the red Delete User Account
icon
The administrator user profile is
deleted from the portal
Change Account
Password
Select Updater Account from the left menu.
Type the new password in Password tab
and retype the password in Confirm
Password tab
Changes the user password
associated with the account
Logging out of the
Portal
Select Logout from the left menu
Logs out of the portal
8.1.3 Server Console Functionality
The table below shows the eventual requirements created, the testing action and the expected
system responses.
Requirements
Description
Action
Expected Results
The site administrator
logs into the console
with invalid
credentials
The administrator enter incorrect User Id
and password
The system prompts the user with
message "Account inactive. Please
contact administrator"
The site administrator
logs into the console
with correct
credentials
The administrator enter correct User Id
and password
The administrator is logged in
Administrator logs
out
The administrator selects "Manage
users->logout"
The administrator is logged out
Administrator exits
the application
The administrator selects "Manage
users->exit"
The system logs the administrator
out and exits the application.
The administrator
imports the
scheduled users for
the day
The administrator imports the user by
selecting the "Manage users->manager
candidates->import candidates" option
and selecting the import excel file
The users are successfully imported
into the system and displayed in the
view candidate section
The administrator
adds a new candidate
to schedule for the
day
The administrator clicks on add user and
enters the details(SSN, First Name,
Middle name, Last name, group name,
team name, selects the schedule date"
and clicks add
The new candidate is added to the
system for the current day.
The administrator
chooses to modify the
profile of an existing
candidate
The administrator selects a user from the
view candidate screen and clicks edit.
He then updates the necessary fields
and clicks update
The user details are modified and
persisted in the system
The administrator
adds an
The administrator selects the option for
add examiner. He fills in the necessary
The evaluator is added to the
system and the list of evaluators is
Deloitte-Federal Healthcare
37 of 57
Final Report / Concept of Operations ( CONOPS )
DCMT TE W81 XWH-07-C-01 1 5
evaluator/examiner
details(SSN, First Name, Middle name,
Last name, user name, password) and
clicks add
displayed on the manage
evaluators screen
The administrator
while adding the
evaluator enters a
user name which
already exists
The administrator adds an evaluator but
enters a user name which already exists
in the system
The system prompts the user with
message "username already exists,
please choose another user name"
The administrator
edits an evaluators
profile
The administrator selects an evaluator
under View Examiner option and clicks
edit. He changes the necessary details
and clicks update
The details of the evaluator are
modified
The administrator
deletes an evaluators
profile
The administrator selects an evaluator
under View Examiner and clicks delete
The evaluator is deleted from the
system
The administrator
adds a coordinator
The administrator selects the option for
add coordinator. He fills in the necessary
details(SSN, First Name, Middle name,
Last name, user name, password) and
clicks add
The coordinator is added to the
system and the list of evaluators is
displayed on the view coordinator
screen
The administrator
while adding the
coordinator enters a
user name which
already exists
The administrator adds a coordinator but
enters a user name which already exists
in the system
The system prompts the user with
message "username already exists,
please choose another user name"
The administrator
edits a coordinator's
profile
The administrator selects a coordinator
and clicks edit. He changes the
necessary details and clicks update
The details of the coordinator are
modified
The administrator
deletes a coordinator
profile
The administrator selects a coordinator
and clicks delete
The coordinator is deleted from the
system
The administrator
refreshes the manage
coordinator screen
The administrator on the manage
coordinator screen clicks the refresh
button
The manage coordinator screen is
refreshed from the database with
latest information
The coordinator
chooses to
synchronize exams
but provides incorrect
credentials or
credentials which
don't exist on the
portal
The coordinator selects synchronize
exams and can choose to synchronize
either each exam individually or all at the
same time by selecting Sync All but
provides incorrect credentials or
credentials which don't exist on the portal
The system responds with error
message communicating to the
user that the account doesn't exist
and the credentials are invalid
The coordinator
chooses to
synchronize exams
and provides correct
credentials
The coordinator selects synchronize
exams and can choose to synchronize
either each exam individually or all at the
same time by selecting Sync All and
provides correct credentials
The exams are synchronized with
the EMT server
The coordinator
chooses to view all
previously
synchronized exams
The coordinator in Synchronize Exams
tab selects Synchronized Exams
The system displays list of all
previously synchronized exams
along with users name, score, and
P/F info
The coordinator
chooses to delete 1
synchronized exam
The coordinator selects 1 exam from the
list of synchronized exams and clicks
Delete button
The selected synchronized exam
gets deleted
The coordinator
The coordinator clicks Delete Sync
The system prompts the user to
Deloitte-Federal Healthcare
38 of 57
Final Report / Concept of Operations ( CONOPS )
DCMT TE W81 XWH-07-C-01 1 5
chooses to delete all
synchronized exam
button
confirm that he wants to delete all
synchronized exams. Once you
user confirms by clicking yes, all the
synchronized exam get deleted
Maximum
Synchronization Time
The synchronization should complete in
Maximum 60 minutes
The synchronization completes in
Maximum 60 minutes
The administrator
chooses to view the
passed candidates
and searches by last
name
The administrator clicks on "exams and
candidates->passed candidates”. He
enters last name of a candidate and
clicks the find button
The system displays the details of
the candidate
The administrator
chooses to view the
passed candidates
and searches by a
particular date
The administrator clicks on "exams and
candidates->passed candidates”. He
enters/selects a date and clicks the find
button
The system displays the list of the
candidates
The administrator
chooses to view the
passed candidates
and searches by a
range of date
The administrator clicks on "exams and
candidates->passed candidates”. He
selects a range date by providing the
values in "From" and "To" and clicks the
find button
The system displays the list of the
candidates
The administrator
chooses to view the
list of passed exams
of a candidate
The administrator selects a candidate
and clicks "View Exam"
The system displays the list of
passed exams for a candidate
The administrator
chooses to view the
failed candidates and
searches by last
name
The administrator clicks on "exams and
candidates->failed candidates" . He
enters last name of a candidate and
clicks the find button
The system displays the details of
the candidate
The administrator
chooses to view the
failed candidates and
searches by a
particular date
The administrator clicks on "exams and
candidates->failed candidates" . He
enters/selects a date and clicks the find
button
The system displays the list of the
candidates
The administrator
chooses to view the
failed candidates and
searches by a range
of date
The administrator clicks on "exams and
candidates->failed candidates”. He
selects a range date by providing the
values in "From" and "To" and clicks the
find button
The system displays the list of the
candidates
The administrator
chooses to view the
list of failed exams of
a candidate
The administrator selects a candidate
and clicks "View Exam"
The system displays the list of failed
exams for a candidate
The administrator
chooses to view the
pending candidates
and searches by last
name
The administrator clicks on "exams and
candidates->pending candidates”. He
enters last name of a candidate and
clicks the find button
The system displays the details of
the candidate
The administrator
chooses to view the
pending candidates
and searches by a
particular date
The administrator clicks on "exams and
candidates->pending candidates”. He
enters/selects a date and clicks the find
button
The system displays the list of the
candidates
The administrator
The administrator clicks on "exams and
The system displays the list of the
Deloitte-Federal Healthcare
39 of 57
Final Report / Concept of Operations ( CONOPS )
DCMT TE W81 XWH-07-C-01 1 5
chooses to view the
pending candidates
and searches by a
range of date
candidates->pending candidates”. He
selects a range date by providing the
values in "From" and "To” and clicks the
find button
candidates
The administrator
chooses to view the
list of pending exams
of a candidate
The administrator selects a candidate
and clicks "View Exam”
The system displays the list of
pending exams for a candidate
The administrator
chooses to view the
current EMT
information
The administrator chooses "Help->About
EMT"
The systems displays the details
about the current EMT application
like "version etc"
The coordinator
chooses to delete
previous
synchronization
The coordinator selects the exam from
the list of listed previously synchronized
exams that he wishes to delete and
selects Delete Synchronization
The system deletes the selected
exam from the displayed list of
synchronized exams
The coordinator
chooses to delete
exam
The coordinator selects the exam that he
wishes to delete and selects Delete
Exam
The system deletes the selected
exam from the list of exams.
The coordinator views
report of all passed
exams
Selects View Exam Reports>AII
Exams>Passed and fills in the search
criteria i.e. date needed to view the
reports of the passed exams
Lists all the exams passed in the
specified timeline
The coordinator filters
the passed exam
records by
examination name
Selects the exam to sort by from the
options listed in Filter Record by
Examination Exam
Lists the selected exam that were
passed in the predefined timeline
The coordinator
chooses to view the
reports of passed
exam by Group Name
Selects the group name the coordinator
intends to view reports for from the listed
group names in Group Name drop down.
Lists all the passed exams by
candidates pertaining to the group
selected in the search criteria
The coordinator
chooses to view the
reports of passed
exam by Team Name
Selects the team name the coordinator
intends to view reports for from the listed
team names in Team Name drop down.
Lists all the passed exams by
candidates pertaining to the team
selected in the search criteria
The coordinator
exports the report to
excel
Selects the export to excel icon
The system opens MS Excel
spreadsheet with the exported data
The coordinator
chooses to rescore an
exam
Selects the exam intended to rescore
and selects view exam icon. Changes
the points awarded as required. Enters
the reason to rescore in the box and
selects rescore box
The system rescores the exam
The coordinator views
the reports of all
failed exams
Selects View Exam Reports>AII
Exams>Failed and fills in the search
criteria i.e. date needed to view the
reports of the failed exams
Lists all the exams failed in the
specified timeline
The coordinator filters
the passed exam
records by
examination name
Selects the exam to sort by from the
options listed in Filter Record by
Examination Name
Lists the selected exam that were
passed in the predefined timeline
The coordinator
chooses to view the
reports of failed
exams by Group
Selects the group name the coordinator
intends to view reports for from the listed
group names in Group Name drop down.
Lists all the failed exams by
candidates pertaining to the group
selected in the search criteria
Deloitte-Federal Healthcare
40 of 57
Final Report / Concept of Operations ( CONOPS )
DCMT TE W81 XWH-07-C-01 1 5
Name
The coordinator
chooses to view the
reports of failed exam
by Team Name
Selects the team name the coordinator
intends to view reports for from the listed
team names in Team Name drop down.
Lists all the failed exams by
candidates pertaining to the team
selected in the search criteria
The coordinator
exports the report to
excel
Selects the export to excel icon
The system opens MS Excel
spreadsheet with the exported data
The coordinator
wishes to view the
exam taken by
candidates
Selects the exam wished to view and
selects view exam icon
The system displays the exam
selected for viewing
The coordinator
chooses to rescore an
exam
Selects the exam intended to rescore
and selects view exam icon. Changes
the points awarded as required. Enters
the reason to rescore in the box and
selects rescore box
The system rescores the exam
The coordinator
chooses to view the
reports of all exams
Selects View Exam Reports>AII
Exams>AII and fills in the search criteria
i.e. date needed to view the reports of all
the exams
The system displays all the exams-
passed and failed taken in the
selected timeline
The coordinator filters
the exam records by
examination name
Selects the exam to sort by from the
options listed in Filter Record by
Examination Exam
Lists the selected exam that were
passed in the predefined timeline
The coordinator
chooses to view the
reports of all exam by
Group Name
Selects the group name the coordinator
intends to view reports for from the listed
group names in Group Name drop down.
Lists all the exams by candidates
pertaining to the group selected in
the search criteria
The coordinator
chooses to view the
reports of all exams
by Team Name
Selects the team name the coordinator
intends to view reports for from the listed
team names in Team Name drop down.
Lists all exams by candidates
pertaining to the team selected in
the search criteria
The coordinator
exports the report to
excel
Selects the export to excel icon
The system opens MS Excel
spreadsheet with the exported data
The coordinator
wishes to view the
exam taken by
candidates
Selects the exam wished to view and
selects view exam icon
The system displays the exam
selected for viewing
The coordinator
chooses to view the
exam statistics
Specifies the dates between which the
coordinator intends to view statistics for
and selects pass or fail as desired
Lists the statistics of all the exams
taken during the time frame
The coordinator views
the statistic based on
Group Name
The coordinator selects the group name
to view the statistics of from Group
Name drop down
Lists the selected group's
performance statistics
The coordinator views
the statistic based on
Team Name
The coordinator selects the team name
to view the statistics of from Team Name
drop down
Lists the selected team's
performance statistics
The coordinator
chooses to export the
report to excel
Selects the export to excel icon
The system opens MS Excel
spreadsheet with the exported data
The coordinator views
the exam statistics for
all the exams taken
on a particular date
In By Date tab selects the date for which
he wishes to the results for in Date
Lists all the exams taken on the
selected exams
Deloitte-Federal Healthcare
41 of 57
Final Report / Concept of Operations ( CONOPS )
DCMT TE W81 XWH-07-C-01 1 5
The coordinator views
the statistic based on
Group Name
The coordinator selects the group name
to view the statistics of from Group
Name drop down
Lists the selected group's
performance statistics on the
specific date
The coordinator views
the statistic based on
Team Name
The coordinator selects the team name
to view the statistics of from Team Name
drop down
Lists the selected team's
performance statistics on the
specific date
The coordinator
chooses to export the
report to excel
Selects the export to excel icon
The system opens MS Excel
spreadsheet with the exported data
The coordinator views
All Exams taken till
date
Selects All Exams link under Exam-wide
Statistics
Lists all the exams taken till date
The coordinator views
the statistic based on
Group Name
The coordinator selects the group name
to view the statistics of from Group
Name drop down
Lists the selected group's
performance statistics till date
The coordinator views
the statistic based on
Team Name
The coordinator selects the team name
to view the statistics of from Team Name
drop down
Lists the selected team's
performance statistics till date
Coordinator exports
the statistics to excel
Selects the intended statistics and
selects Export to Excel
The system opens MS Excel
spreadsheet with the exported data
Coordinator views the
rescored exams
between two specified
dates
Defines the dates from and to for which
intended to view the result for under
Between Dates
The system displays all the exams
rescored in the specified timeline
Coordinator chooses
to view the rescored
exams based on
Examination name
Selects the exam to sort by from the
options listed in Filter Record by
Examination Name
Lists all the rescored exams with
the specified Exam name between
the predefined dates
Coordinator chooses
to view the rescored
exams based on
Group Name
Selects the group name the coordinator
intends to view reports for from the listed
group names in Group Name drop down.
Lists all the rescored exams with
the specified Group name between
the predefined dates
Coordinator chooses
to view the rescored
exams based on
Team Name
Selects the team name the coordinator
intends to view reports for from the listed
group names in Team Name drop down.
Lists all the rescored exams with
the specified team name between
the predefined dates
The coordinator
exports the report to
excel
Selects the export to excel icon
The system opens MS Excel
spreadsheet with the exported data
The coordinator
wishes to view the
exam taken by
candidates
Selects the exam wished to view and
selects view exam icon
The system displays the exam
selected for viewing
Coordinator views the
rescored exams on a
specific date
Selects the date for which intended to
view the result under By Date
The system displays all the exams
rescored on the specific date
Coordinator chooses
to view the rescored
exams based on
Examination name
Selects the exam to sort by from the
options listed in Filter Record by
Examination Name
Lists all the rescored exams with
the specified Exam name on the
specific date
Coordinator chooses
to view the rescored
exams based on
Group Name
Selects the group name the coordinator
intends to view reports for from the listed
group names in Group Name drop down.
Lists all the rescored exams with
the specified Group name on the
specific date
Coordinator chooses
Selects the team name the coordinator
Lists all the rescored exams with
Deloitte-Federal Healthcare
42 of 57
Final Report / Concept of Operations ( CONOPS )
DCMT TE W81 XWH-07-C-01 1 5
to view the rescored
exams based on
Team Name
intends to view reports for from the listed
group names in Team Name drop down.
the specified team name on the
specific date
The coordinator
exports the report to
excel
Selects the export to excel icon
The system opens MS Excel
spreadsheet with the exported data
The coordinator
wishes to view the
exam taken by
candidates
Selects the exam wished to view and
selects view exam icon
The system displays the exam
selected for viewing
Coordinator views all
rescored exams till
date
Selects All Rescored under Rescored
exams
The system displays all the exams
rescored all the exams till date
Coordinator chooses
to view the rescored
exams based on
Examination name
Selects the exam to sort by from the
options listed in Filter Record by
Examination Name
Lists all the rescored exams with
the specified Exam name till date
Coordinator chooses
to view the rescored
exams based on
Group Name
Selects the group name the coordinator
intends to view reports for from the listed
group names in Group Name drop down.
Lists all the rescored exams with
the specified Group name till date
Coordinator chooses
to view the rescored
exams based on
Team Name
Selects the team name the coordinator
intends to view reports for from the listed
group names in Team Name drop down.
Lists all the rescored exams with
the specified team name till date
The coordinator
exports the report to
excel
Selects the export to excel icon
The system opens MS Excel
spreadsheet with the exported data
The coordinator
wishes to view the
exam taken by
candidates
Selects the exam wished to view and
selects view exam icon
The system displays the exam
selected for viewing
Coordinator chooses
to view the rescore
statistics between two
dates
Specifies dates in From and To under
Between dates menu in the Rescore
Statistics
The system displays the statistics of
rescored exams during the
specified timeline
The coordinator views
the rescore statistic
based on Group
Name
The coordinator selects the group name
to view the statistics of from Group
Name drop down
Lists the selected group's rescore
statistics between the two
predefined dates
The coordinator views
the statistic based on
Team Name
The coordinator selects the team name
to view the statistics of from Team Name
drop down
Lists the selected team's rescore
statistics b/w the two predefined
dates
The coordinator
chooses to export the
report to excel
Selects the export to excel icon
The system opens MS Excel
spreadsheet with the exported data
Coordinator chooses
to view the rescore
statistics for a
specific date
Chooses the date for which wished to
view the statistics of rescored exams
under the By Date
Lists the statistics of rescored
exams on the specified date
The coordinator views
the statistic based on
Group Name
The coordinator selects the group name
to view the statistics of from Group
Name drop down
Lists the selected group's rescore
statistics on the specific date
The coordinator views
the statistic based on
The coordinator selects the team name
to view the statistics of from Team Name
Lists the selected team's rescore
statistics on the specific date
Deloitte-Federal Healthcare
43 of 57
Final Report / Concept of Operations ( CONOPS )
DCMT TE W81 XWH-07-C-01 1 5
Team Name
drop down
The coordinator
chooses to export the
report to excel
Selects the export to excel icon
The system opens MS Excel
spreadsheet with the exported data
Coordinator views
rescore statistics of
all exams rescored till
date
Selects All Exams link under Rescore
Statistics
Lists the statistics of all rescored
exams till date
The coordinator views
the statistic based on
Group Name
The coordinator selects the group name
to view the statistics of from Group
Name drop down
Lists the selected group's rescore
statistics till date
The coordinator views
the statistic based on
Team Name
The coordinator selects the team name
to view the statistics of from Team Name
drop down
Lists the selected team's rescore
statistics till date
Coordinator exports
the statistics to excel
Selects the intended statistics and
selects Export to Excel
The system opens MS Excel
spreadsheet with the exported data
The coordinator
chooses to view
evaluator reports as a
daily report
Defines the date for which the
coordinator intends to view the report for
in Date under Daily Reports
Lists the evaluator reports for the
defined date
The coordinator
chooses to view the
daily report sorted by
Evaluator's name
The coordinator selects evaluator's name
to view the statistics for
Lists the selected evaluator's daily
reports
The coordinator
chooses to export the
report to excel
Selects the export to excel icon
The system opens MS Excel
spreadsheet with the exported data
8.1.4 Preliminary Test Results
The Deloitte Team conducted system tests beginning with the April 1, results below
Section
Number of
Test Areas
Passed
Failed
Not Tested
Evaluator/Table PC
28
28
0
0
Portal
74
45
19
10
EMT Console
121
104
8
9
Total Test Cases
223
177
27
19
Continual testing and development conducted until the functionality listed above were complete.
Deloitte-Federal Healthcare
44 of 57
Final Report / Concept of Operations ( CONOPS )
DCMT TE W81 XWH-07-C-01 1 5
8.2 December 2009 Field Testing
Deloitte conducted a train the trainer session and performed field system testing during a four
day period. The following personnel were in attendance.
DCMT Personnel:
• Mr. Marcus Vigil
• Ms. Chavez
• CPT Brookins
• DCMT Instructors
Deloitte Personnel:
• Mr. Pawan Singh
• Mr. Raman Singh
• Mr. Devchand Chute
• Mr. Shivaraman Raghuraman
8.2.1 Day 1 Environment and Setups
Setup 64 Tablet PCs for use in the ALL SKILLS examination.
We set up the Tablet PC application on 64 Tablet PCs. We added the mapping between the
server name and its static IP address into all the Tablet PC’s. Each Tablet was completely
charged. We also installed and configured the EMT Console application on the local server
laptop along with the Application and Database on the laptop. The laptop was configured with a
static IP address.
Setup Wireless Environment
The secure wireless environment was set up at the exam rooms in the following manner. We
setup the Laptop Server with the Console application in the coordinators room. ( Conf Room
120). A wireless controller acting as the main wireless gateway was also setup in the same
room. The laptop server was connected to the gateway using an Ethernet cable. In each of the
exam rooms, one wireless access points was setup to enhance the wireless connectivity for all the
Tablet PCs.
8.2.2 Day 2 Training
The scope of training provided included:
• Train users on the process and data flow of the solution including Tablet PC, Console and
Portal application.
• Demonstrate Tablet PC application including Exam navigation, completion, signing and
submission.
Deloitte-Federal Healthcare
45 of 57
Final Report / Concept of Operations ( CONOPS )
DCMT TE W81 XWH-07-C-01 1 5
• Demonstrate Console application including Reporting, Dashboard, Re-Score and
synchronization functionality.
• Demonstrate Web Portal application including data verification, reporting and printing.
Training conducted:
• Around 40 Instructors/instructors were trained in a quick training session which covered
the following:
• Demonstrated Tablet PC application including form navigation, completion, signing and
submission.
• Demonstrated Console application including Reporting and Dashboard functionality.
• We also answered questions about the ability to write on the tablet PCs.
• The instructors were then given a Tablet PC each to practice.
• A training database was setup with dummy candidates and instructors
• The instructors were allowed to complete forms and submit exams for these dummy
candidates, so as to get a feel of the exam.
Issues encountered and corrected
During the training exercise, we uncovered an issue. When more than 6 users submitted forms at
the same exact time, a deadlock condition arose. To fix this issue, we modified the Tablet PC
application to listen to the deadlock error. Upon the occurrence of the error, we resubmitted the
form data automatically until the submission was successful. This fixed the issue.
8.2.3 Day 3 Field Testing
Instructors Trial Testing
The instructors were assigned one Tablet PC each and we signed the Tablet PCs out. The
candidates were listed in the Tablet PC application and the evaluators were able to pick them out
easily. The instructors used the various search features to select the candidate, and a list of
exams available for each candidate was shown. The instructors loaded the skill sheet for the
required exam and graded the candidate by starting the timer. The instructors were also able to
read out instructions to the candidates by bringing up the instructions on the application. The
instructors were also able to bring up scenarios, when necessary, on the Tablet PC using the
application
Coordinators Trial Testing
The coordinator was able to track exams as they were being submitted. The coordinator was able
to track candidates who failed exams and was able to counsel them immediately and send them
to a Re-Teach session, if necessary. The coordinator was also able to track if any candidate failed
the ALL SKILLS EXAMINATION, on an immediate basis and he coordinator was able to
generate reports as necessary.
Testing Results
A total of 2790 exams were submitted by the end of the All Skills Examination. (This includes
the passed and failed exams). The coordinator had very little work except to click buttons to
Deloitte-Federal Healthcare
46 of 57
Final Report / Concept of Operations ( CONOPS )
DCMT TE W81 XWH-07-C-01 1 5
generate the required reports at the end of the examination. Instructors found the Tablet PC
application much easier and faster to use vs. the paper based forms. Instructors noticed the time
saved in pulling up the forms on Tablet PC with all the candidate information pre-populated.
Instructors felt comfortable writing comments on the Tablet PC. Instructors also liked the auto-
populated comments for critical criteria, which reduced the amount of redundant typing required
on their part.
Instructors also noticed that the application and wireless environment were very stable with
absolutely no issues on the second day of All Skills. Battery life of Tablet PC was down to
anywhere from 65% to 80% after 3 hours of usage. Instructors noticed they saved time in not
having to count and grade the exam manually, as the application did that for them automatically.
Some issues pointed out to us by the instructors and the coordinator were:
• Need for count of exams by attempt for the candidates.
• Need to fix some typos in the instructions and the scenarios.
• Overall the feedback was very positive.
8.2.4 Additional Functionality Identified and Implemented as a Result of
Field Testing
The table below shows additional work needed and later completed by this project.
Additional Functionality Requested
Technical Solution
Need capability to find previous attempts
easily
Add a link for the skill on the dashboard.
Clicking on the link will take the coordinator
to a page which will display all the attempts for
the skill for that candidate
Need functionality to import instructors too
Add an import instructor feature similar to the
Import Candidates feature, on the Manage
Users tab.
Need to be able to create an "Instructor list"
for signing Tablet PCs in and out during the
All Skills Examination
Add Export to Excel feature to the Instructor
List in the Manage Users tab.
Need to have the ability to disable the
enforcement of the password policy for the
users.
Add a check box on Add User pages to disable
the enforcement of password policies.
Need to be able to delete any candidate’s
exams.
Add a delete button and functionality on the
Exams List page.
Need to be able to generate an excel report
of the exams not taken by the candidates
Add Export to Excel functionality to the
Exams Not Taken report
Need to able to search Instructors by name
Add Search by Name functionality to the
Instructor List in the Manage Users tab.
Need the ability to search candidates by
name on the Dashboard
Add Search by Name functionality to the
Candidate Dashboard.
The Candidate roster report needs to show
the count per attempt for each skill
Add the additionally required counts to the
report. Also generate a separate report
Deloitte-Federal Healthcare
47 of 57
Final Report / Concept of Operations ( CONOPS )
DCMT TE W81 XWH-07-C-01 1 5
specifically for this.
Column headers on the Dashboard need to
be made more easy to understand
Change the Dashboard column headers
Company name needs to contain the
number and year
Change the company name on all reports
Spinal Immobilization Supine Patient needs
to have "LSB" suffixed to it
Change the name to contain LSB wherever
applicable
Spinal Immobilization Seated Patient needs
to have "KED" suffixed to it
Change the name to contain KED wherever
applicable
"Reteach" stamp should be changed to
"RE-TEST/RE-TACH "
Change the image for the same
Instructions and Scenarios need to be
updated
Update the html files for the scenarios and
instruction
Need to be able to overwrite a passed exam
to failed
Change the rescore feature to allow failing a
passed exam too
8.3 Final Testing January 2010
Deloitte conducted a train the trainer session and performed field system testing during a four
day period. The following personnel were in attendance:
Deloitte Personnel:
Mr. Pawan Singh
Mr. Raman Singh
Mr. Devchand Chute
Mr. Shivaraman Raghuraman
DCMT Personnel:
Mr. Marcus Vigil
Ms. Chavez
Cpt. Brookins
DCMT Instructors
8.3.1 Training and Field Trial
The visit to the client site was scheduled to train a set of end-users (Instructors/instructors) and
run the entire ALL SKILLS EXAMINATION using the application for four of the eight teams
being tested.
Training:
We set the following training objectives for this trip:
• Train users on the process and data flow of the solution including Tablet PC, Console and
Portal application.
• Demonstrate Tablet PC application including Exam navigation, completion, signing and
submission.
Deloitte-Federal Healthcare
48 of 57
Final Report / Concept of Operations ( CONOPS )
DCMT TE W81 XWH-07-C-01 1 5
• Demonstrate Console application including Reporting, Dashboard, Re-Score and
synchronization functionality.
• Demonstrate Web Portal application including data verification, reporting and printing.
Field Trial
We also set the following goals for the field testing:
• The instructors use the tablet PC application instead of the paper forms to conduct the ALL
SKILLS EXAMINATION for all the eight teams being tested.
• The coordinator uses the EMT Console application to track status and coordinate the ALL
SKILLS EXAMINATION, instead of manually compiling the results and generating reports.
8.3.2 Environment and Set Up Day 1 | January 7, 2010
We setup the Tablet PCs for use in the ALL SKILLS EXAMINATION. We charged about 50
tablet PCs completely. We then pointed all tablet PCs to the training server for the instructors to
play around with and get comfortable with the system. We trained about 20 new instructors to
use the Tablet PC application in a quick training session which covered the following:
•Demonstrated Tablet PC application including form navigation, completion, signing and
submission.
•We also answered questions about the ability to write on the tablet PCs and also if portable
keyboards could easily be used to write comments on the application.
•The instructors were then given a tablet PC each.
•A training database was setup with dummy candidates and instructors
•The instructors were allowed to complete forms and submit exams for these dummy candidates,
so as to get a feel of the exam.
We also trained two new coordinators to use the Console application. We then tested the wireless
network strength and stability by walking around the building and submitting forms from various
parts of the building. The wireless environment, the tablet PC application and the Console
application were not changed. We finally received the list of candidates for the exam. We trained
the coordinators on the import process and also created a template for the candidate list that
could easily be imported into the system. We also added the new instructors into the system.
8.3.3 Day 2 Field Trial Instructors and Evaluators | January 8, 2010
Field Trial -Instructors
The instructors were assigned one Tablet PC each and were signed out to each. The candidates
were listed in the Tablet PC application and the evaluators were able to pick them out easily. The
instructors used the various search features to select the candidate, and a list of exams available
for each candidate was shown. The instructors loaded the skill sheet for the required exam and
graded the candidate by starting the timer. The instructors were also able to read out instructions
to the candidates by bringing up the instructions on the application. The instructors were also
able to bring up scenarios, when necessary, on the Tablet PC using the application.
Deloitte-Federal Healthcare
49 of 57
Final Report / Concept of Operations ( CONOPS )
DCMT TE W81 XWH-07-C-01 1 5
There were about 1686 forms submitted, amounting to about three and a half skills, by the end of
the day.
Field Trial -Coordinators
The coordinator was able to track exams as they were being submitted. The coordinator was able
to track candidates who failed exams and was able to counsel them immediately and send them
to a Re-teach session, if necessary. The coordinator was also able to track if any candidate failed
the ALL SKILLS EXAMINATION, on an immediate basis and the coordinator was also able to
generate reports as necessary.
8.3.4 Day 3 Feedback and Results | January 11, 2010
A total of 2571 exams were submitted by the end of the All Skills Examination. The coordinator
only had to except by a click to generate the required reports at the end of the examination.
Instructors found the Tablet PC application much easier and faster to use vs. the paper based
forms. Instructors noticed the time saved when the Tablet PC pulled up the forms with all the
candidate information pre-populated.
Instructors felt comfortable writing comments on the Tablet PC, and Instructors liked the auto-
populated comments for critical criteria, which reduced the amount of redundant typing required
on their part. Instructors also noticed that the application and wireless environment were very
stable with absolutely no issues throughout All Skills test. Instructors noticed they saved time in
not having to count and grade the exam manually, as the application did that for them
automatically.
Battery life of Tablet PC was down to anywhere from 65% to 80% after 3 hours of usage.
Instructors and coordinators pointed out the following issues to us:
-Need for more generic comments for situations when the candidates failed because of not
meeting the minimum score.
-Need to reduce the typos in the instructions and the scenarios.
Overall the feedback was very positive.
8.3.5 Additional Functionality implemented as a result of the trip
Additional Functionality Requested
Technical Solution
Need to be able to track class name for each
candidate
Add and additional column called Class to the
Candidate Table
Need ability to track which candidates have
been called for counseling
Mark the records that have been printed on the
list of exams
Need ability to filter the list of failed exams
by instructor
Add a filter to the list of failed exams to filter
by exams
Deloitte-Federal Healthcare
50 of 57
Final Report / Concept of Operations ( CONOPS )
DCMT TE W81 XWH-07-C-01 1 5
Need ability to be able to repair the wireless
connect from within the application
Place button into the application or re-start the
router manually
Instructors must be forced to read
instructions to the candidate before
beginning the evaluation
Lock the exam from until the instructions to
the candidate page is not opened by the
instructor
Critical criteria should be automatically
checked based on responses to
corresponding questions
Check the responses to questions and if it
matches requirements, check the corresponding
critical criteria automatically
Need for the ability to fail the candidate if
time has been exceeded
Add a checkbox in the critical section for time
exceeded criteria. Check box should not
appear elsewhere
Field Size for username and password fields
need to be bigger for easier access
Increase the size of the text boxes required for
input
Instructors not aware of minimum score for
each skill
Comments need to be automatically added to
skill sheet if the candidate does not achieve the
minimum score
9 Conclusion
9. 1 Strategic Outcomes
The Deloitte approach to this research effort was straightforward with maximum flexibility
toward creation of a useable end product.
Begin with a broad hypothesis
The project was conducted in to one-year
phases. During the first phase, we developed
the technology and the initial requirements
around the hypothesis.. During the second
phase, we implemented the requirements into
the DCMT system
Develop initial set of requirements based
upon the hypothesis
During the second phase, we implemented and
tested the initial requirements by building the
DCMT system
Work with DCMT personnel to review the
initial requirements
DCMT personnel reviewed and approved the
resulting set of functionalities provided by the
system
Develop the final functionality based upon
field testing and adoption of enhancements
During separate field testing trips, we fully
tested the functionality and made many
enhancements. DCMT are continuing to use
the system to enhance their testing processes.
What the strategy was going in? _ How did the strategy work?
Deloitte-Federal Healthcare
51 of 57
Final Report / Concept of Operations ( CONOPS )
DCMT TE W81 XWH-07-C-01 1 5
9.2 Performance Results
The table below describes the results mapped to their original hypothesis
Category
Criteria
Results Achieved in the Pilot
1. Increased
Efficiency
a. reduced paper use and
associated storage and
management cost
b. free up manpower used as
runners and processors
c. instant scoring/feedback
including details for corrective
assistance
d. decreased time of overall
testing process
a. Significant (eliminates
paper involved in 10,000+
individual exams/year)
b. Significant (eliminated 10
runners + 9 processors per
shift, 2 per day, 2 days
every other month - 1920
man-hour s/yr.)
c. Significant (the ability for
instant scoring provided
more effective training
environment allowing the
instructor instant access to
candidate achievements
d. Significant (30% reduction
in time required for testing -
from 20 hours to 14 hours)
2. Increased
Accuracy
a. legible capture of instructor
comments
b. validation of input prior to
submission
c. import of student demographics
a. Significant
b. Significant
c. Significant (instructor
comments very positive on
this being a substantial
time-saver)
3. Administrative
Simplification
a. enable automated management
of performance indicators
b. web-enabled access
c. automated synchronization of
TEMTPC and EMT Web Portal
d. digital data sharing capability
a. Significant (includes
performance of instructors)
b. This feature declined
c. Significant
d. Significant (decreased
processing time and
shipping/handling cost)
4. Enhanced Data
Security
a. encryption during testing
process during transfer between
instructor to evaluator
b. encrypted storage
c. encryption for transfer
a. Significant
b. Significant
c. Significant
Deloitte-Federal Healthcare
52 of 57
Final Report / Concept of Operations ( CONOPS )
DCMT TE W81 XWH-07-C-01 1 5
9.3 General Comments
This engagement demonstrates our ability to lead a multifaceted, large-scale business process
and technology implementation in a highly controlled environment. Deloitte was able to
assemble a team with the necessary technology experience as well as the detailed understanding
of DCMT processes and policies necessary to succeed on this project. We’ve applied leading-
edge enterprise-service wireless technology to provide an integrated environment for the client
without a full-scale replacement of its technology portfolio. The technology, training and
solution deployment experience gained on this project will be able to be used to help other health
care organizations implement more effective technology in a way that will meet their
requirements.
9.4 Recommendations to Expand Utilizations into Other Areas
From a high level, it was obvious that the application and devices were well received by the
DCMT users. Our intention was to meet the overall project goals and objectives with an
innovative product. Inherent in any research and proof of concept roll-out are both the
limitations of the solution and opportunities for expansion. At this point in the operationalization
of the tools, the overall system has moved into sustainment. Recommendations are provided
below:
9.4.1 Operational Recommendations
Operational recommendations are categorized by the following three components
a. Hardware - The Tablet PCs and Servers will need to be maintained both from a systems
operation point to keep the infrastructure current with latest patches as well as physical
security of the devices and replacement when damaged or defective.
b. Software - The applications loaded onto the laptop could become unstable and need to be
re-imaged. Data stored on the servers will need to be backed up with a system restore
capability designed and tested to ensure that the overall recovery scheme will work.
c. Configuration - The system may need to accommodate user generated requests for
additional functionality and/or deletion of current ones.
9.4.2 DCMT Expansion
The delivered system could be expanded with additional use cases by adding functionality in
each of the hardware components
• Tablet PC’s - The current use is limited to conducting the exams. These tools can be
expanded for use as training aids for students by simply adding documents, instruction
plans and commercial interactive training tools purchased or downloaded from other
sources
• Servers - The wireless connections between the server and the PCs could also be used for
group training sessions, surveys, etc. as part of an overall training regimen to test while
training to determine candidate knowledge and retention.
• Portal - The portal could be expanded to include such things as content management,
additional reporting, expanded business intelligence, and performance management. The
portal could also be expanded as a recruiting tool and overall knowledge management
portal to organize DCMT materials.
Deloitte-Federal Healthcare
53 of 57
Final Report / Concept of Operations ( CONOPS )
DCMT TE W81 XWH-07-C-01 1 5
9.4.3 Replication to Other Services
The system could be ported to other services by minor changes to the functionality to
accommodate individual service needs or other services could send candidates to DCMT for
training and certification. Either way, there could be significant potential savings.
Deloitte-Federal Healthcare
54 of 57
Final Report / Concept of Operations ( CONOPS )
DCMT TE W81 XWH-07-C-01 1 5
KEY RESEARCH ACCOMPLISHMENTS
Automating the testing process improved the efficiency of the Practical Exam administration
process and enhanced training effectiveness through adaptive testing and real time feedback on
performance.
Piloting the automated process:
• Decreased the amount of time required for scoring the practical exam and provides real-time
feedback to students and instructors
• Allowed trend analysis of the results of the exam to identify training deficiencies of the
individual as well as the program.
• Provided students additional learning opportunities where they are needed by decreasing the
overall amount of time required for the PE.
• Supported the development of a common framework that will automate additional training
programs and examinations.
• Enabled a plan for enculturation of instructors into an environment of digitized training.
• Returned soldiers currently used for runners and data entry to productive work and study.
• Identified opportunities for improvements and workflow effectiveness with the use
integrated technology
REPORTABLE OUTCOMES
This project has been used by Deloitte as a qualification to show experience in the development
of electronic forms and/or secure wireless portal solutions for two winning engagements to date.
CONCLUSION
The reaction of DCMT staff to the pilot implementation of the prototype designed indicates a
successful outcome in improving the testing process. The coordinator had very little work
except to click buttons to generate the required reports at the end of the examination. Instructors
found the Tablet PC application much easier and faster to use vs. the paper based forms.
Instructors noticed the time saved in pulling up the forms on Tablet PC with all the candidate
information pre-populated. Instructors felt comfortable writing comments on the Tablet PC.
Instructors also liked the auto-populated comments for critical criteria, which reduced the
amount of redundant typing required on their part.
The measured results also support a successful study - proving gains in efficiency at a level that
supports serious consideration for expanded use of the technology used for this study.
Specifically, a reduction in the time required for the overall testing process of 30 percent is a
major improvement. The reduction of man-hours required measured (more than 1900/year) did
not include the time freed up by the reduction in the time for the testing process, it is simply a
reduction in needed manpower for runners and processing personnel - the reduction of overall
testing time allowed the staff to reduce overtime demands for the first day of testing.
The application of technology to automate paper-based processes is often thought of as
expensive due to the initial investment, however the combination of efficiencies gained and the
Deloitte-Federal Healthcare
55 of 57
Final Report / Concept of Operations ( CONOPS )
DCMT TE W81 XWH-07-C-01 1 5
improved capability for managing data that extends beyond the reach of the original paper-based
process is important to consider as well. The pilot allows the course coordinator to evaluate not
only students taking the exam, but also to do analysis on test stations and instructors that will
provide insight for improving teaching methods and have a definitive impact on the overall
outcome of the training program.
The testing and evaluation process for the pilot system provided numerous opportunities for the
staff and the Deloitte team to interact, identify potential improvements, take action on those
inputs, and resulted in a final product that continues to be lauded by the 68W training program
staff.
Deloitte-Federal Healthcare
56 of 57
Final Report / Concept of Operations ( CONOPS )
DCMT TE W81 XWH-07-C-01 1 5
10 Appendix A
CKMBT_ 6000901271
45101Exemption.pdf
Deloitte-Federal Healthcare
57 of 57