Document text
Research Problem Review 78-16
TRAINING EVALUATION OF THE INDIVIDUAL
STAFF USER ADR TRAINING PROGRAM IN
TCATA TEST FM 222 TACTICAL OPERATIONS
SYSTEMS (TOS) CONCEPT TEST
William K Earl
ARI FIELD UNIT AT FORT HOOD. TEXAS
D D C
MM le 1979
U S Army
lOrVaJTICN 5T ATLMTVT _A
Apprrrid 1m put'll c
I'Samr-uf’.'w Onljaai**
Research Institute for the Behavioral and Social Sciences
September 1078
%
THIS DOCUMENT IS BEST
QUALITY AVAILABLE. THE COPY
FURNISHED TO DTIC CONTAINED
A SIGNIFICANT NUMBER OF
PAGES WHICH DO NOT
REPRODUCE LEGIBLY.
U S. ARMY RESEARCH INSTITUTE
FOR THE BEHAVIORAL AND SOCIAL SCIENCES
A Field Operating Agency under the Jurisdiction o( the
Deputy Chief of Staff for Personnel
|ONl I'll /i 1 1 >N F H
I n linn jl I Yiici I < x
WII.IIAMI FIAISIK
( oionrl. I 'S Ai my
1 iinmiiiitlrf
NOTiCf S
PlSTWlgljTlCyn »• — 0» !►., Ittvyl -» M*' AWt Mvm
of •vpo'll •» U i A"«v Bcmrrft lw*l >v** *<V >•» r»a V».
AtT»4 ^fni P W01 f Ar#*«rvv.« v 77JJ3
f iHAv Disposition t** '•on«» n» » •« *© Team f ••• -v>» .<
’►♦US *'"N »o* »•*—>■<*»< 1<V4 We-vei
SOTL Th# ' >vt »9« «* ?*•« '•CWVT M NFI to M ro*H'U»«l M •* oM’C«l Dto*»rm»«l o' »*• Army po* » -or
r #W «n t»y Of*** lutN)» red rtOCv»m*r»in
^ Army t'ro ]tct Niuat^r
/ 2076374 JA775 /
Human Performance in
Field Assessment
(Hj /\rtl /?£$ k£.V-76-1 G
Research Problem Review 78-16
( L T^Jkaininc evaluation of the ^individual ^jakf usfr adp
V 1 ' i n inc. r r>.ram in t at£ test yx 222
ITEM CTO? ) -
jypKR^TIONS^^STEM (T®) £pNCEPT
2 .
fEST
r
(£2
William KJ Ear
{'
i
ARI FIELD UNIT AT FORT HOOD, TEXAS
>AJ 3 i
Submitted as complete and Approved by:
technically accurate, by:
George M. Gividen
Field Unit Chief A. H. Birnbaum, Acting Director
Organizations and Systems Research
Laboratory
Joseph Zeidner
Technical Director
U.S. Army Research Institute for
the Behavioral and Social Sciences
Research Problem Reviews are special reports to military management.
They are usually prepared to meet requirements for research results bear¬
ing on specific management problems. A limited distribution is made —
primarily to the operating agencies directly involved.
0u
^0? O 1-0
FOREWORD
The Fort Hood Field Unit of the Army Research Institute for the
Behavioral and Social Sciences conducts research under Army Project
2Q7f> J74 JA775 and provides technical advisory service for the Traininq
and Doctrine Command (TRADOC) Combip^d Arms Test Act^vit^ (TCATA) . This
report is responsive to a request by rhi^Ccsisnah^ systems Test Director¬
ate oi lv'AT«£>for technical assistance in the conduct of the automatic
data processing (ADP) traininq evaluation effort for the Tactical Opera¬
tions System (TOS) Concept Test, TCATA Test FM 222. The U.S. Army cur¬
rently has under develojsnent a number of automatic data processing
a;? systems for employment in tactical missions. One of these is the TOS,
which is designed to facilitate the Command and Control decision pro-
ess by quickly gathering, analyzing, and disseminating tactical infor¬
mation to field commanders in sufficient time to allow quick reaction
to rapidly developing tactical situations.
The purpose of the TOS Concept Test (FM 222) was to provide a data
base to evaluate a concept of -automated data processing assistance to
the division staff. A limited system, called TOS Operable Seqment
(TOs£) , was used as the tost bed for conducting the test. The ADP
training program was to provide all staff user personnel assigned to
operate the TOS? with the knowledge and skills required to effectively
perform their tasks during execution of the test. Findings from as¬
sessment of the TOS 5, staff user training program provided information
for answering training evaluation objectives of the test.
K
TRAINING EVALUATION Of THE INDIVIDUAL STAFF USEh ADR TRAINING'. PROGRAM
IN TCATA TEST FN 222 TACTICAL OPERATIONS SYSTEM (TO.S) CONCEPT TEST
BRIEF
Requirement :
To obtain data that permits assessment of the training requirements
necessary for the division to employ TOS2.
Procedure :
The training schedule consisted of 5 weeks of 4 hours per day.
(The other 4 hours of each workday were spent in routine unit duties.)
Performance, demographic, and opinion data were collected to answ« i the
requirement . Performance data were obtained by means of end-of-traimnq
tests of trainees. Demographic and opinion data were obtained from
questionnaires and training reports made out by the trainees and evalu¬
ators at the end of the main lessons, at the end of the training program,
and at the end of the test. The data consisted of measures of of>erator
per formance in processing Intelligence information on TOS* input/output
terminals, and classification and rating scale scores, remarks, opin¬
ions, and observations concerning the training program. Descriptive and
inferential statistical tents and techniques were used to analyte the
data.
Findings :
The effectiveness of the training program was degrade by insuffi¬
cient total hours allocated for training, the use of 4-hour rather than
8-hour daily training sessions, the compressed training schedule, and
the dual commitment of students to classroom and unit duties. The ef¬
fectiveness of the training program would be improved if the 5 weeks of
4 hours per day were changed to weeks of 8 hours per day. An ex¬
panded training program of 5 weeks of 8 hours per day should be
considered .
The quality of instruction presented during training was rated as
excellent. However, because the ratio of students to input/output de¬
vices was too high (3.3 to 1), training effectiveness was degraded.
Trainees rated the practical exercise method, which emphasised
hands-on training, as more effective than the classroom- lecture method
of training. The proper balance between classroom- lecture training
and hands-on training must be determined by future research.
The proficiency level achieved by enlisted trainees was generally
higher among those who possessed one or more of the following charac¬
teristics: previous functional background experience: the appropriate
primary military occupat ional specialty (MOS 968, Intelligence Analyst);
and higher aptitude scores in general technical (GT) , combat, and sur¬
veillance and communications areas. The best single prediction of pro¬
ficiency level was GT score.
Utilization of Findings:
This report was incorporated as part of TCATA test report FM 222
Tactical Operations System (TOS) Concept Test RCS ATCD-8, November 1977.
Test results were employed as a part of the data from which to develop
requirements and recommendat ions for a TOS at division and corps, which
were presented to the CY 77 Army Systems Acquisition and Review Council
II (AS ARC ID/Defense Systems Acquisition and Review Council II (DSARC II).
TRAINING EVALUATION OK THE INDIVIDUAL STAFF USER ADP TRAINING PROGRAM
IN TCATA TEST FM 222 TACTICAL OPERATIONS SYSTEM (TOS) CONCEPT TEST
CONTENTS
Page
INTRODUCTION . 1
Test Purpose and Scope . 1
TOS ^ System Description . 2
TOS* ADP Individual Staff User Training Program . 3
Training Evaluation Objectives . 3
METHOD . S
Trainees . 5
Questionnaires .
Evaluator Reports .
Final Examination .
Demographic Data Survey . 7
STATISTICAL ANALYSIS . 8
STAFF USER TRAINING EVALUATION . 9
End-of-Lesson Ratings and Evaluations . 9
Trainee End-of-Trainlng and End-of-Test Ratings,
Evaluations, and Performance Measures . 12
Discussion . * . . . 16
OPERATOR STAFF USER TRAINING EVALUATION . 21
End-of-Lesson Ratings and Evaluations . 21
Discussion . 24
COMMANDER AND PRINCIPAL STAFF TRAINING EVALUATION . 24
ANALYSIS OF DEMOGRAPHIC VARIABLES . 24
Results .
Discussion
25
26
LIST OF TABLES
Page
Table 1. TOS" ADP individual staff user training program
of instruction . 4
2. Training subjects that trainees evaluated with
end-of- lesson questionnaires . 6
1. Biographical and aptitude measures included in
the destoqraphic data survey . 8
4. Trainee end-of- lesson ratings of staff user lessons . . 10
5. Trainee comments on ENSIT and FRENSIT file classes ... 11
6. Trainee ratings on training proficiency and
lesson time . 12
7. Trainee recommendations at end-of-training . 14
8. Trainee recommendations at end-of-test . 15
9. Trainee classroom attendance records . 16
10. Trainee scores on final examination . 17
11. Messages processed correctly on final examinations ... 18
12. End-of-lesson rating of instruction . 22
13. Trainee comments on MIQD operator lessons . 23
14. Evaluator ratings of MIOO operator lessons . 23
15. Summary of stepwise regression analysis for data from
MIOD operators in ENSIT and FRENSIT classes . 25
mm
TRAINING EVALUATION OF THE INDIVIDUAL STAFF USER ADP
TRAINING PROGRAM IN TCATA TEST FM 222 TACTICAL
OPERATIONS SYSTEM (TOS) CONCEPT TEST
INTRODUCTION
Advances in military technology have produced the means for con¬
ducting tactical operations at a much faster pace and intensity than
was formerly possible. This acceleration of activity means that the
cocnand and control system mus c process more tactical information for
decisionmaking but has les» _imc for making decisions. There is an
urgent requirement to gather large amounts of tactical information
quickly, analyze the inf ormation , and disseminate it to field commanders
for decisions and actions necessary for defense against rapidly devel¬
oping threats. As this pace increases, manual methods become less ef¬
fective for the necessary q licker reaction times.
The Army is aware of the problem and has been seeking to improve
the tactical command and control system by using computer technology
in the decision process. This approach is called the Tactical Opera¬
tions System (TOS) concept, and it has involved an evolutionary, itera¬
tive process of interactive studies, teats, and sof t war c/hardware de¬
velopment efforts. The test discussed here was designed as an operational
test of the TOS concept applied at the division level.
Test Purpose and Scope
The purpose of the test was to provide a data base with which to
evaluate the TOS concept of automated data processing (ADP) assistance
to the division staff, with emphasis on testing division staff's capa¬
bility to use information input from selected volumes of intelligence
and combat information from corps. Test results were to be used in de¬
veloping requirements and reconsnendations for a TOS at division and
corps.
The main part of the test was conducted as a series of command
post exercises, usinq both the manual system and a computer system--
the TOS Operable Segment (TOS^) , a test-bed system designed to test
the TOS concept. Participating command and staff echelons were se¬
lected elements of division, brigade, and battalion staffs. Control
organizations simulated corps, direct support artillery battalions,
and other division units. The test scenario placed the player division
in reserve for approximately 4fl hours and in contact with an enemy force
for approximately 72 hours. Scenario events included a mix of tacti¬
cal operations, command post displacements, continuity of operation
conditions, cownand post displacements, continuity of operation con¬
ditions, and electronic warfare and security threats.
1
TOS~' System Description
The TOS* is a limited application of automatic data processing of
the tactical operations of a division in the field. It is a complete
computer system consisting of an integrated assembly of ADP hardware,
software, personnel, and communicat ions designed to support G2/GJ op¬
erations within division headquarters. Tactical information is entered
into and retrieved from a central computer through message input/output
devices (MIOD) using variable- length, preformatted messages. Informa¬
tion from the system may be recorded in hard copy on the MIOD electronic
line printer, displayed on the display editor, or portrayed graphically
on the group display device (GDD) .
The TOS^ computer provides general services that include editing
and validation, data access control, file maintenance, report genera¬
tion, data base query, and automatic dissenu nation of information via
Standing Requests for Information (SRI). In addition, there are specific
processes associated with the three main functions automated in the TOS2.
Enemy Situation (ENSIT). This function stores, retrieves, sunsna-
rlzea, and disseminates selected portions of the information utilized
by intelligence elements in tactical operations. The categories of
information include essential information about specific enemy units,
their combat effectiveness, and history, plus spot report information
concerning enemy activities, potential targets, losses, and other enemy
situation data. Provisions are made for an intelligence working file
and for the preparation and dissemination of an intelligence summary,
as required.
Friendly Unit Information (FRENSIT) . This function provides for
the collecting, filing, display, and dissemination of the current status
of friendly units assigned or attached to the division. The categories
of information include location and tactical disposition of units; as¬
signment of units within a task organization or task force; essential
personnel, equipment, and supply status of units and task forces; and
operations reports. FRENSIT contains the capability to prepare the
division operations journal and situation reports.
FRENS I T/ENS I T (COMMON) . The COMMON function supports both the
FRENSIT and ENSIT functional areas by providing the capability to desig¬
nate and identify Named Area of Interest (NAI) in the data base and
monitoring Standing Requests for Information, when used with appropri¬
ate ENSIT or FRENSIT messages, the COMMON function enables the division
staff to accomplish, to a limited degree, tomplatinq processes.
2
TUS1* ADP Individual Staff User Training Progi am
Prior to testing , an ADP training program was conducted for di¬
vision personnel who would operate the TOS* system; they were designated
alternately as staff users, trainees, operators, or players. The pro¬
gram was designed to train staff users to employ the TOS^ hardware and
software applications that they would use in their duty assignments
during the test. The program was developed and implemented by training
personnel from Army Tactical Data Systems Command Systems Field Office
(ARTADS-CSFO) and United States Army Combined Arms Center (CACDA) . The
course consisted of individual as well as unit training phases, but
training evaluation was concerned with the individual training phase.
Individual ADP training was designed to provide selected player
personnel with a general knowledge of the capabilities and character¬
istics of individual TOS* messages and functions and the basic knowledge
and skills required to perform their staff duties in the test. The
course of instruction was divided into FRENSIT and ENSIT training class¬
es, in accordance with the staff functions automated In the TOS^. The
programs of instruction for the two classes were similar except that
specific FRENSIT and ENSIT subjects were taught only in each respective
class. Players wore assigned to one of the training classes on the
basis of the primary staff function they would perform in the test.
Those with G3 functions were assigned to the FRENSIT class, and those
with G2 functions attended the ENSIT class. The program of instruction
is outlined in Table 1.
Training Evaluation Objectives
One test objective concerned training evaluation; the test was to
obtain data that permitted assessment of the training requirements neces¬
sary for the division to employ TOS 2 . of primary interest was evaluation
of the effectiveness of the TOS^ ADP Individual Staff User Training Pro¬
gram. One aim was to determine (a) whether or not this instruction was
adequate for preparing the trainees to meet the program objectives and
(b) how effectively the program had been carried out. The other aim of
the evaluation was to ascertain if the trainees had accomplished the
program objectives at the end of training. A related interest was
identification of individual aptitudes or characteristics that could be
used as variables for predicting success in TOS* ADP training. The
variables could be used as criteria for selecting trainees for assign¬
ment to ADP training. The present evaluation was undertaken to investi¬
gate these problems.
3
Table l
TQS
2
ADI Individual Staff
User Training Program of
Instruction
Title
ENSIT
class
S2/G2
(hours)
FRENSIT
class
S3/G3
(hours)
Introduction to Staff User Course (TOS2)
4
4
TOS2 Demonstration
1
1
MIOD Operating Procedures
2
2
Assembly/Disassembly of Cabling
4*
4a
Digital Data Terminal
2*
2*
operator Maintenance/Trouble Shooting
2®
2*
Communications Methods
4*
4*
ADP Tactical Applications
4
4
Introduction to TOS* Messages
6
6
F-xami nation (Introduction to TOS2 Messages)
4
4
FNSIT Files, Formats, Applications, and PE
24
0
FRKNS IT Files, Formats, Applications, and PE
0
34
Examination (FRENSIT/ENSIT, Files)
4
4
Templating
2
2
Continuity of Operations (CONOPS)
2
2
Discussion of Brigade Relay
2
2
File Managers
2
2
Staff Interface in TOS2
6
6
Applications Review
2
2
Graphic Application
eb
0
Functional Applications
18
18
GDD Operations and Maintenance
8b
0
File Manager
2C
2C
Total Hours
123
105
*Only Selected MIOD Operators received these classes.
b
Intelligence personnel of AfcP only received this instruction.
Given only to personnel identified as file managers.
4
METHOD
Tuinco*
In all, 103 officer and enlisted personnel front the Ftrat Cavalry
Division and a military intelligence battalion froet III Corpa were as¬
signed aa players in the teat and aa trainees In the ADP training pro¬
gram. The backgrounds and abilities in the heterogeneous group varied
widely. Formal education ranged from 9 to 17 years. Military grade
ranged from E2 to 05, and time in service ranged from a few months to
over JO years. Almost all the officers had occupational specialties
in either combat arms or military intelligence, and large proportions
of enlisted soldiers had either a combat arms military occupational
specialty (MUS) or an Intelligence analyst MOS.
The training classes were integrated with all grades, and all
trainees in each class received the same instruction. The FRENSIT
class consisted of 24 officers and 27 enlisted personnel, totaling
51 trainees; the ENSIT class consisted of 15 officers and 37 enlisted
personnel, totaling 52 trainees. This trainee mix In both classes,
therefore, meant that there were wide variations in learning caj>abllity.
Questionnaires
Trainees were administered questionnaires at thtee points in the
program: at the end of lessons covering maior subjects, at the end of
the training program, and after the test had been completed. The end-
of-lesson quest lonnal res were oriented toward obtaining trainee opinion
concerning the quality of training and how well the lessons had been
taught. The end-of-traintnq and test questionnaires were oriented
toward obtaining trainees' estimates of their level of training profi¬
ciency before and after the test.
FJid-of-besson Questionnaires. These quest lonna l re* were designed
to obtain information concerning the effectiveness of the main compo¬
nents of lessons covering the major subjects in the program of Instruc¬
tion. F.ach questionnaire contained four or five questions. Question 1
asked trainees to rate the instruction in accomplishing the lesson ob¬
jectives on a 5-point scale from Very Effective to Very Ineffective.
Question 2 called for a yes-no response to whether they had received
enough instruction to fully understand the lesson objectives. Question
3, included for lessons having practical exercise sessions, called for
a yes-no response to whether they were given enough hands-on practice
with the equijwent to learn the operator skills covered in the lesson
objectives. Question 4 asked trainees whether they had accomplished
all the lesaon objectives and to indicate their Judqmont with a yea-no
response. Question 5 presented the amount of training time spent on
the subject and asked trainees to Judge whether aor^ the same, or
less time was required to learn fully the material "red by the
lesson objectives. Following the questions, space was provided for
recordmq remark* and observat ions . Table 2 lists the trainlnq sub*
<• . covered by end-of- lesson questionnaires! these subjects accounted
for approximately 70% of the trainlnq t law in the program.
Table 2
Trainlnq Subjects That Trainees Evaluated with
End-of-Leason Questionnaires
Subject
Trainlnq
hours
MI00 Operating Procedures 2
As ses&ly /Disassembly of Cablinq 4
Diqital Data Terminal 2
Operator Ha intenance /Troubleshoot lnq 2
CosHunication Hethods 4
Introduction to TOS' Nossaqes 6
ENSIT Files, Formats, Applications, and PE 24
FREHSIT Files, Formats, Applications, and PE 34
Functional Applications IS
File Manaqer 2
Total 98
End -of -Training and Test Quest lonnalres . These que s 1 1 onna Ires
were designed primarily to obtain trainee appraisals of their profi¬
ciency at the end of training and testing, and the relative amount of
training time required to attain proficiency in each subject in the
program of instruction. Question 1 consisted of two parts. In the
first part, trainees rated their proficiency on each trainlnq subject,
using a 3-point scale of Adequate, Borderline, and Inadequate. In
the second part of Question 1, trainees made a relative judgment of
the amount of training time required to attain adequate proficiency
on each subject, when considering the time allotted to each subject
In the training program: they used a 3-point scale of More, Same, and
less Time. Question 2 asked respondents to rate how effectively the
individual and unit training prepared them to perform their duties in
the test, using a S-polnt scale that ranged from Very Effectively to
Very Ineffectively. Question 3 requested the trainees to give their
overall opinion of the trainlnq program, using a 5-point rating scale
from Very Good to Very Poor. Question 4 asked for recommendations to
improve the training program.
6
Evaluator Report a
Tha evaluators completed end-of- lesson reporta of the lessons for
all training subjects in the program. Supplemental y reports contained
recorded observations of trainee performance in the final exanination
and opinions concerning the overall training programs.
The evaluator end-of-lesaon reports were designed to obtain evalu¬
ator judgments of the effectiveness of training provided for each sub¬
ject in the program of instruction. Item 1 reguired the evaluator to
rate the effectiveness of the training in terms of four components:
lecture session, practical exercise session, instructor's teaching
ability, and total lesson. Item 2 called for a yes-no response as to
whether or not all lesson objectives were accomplished. Item 3 asked
for the recording of all noteworthy observations and opinions.
Final Examination
A final examination for the training course was given at the con¬
clusion of ENSIT and FRENSIT training. The examination consisted of a
written part and a performance part. The written part was made up of
a series of true-false and fill-in questions concerning basic knowledge
of system functions and the preparat ion of TQS 2 messages. The items
required an understanding of the characteristics of message formats,
such as their uses, coding methods required for entering information
c°rr*ctly into message data sections, and system processes activated
by message ccsiponents.
The performance part of the examination was designed as a test of
the primary objective of the training program. A set of performance
standards was defined as the training criteria. The main criterion was
that a trainee should be able to send a selected group of 10 messages
representative of his section's ADp activities to the centrsl processing
unit within indicated time limits. All messages had to be accepted
(without error) by the computer system. The test consisted of a aeries
of 10 or 11 information requirements of different types that the trainee
had to translate into the appropriate TOS* message format and correctly
enter into the computer. Each trainee was seated at a MIOO by himself
and given approximately 1 hour and 20 minutes to complete the task.
.'X»sx>jraphlc Data Survey
Demographic data were collected on the enlisted trainees in the
ENSIT and FRENSIT classes: 24 biographical measures and aptitude scores
were collected from the trainees and recorded from their personnel files.
Table 3 lists the data elements collected.
7
Table J
Biographical and Aptitude Measures Included
in the DMoqraphlc Data Survey
A. Biographical Scores and Measures
1 . Grade
2 . Sax
3. Age
4. Civilian Education
5. Tiiae In Service
6. Military Education
7. Primary MQS Score
8. Tine in Primary MOS
9 . NOS Group
10. School Trained NOS
11. Typewriting Experience
12. Teletypewriting Experience
11. Keypunching Experience
14. Class Attendance in TGS* ADP Training
l. Aptitude Area Scores from the Army Classification Battery
15.
CO
(Combat)
16.
FA
(Field Artillery)
17.
EL
(Electronics Repair)
18.
GM
(General Maintenance)
19.
MM
(Mechanical Maintenance)
20.
CL
(Clerical)
21.
GT
(General Technical)
22.
ST
(Skilled Technical)
21.
SC
(Surveillance and Oxiwuni cat ions)
24.
r
(Operations and Food)
STATISTICAL ANALYSIS
The questionnaire data were analyzed with statistical tests of the
goodness-of-f it type that tested the hypothesis that the sasirles of
trainee responses were based upon experiencing an effective and efficient
training program. A set of specified distributions was calculated for
the items in the questionnaires to represent those that would be expected
frees a population who had received effective training in a well-run
8
program. The null hypothesis was that there was no difference between
the observed trainee response distributions and the distributions speci¬
fied for an effective training program. Statistical testa were aude of
the data to determine if there were significant differences between the
observed proportions and the proportions e*j>ected on the basis of an
effective training program. When the actual training program yielded
results that were not as good as the criterion specified for an effec¬
tive program, and where these differences were statistically signifi¬
cant, It was assumed that the actual training was not effective. The
level of significance (alpha) selected for the tests was .05.
T'wo statistical tests were chosen for analyzing the questionnaire
data. The chi-square one-sample test (X^) was used for analyzing data
from items requiring yes-no responses, which produced two categories
of data. The Kolmoqorov-Smi rnov one-sample test (D) was used for ana¬
lyzing data from items calling for one of several choices falling along
a continuous measurement scale.
A multiple regression analysis was run on the demographic data
collected on the enlisted operator trainees to discover variables for
predicting success in TQS* ADF training. Dixon's^ ecssputer program
for stepwise regression analysis (R) was used for making all calculations.
STAFF USER TRAINING EVALUATION
The results are presented in four sections: staff user training,
operator staff user training, commander and principal staff training,
and analysis of demographic variables. The first section presents
findings from the end-of- lesson ratings and evaluations; findings from
ratings and evaluations made at the end of training and testlngj and
class attendance records and results from the final examination.
Er.d-of-Leason Ratings and Evaluatl on a
Trainee Ratings. The staff user trainees were divided into two
classes — an ENS1T class and a FRENSIT class. Each class had a course
of instruction consisting of 10 subjects or lessons. The lessons for
both classes were Identical except for the central lessons, which cov¬
ered ENSIT flies and applications for the ENS1T class and FRENSIT files
and applications for the FRENSIT class. The trainees rated 5 of the
10 lessons at the end of each lesson, which accounted for approximately
60 hours, or 79*, of the training. The five lessons not rated covered
approximately 16 hours, or 21», of the training. Table 4 summarizes
lDixon, w. J. Biomedical Computer Programs. University of California
Tress, 1975.
9
the result* from the end -of- lesson questionnaires, listlnq the lesson
titles and the instructional elements rated by the trainees.
Table 4
Trainee End-of -Lesson Ratings of Staff User Lessons
Instructional elements**
Lesson rated*
Traininq
hours
Classroom
traininq
Hands-on
training
Attaining
lesson
objective
Lesson
time
MIOD operating proc
2
S
S
S
S
Intro TOS ^ messages
6
X
X
X
X
ENSIT applications
34
S
s
s
X
FRENSIT applications
34
s
X
X
X
Functional applications
16
X
X
X
s
File manager
2
••
••
••
"'lessons not rated i ADP doctrine (4 hr): COHOPS (4 hr) : staff Interface
in TOS* (4 hr); graphic applications (2 hr): and GOO ops and Mint (2 hr).
b
S ■ Satisfactory smdian ratinq.
Q
X - Unsatisfactory setdian ratinq.
Trainee Opinions. Table 5 presents trainee cusi nts on the end-of-
lesson questionnaires about the two main subjects of instruction--ENSIT
and FRENSIT files. Of these convents, 86% pertained to sosm aspect of
the hands-on traininq, and 57% of these (494 of the total) pertained
to the insufficient time that was devoted to hands-on traininq and prac¬
tical exercises.
Evaluator Ratings. All lesson elements received a median ratinq
of Effective for classroasi instruction, instructor's ability, and the
overall lesson. The practical exercise sessions were rated as border¬
line on all lessons, even though the lesson objectives were judged to
have been accoeq>lishod.
Evaluator Observations. Generally, evaluator conssenta covered
several problems and conditions that may have affected the traininq
pr oqrttn. Conditions observed to have a negative effect on traininq
were as follcws.
10
Table 5
Trainee Comments on ENSIT and FRENSIT File Cl
V
C
\
u
Number
t lacs made
Percent
distribution
Hoie hands-on training with the MIOO's and
more practical exercises (PE)
38
49
PE's inadequate in content and amount; they
should be embedded in scenario problems
so trainees can experience software inter¬
action with message inputs and outputs
13
17
Less lecturing and classroom instruction
9
12
More varied training on the MIOO's
4
5
Too many students assigned to MIOO's during
hands-on training
2
3
ADP training not necessary for evaluators
3
4
Structure training more toward specific
positions
2
3
More emphasis on format instruction
2
3
More training on interaction of files,
messages, and queries
1
1
Allot more time for teaching FRENSIT subjects
1
1
Replacements should be provided for students
who drop out of training
1
1
Students should be on special duty status
and released from all unit duties
1
1
Total
77
100
1* A high trainee- t o-MIOD ratio (between 3 to 1 and 4 to 1) j
2. A lack of enough assistant instructors during the early part
of training;
3. A great range in the backgrounds and abilities of the trainees;
4. The doubtful effectiveness of using half-day training sessions
(full-day sessions should be used) i
5. The relatively low effectiveness of the combination of lecture
anil hands-on training that was used (a better balance between
lecture and hands-on training is needed) ; and
»•
6. Inadequate support fresn the trainees' unit commanders.
A condition observed as having a positive effect was the profi¬
ciency of the faculty, whose training and instruction quality through¬
out the training program wore judged to be very high in all respects.
Trainee Knd-of -Training and End-of-Teat Ratings, Eva lust ions, and
Pur f ormance Measures
Trainee Ratings. Table 6 summarizes the results from the end-of-
training and end-of-test questionnaires. The table lists the main les¬
sons comprising the staff user training program.
Table 6
Trainee Ratings on Training Proficiency and Lesson Time
End of training*
End of
test*
Lesson
Training Lesson
proficiency time
Training
proficiency
Lesson
time
MIOO operating procedures
X
X
s
s
A OP doctrine
S
s
s
s
COMOPS procedures
X
s
s
s
Intro to TOS^ messages
s
s
s
s
ENSIT applications
s
X
s
s
FRENS1T applications
s
X
s
s
Staff interface in TOS^
X
X
X
s
Graphic applications
X
X
X
s
functional applications
s
s
X
s
GOD ops and maint
X
X
X
s
Pile manager
X
s
X
s
ADP practical experience
—b
X
X
*X ■ Unsatisfactory median rating (less than adequate).
S • Satisfactory median rating.
‘ Mo ratings made on end-of-tralnlng questionnaire as It was administered
subsequent to class training but prior to practical exercise.
12
Oi both the end-of-training and the end-of-teat questionnaire, the
trainees were asked to rate the effectiveness of individual and unit
training In preparing them to perform their duties in the TOS2 field
test. On both questionnaires, the great majority rated the individual
training as effective. On the end-of-test questionnaire, a significant
proportion (D » 2.43, p < .01) of trainees, 40%, rated their prepara¬
tion in unit training as less than effective.
The trainees wore asked to give their overall opinion of the ADP
training. On the end-of-training questionnaire, the majority rated it
as good. On the end-of-test questionnaire, however, a significant
proportion (D * .169, p < .05), 33%, rated the ADP training as less
than good.
Trainee Recommendations. Trainees were asked to make i i i imwiiiiihIii
tions for improving the training program. Their comments are given in
Table 7 for the end-of-training questionnaire and in Table 8 for the
end-of-test questionnaire. In general, approximately half the train¬
ees' recocwendat ions , 47% and 52%, were directed toward expanding or
improving hands-on training, and another large proportion dealt with
other aspects of the traininq. Overall, 76% of the recosmtendations on
the end-of-traming and 63% on the end-of-test questionnaires addressed
the improvement of the training methodology. Another substantial pro¬
portion of the recommendat ions , 20% and 30% on the two questionnaires,
concerned the overall course of instruction, pointed out weaknesses in
the course content, and suggested changes for improvement.
Evaluator Ratings and Recommendations. The evaluators rated the
effectiveness of the overall training program by observing class and
test sessions and recording impressions of these proceedings. Their
observations constitute a general assessment of program effectiveness
and level of student proficiency at the end of the training.
In general, the evaluators stated that students showed a lack of
familiarity with applying the methods and procedures for processing
test problems, seemed somewhat disorganized in their efforts, and were
slow and hesitant in feeling their way toward solutions. With few ex¬
ceptions, the students usod the maximum time for tests and even then
did not complete all the problems. This result indicates a need for
more practice in performance testing to develop the ability to detect
and correct input errors. Evaluators also stated that students need
exposure to test conditions during training, to develop the self-
reliance and problem-solving skills and abilities required for ADP
operations .
Class Attendance. Table 9 presents statistics of trainee daily
classroom attendance for the two ADP training classes. Average class¬
room attendance for the G2 (ENSIT) and G3 (FRENSIT) classes was 90%
and 83%, respectively. Of all trainees, 91% in the ENSIT class and
81% in the FRENSIT attended class at least 81% of the time.
13
Table 7
Trainee Recommendation* at Knd-of-Traininq
Recoi
L'tldat Ion
Number Percent
made distribution
More hands-on, on-line training with the HIOD's
tn practical exercises. 32
Maintain a lower student to MIOD ratio, two-to-one
maximum. 8
Increase practical exercises, minimize lectures. 6
feature more difficult problems to be solved
in practical exercises. 6
More assistant Instructors to help students
during practical exercises. 4
Have smaller classes. 6
Computer system breakdowns Interfered with
training; a backup computer would minimize
these interruptions. 6
Lectures were loo long. 5
Training day should be 8 hours rather than
4 hours. 4
Problem solving In message fonsats classes should
be emphasized by making student select the
correct format on the basis of how, when, and why. 4
A 5-week , half-day course Is not adequate to
train staff users; more training time. 3
Design training to be more self-paced, allow fast
and slow learners to progress at their own pace. 3
Should be more written and performance tests. 3
Should be more training on message formats and
more practice on writing formats. 6
Topics are too general and not oriented to the
specific functions of the position, section
leaders need more training on managing system
problems and functions. 5
More training on the Interpretation, inter¬
relations, and associations of various
messages and queries, 4
Creator emphasis on SOP’s and OFM's since they
regulate ADP play. 3
Staff should be trained to function together. 2
User's manual and SOP should be better Integrated
and made more explanatory. 2
Need better SOP for CM&D, A4P, and ATSE on the
subject of collection manages*nt. 1
The SOTAS needs a supplement. 1
Students should be assigned full time to ADP
training (released from company duties). 4
Students should be assigned to TOS positions
ensssensurate with their MOS . 1
27
7
5
5
3
3
5
4
3
3
3
3
3
J
4
3
2
2
2
1
1
3
1
TOTAL
119
100
Tdt ln 8
Trainee KecunmvndtUoni at End-of-Test
Recummendat ton
Number
made
More hands-on, on-line training with MlOD's,
using drills and practical exercises (PE's). 30
Expand scope of PE's to Include operations tests
of all sections, a complete KTX, a brigade TOC
eaphas lz lug staff and coaasand Interrelationships,
and a CPX with division, brigade, and battalion. 6
Maintain a low student to M10D ratio (?-to-l). 5
Improve quality of PE’s and increase number of
Scenario problems. 4
Devote less time to lectures. 4
Improve system reliability to reduce Inter¬
ruptions of training. 2
More format training with the MIOD's. 1
In lectures, add demonstrations (examples) of
Individual functions. 1
Lectures were boring. 1
More performance tests with specific performance
objectives. 1
More class time; training time is Inadequate. 6
(live more training on following topics:
Applying TOS' principals and using system
capabilities. 4
Interaction of files and processing functions. 1
System operational concepts. 1
Communication procedures, querying, and
purging fllea, and on CONOPS. 1
Interaction between messages files ESD, EUS ,
and CUB. 1
KRENSIT formats DCS, CSR, and RPV reported on
LTOA formats. 1
Communications wiring and troubleshooting. 1
KRENSIT subjects. 1
Pile manager subjects. 1
Division rapid reaction force (DRRF). 1
battalion use of I'STC and ITTDS messages In
changing altuatlona. 1
Need better understanding by staff supervlaors
of ADP functions. 1
Generals and colonels need more training. 1
Need separate training programs for MIOD
operators and staff. 1
MIOD operators need training to higher level
of competence and ability. 1
Students should he assigned full time to ADP
trslning (released from company duties). 3
Establish selection criteria for personnel for
ADP training; essential they have operations
experience. 3
Percent
distribution
34
7
6
5
5
2
1
1
1
1
7
5
1
1
1
1
1
1
1
1
1
1
X
1
1
1
3
3
TOTAL
87
100
Table 9
Trainee Classroom Attendance Records
Attendance
record
ENSIT
class
FRENSIT class
(percent)
Frequency
Percent
Frequency Percent
91-100
36
59
27
50
81-90
19
32
17
31
71-80
4
7
4
7
61-70
0
0
1
2
51-60
1
2
0
0
41-50
0
0
1
2
31-40
0
0
0
0
21-30
0
0
1
2
11-20
0
0
1
2
0-10
0
0
2
4
Total
60
—
54
—
Final F-xaminat ion . Descriptive statistics of the cumulative fre¬
quency distribution of trainee scores on the written and performance
parts of the final examination are presented in Table 10. Statistics
on the cumulative frequency distribution of the number of messages pro¬
cessed correctly on the final examination are presented in Table 11.
The mean number and mean percentage of messages processed correctly
were 8.29 (83%) and 7.21 (66%) for the ENSIT and FRENSIT classes, re¬
spectively. Of the trainees, 60% of the ENSIT class and 49% of the
FRENSIT class made nine or more correct message entries, which was the
assumed system performance criteria required. Tests for significance
of a proportion were made to determine if class performance satisfied
the specification, and results showed that the percentage of students
meeting the standard was significantly less than the hypothetical per¬
centage of 84% for both the ENSIT and the FRENSIT classes (e proportion
• 4.57, p < .001 for ENSIT and e proportion ■ 5.96, p < .001 foi
FRENSIT) .
Discussion
The primary purpose of the training evaluation was to determine
trainee proficiency at the completion of instruction. The training
objective was to bring the students to a level acceptable for perfor¬
mance within system specifications during the testing phases of FM 222.
16
—
Thus, training objectives were used a* the criteria for evaluating
trainee achievement and the relative effectiveness of the instructional
program.
Table 10
Trainee Scores on Final Examination
Raw
score Frequency
91-100 38
81-90 16
71-80 3
61-70 2
51-60 0
0-50 0
Total 59
Mean 91.2
91-100 21
81-90 21
71-80 6
61-70 1
51-60 0
41-50 0
31-40 0
21-30 0
11-20 0
0-10 0
Total 49
Mean 89.1
Written
Performance
Cumulative Cumulative
Percent percent Frequency Percent percent
ENSIT
64
100
27
59
100
27
36
11
24
41
5
9
4
9
17
4
4
3
6
8
0
0
1
2
2
0
0
0
0
0
--
--
46
—
90.3
—
—
FRENSIT
43
100
18
46
100
43
57
1
3
54
12
14
2
5
51
2
2
2
5
46
0
0
4
10
41
0
0
5
13
31
0
0
5
13
18
0
0
1
3
5
0
0
0
0
2
0
0
1
2
2
--
—
39
—
70.8
•-
Note. Standard deviation for ENSIT was 7.10 and 10.13 and for FRENSIT
was 8.37 and 28.45.
Table 11
Messages
Processed
Correct ly
on Final Examinations
ENSIT
FRENSIT
Number
correct
Frequency
Percent
Cumulative
percent
Frequency Percent
Cumulative
percent
11
—
—
—
8
21
100
10
11
27
100
9
23
79
9
13
33
73
2
5
56
8
5
12
40
0
0
51
7
5
12
28
3
8
51
6
5
12
16
1
2
43
S
1
2
4
3
8
41
4
0
0
2
8
21
33
3
1
2
2
2
5
12
2
0
0
0
2
5
7
1
0
0
0
1
2
2
0
0
0
0
0
0
0
Total
41®
100
—
39
100
—
Mote. Standard deviation for ENSIT was 1.68 and for FRENSIT was 3.32.
®Does not Include scores from five students whose message processing
performance was obstructed by computer malfunction.
Trainee Evaluation. In the staff user training program, each staff
user was required to perform an information-processing task by sending
10 basic types of messages to the ccsnputer within specified time limits.
All messages had to be accepted by the computer, thereby indicating that
the entry of all information elements was essentially correct. The
evaluation criterion, therefore, was whether or not each trainee cor¬
rectly entered 10 messages. The percentage of trainees who met this
criterion was the empirical measure used to estimate the degree to which
the training objectives were achieved.
If the criterion of 10 correct message entries is used, and other
factors are not taken into consideration, then 35% of all the trainees
achieved this standard. Examination of the distribution of scores for
the ENSIT class (Table 11), however, reveals that one-third of that
class made 9 correct message entries. The distribution of scores for
the class is positively skewed and dichotomized between those making
9 and 10 correct entries and those making 8 or fewer. The 10-message
18
criterion Joes not conform to the dichotomy end does not reflect the
natural division in performance differences brought about by the test.
The distribution implies a doubt as to whether there are any real dif¬
ferences in proficiency between trainees making a score of 9 or 10 on
the test, whereas there is more confidence that scores of 8 and 9 miqht
indicate meaningful differences. A reluctance to exclude the trainees
who had scores of 9 from consideration stemmed from the fear of maklnq
an error in judgment that miqht result from an appraisal that was in
conflict with the dichotaaxius distribution of scores.
A criterion chanqe to nine or more correct messaqe entries expands
the ranqe to include all scores that could possibly meet the trainmq
objectives. When this criterion is applied, the results show that S4*
of the trainees achieved the standard. This flqure is believed to be
a more accurate indicator of the extent to which the traininq objectives
were achlevod.
Program Evaluation. The assessment of proqram effectiveness in¬
volves consider inq the organirat lonal qoals that the proqram was de-
siqned to accomplish. If 100* criterion achievement was the desired
orqanirational qoal, then the proqram failed to attain its objective
by a wide margin. On the other hand, when the proqram desiqn is studied
it becomes obvious that the conditions necessary for total success were
not established. The primary deficiency was the absence of any pretrain
inq selection process to screen out individuals who lacked the potential
to succeed. All individuals assigned to the proqram were accepted into
it. A few of the trainees had no more than 1 or 4 months total time in
service, and others had little understanding of G2 or G3 operations.
These trainees did not have the necessary background knowledge for
achieving the traininq objectives. The wide diversity in the trainees'
experience, background, and ability was apparent to instructors and
evaluators alike.
This situation carries the implication that the potential for
achieving traininq objectives was a normally distributed variable among
the trainees. This implication, in turn, tends to modify the goal of
the training proqram. If it is assumed that trainee potential was nor¬
mally distributed, and that the short-term, fast-paced tralnir.q proqram
was effective for traininq only trainees whose potential was at least
high as one standard deviation below the mean, then the maximum
proportion of trainees that could be expected to achieve the traininq
objectives would be about 86*. Considering the wide diversity in train¬
ee ability, this level seems to be a more realistic goal. Assessing
the program with this value increases the effectiveness rating to 64*»
this latter flqure appears to provide the most equitable estimate of
the effectiveness of the traininq proqram. The assessment Indicates
that the proqram was fairly effective, but room for improvement remains.
The problem of findlnq ways to improve proqram effectiveness will be
taken up next by evaluating the strengths and weaknesses of the proqram.
Moth trainees and evaluators gave consistently high ratings to
the quality of the program of instruction and the teaching ability of
the instructors on most of the lessons. The course content was well
locumented, in the form of a highly organized, extensively detailed
staff user training manual supported with supplementary volumes of
standing operating procedures (SOP) . Trainees used the trainlnq man¬
ual continuously as a classroom text during lectures and as an opera¬
tors* guide and reference during hands-on training. The subject matter
was comprehensive and covered all aspects of the basic system operations
with which staff users were involved. Most technical matters and staff
user functions seemed to have been thoroughly covered. As an introduc¬
tory course, the program of instruction was satisfactory in covering
the basic functions. The only cosgilaints concerned methods of perform¬
ing in certain staff positions. The lectures were well organized,
clearly presented, and comprehensible. The instructors had a good grasp
of the subject suit ter and were able to provide satisfactory explanations
for most of the questions raised by the trainees. During the lectures,
instructors used slide projectors extensively as visual aids, which
helped to clarify many procedural details. The lecture phase of train¬
ing was used effectively for teaching the large volume of knowledge
comprising the course content.
The trainee and evaluator ratings and comments clearly Identified
three major factors that costoined to substantially diminish the train¬
ing capability inherent in this program. The most Influential factor
was that the trainee load was too heavy for the limited training resour¬
ces made available. The program had the capacity Co train perhaps 70
to 80 trainees under th« training conditions prevailing In FM 222 > how¬
ever, approximately 190 staff users. Including controllers, were admitted
to trainlnq. For FM 222, the classes were simply enlarged to double
their capacity, and the original training schedule was followed without
mod 1 f icat ion •
The second factor was that the aaiount of tisw> scheduled and the
training technique used for the practical exercises were Inadequate.
The common complaint of trainees was that they did not experience
enough hands-on training and could not apply the knowledge learned
from lectures by actually performing data processing operations with
the computerized system. MIOD time allotted for one user had to be
divided up among three or four users. Thus, some trainees actually
experienced only one-fourth the amount of hands-on training they should
have had. The practical exercises were designed for individual learn¬
ing wherein a trainee could develop the important skills necessary for
independent interaction with the computer system. In many sessions,
the opportunity was not there, because the practice became merged into
a group effort. Three or four trainees would gather around one MI00,
and they all would participate in working through a problem that one
of them should have been doing independently. The effects of this lack
of practice surfaced dramatically in the performance exam, where many
trainees experienced great difficulty in structuring the problems and
working them out in a logical or systematic manner.
20
Another week aspect of the practical exercises identified by the
trainees was that there was not enough direct interaction with the TC6
software system in the training. Much of the training was devoted to
the fundamental problems of message format selection, structuring the
ihto the format fields, and data-entry and error-correction opera¬
tions. The trainees obtained little experience in how the TOS software
system processed the input data, updated files, and sent the output
back to the originator in a feedback loop, simulating how the actual
system would function.
.ralnees also acquired little experience in observing how their
inputs affected various files and functions already in the system's
meswry, or how the new information was used with the old information
under a specified set of conditions. The practical exercises did not
include any practice tests similar to the final performance examination;
therefore, until the final performance examination, they never faced
hands-on test situations involving difficult problems that would expose
their inadequacies of knowledge and skill. The perfors^nce examination
was their first experience with operating the system without assistance
from others. A performance examination scheduled to be given on the
first examination during the second week of tralnlnq was cancelled be¬
cause of computer breakdown; thus the trainees were deprived of valuable
information about their ability for lnde(>*ndent operation.
The third factor was that significant percentages of trainees felt
that too little time was allotted to teaching the main lessons on the
ENSIT and FRENSIT files. These lessons covered a large asiount of sub¬
ject matter in a relatively short time. The pace of instruction wss
fast, and emphasis was placed on the fundamentals of each topic. Less
complete coverage was given to the higher- level and store intricate
application of functions during tacticsl operations. Trainees indi¬
cated that they desired store training time for learning the main sub¬
jects and subjects dealing with functional applications and information
management .
OPERATOR STAFF USER TRAINING EVALUATION
These findings present results from measures of operator staff
user training taken at different times in the test program. Ratings
and evaluations were made at the end of each lesson, at the end of the
training program, and again at the end of the T06^ field test.
End-of -Lesson Ratings and Evaluations
The training program for operator staff users wss a subordinate
part of the staff user program of Instruction. It consisted of four
lessons taught to a selected group of approximately 57 enlisted staff
users, with the purpose of teaching operators the skills involved in
moving and setting up the MIOD for operation.
Trainee Ratings. Table 12 presents a suwnary of the end-of- lesson
questionnaire results. On the items dealing with the effectiveness of
instruction, the trainee median rating of the quality of instruction
was effective or very effective on all four lessons. On the items con¬
cerned with the practical exercise part of the lessons, the trainees
felt they had received adequate practice in two of the lessons, and in¬
adequate practice on the other two. Forty-three percent of the train¬
ees indicated they had insufficient practice on the digital data termi¬
nal. On the lesson on communication methods, 30% Indicated they had
insufficient practice on the equipment. Most of the trainees felt that
they had accomplished the lesson objectives on all four lessons and
that the time allotted for each lesson was adequate.
Table 12
End-of -Lesson Rating of Instruction
_ _ Instructional elements _
Attaining
Practice lesson Lesson
Lesson Instruction session Objectives time
Assembly and disassembly
of cabling
Digital data terminal
Operator maintenance and
troubleshooting
Communication methods
A
A
AAA
X A A
AAA
A X A
A
A
Note . A indicates adequate; X indicates inadequate.
Trainee Opinions. Table 13 presonts a classif ication and tabula¬
tion of trainee remarks in the end-of-lesson questionnaire covering the
four lessons. The most frequent remark made was that there were too
many trainees for the number of MIOO's available, and the third most
frequent was that they needed more practical exercise time on the MIOD.
These two directly related problems accounted for 52% of all the comments.
Evaluator Ratings. A summary of the median ratings of the lesson
elements for the four lessons for selected MIOD operators is shown in
Table 14.
22
Table 13
Trainee (.'amenta on MIOO Operator Lessons
Remark
Number
made
Percent
dlstr Ibut ion
Too many trainees per MIOO; class too large
for amount of resources available
14
33
Distractions in classroom degraded learning
conditions
9
21
Need more practical exercise on the equipment
8
19
Items of equipment nonoperat lonal or not
aval lable
4
9
Need more detailed instruction on specific
topics
3
7
Assistant instructors not available to
trainees needing guidance
3
7
Instructor was a good teacher
3
4
Total
43
100
Table 14
Evaluator Ratings of MIOO Operator Lessons
Lesson element
Median rating
Lecture session
Effective
Practical exercise session
Borderline
Instructor's ability
Effective
Total lesson
Effective
Lesson objective accomplished
Yes
Evaluator Observations . In general, the evaluatora noted the same
problems identified by the trainees — too many trainees for the nvsaber
of MIOD's available and Insufficient practical exercise time on the
MIOD's. There were 18 HIOO's available and 60 trainees, for a student-
to- MIOO ratio of 3.3 to 1. The class instructor, who had worked with
23
the TOS system for several years and was familiar with the technical
aspects of TOS training, held the definite opinion that the ideal train-
ee-to-MIOO ratio is 1 to 1 and should not be higher than 2 to 1 . In
addressing the practical exercise time, the evaluators stated that,
with three or four trainees assigned to each MIOD, it was difficult
for everyone to obtain an adequate amount of time for practice.
End-of-Training and End-of-Toat Ratings. In general, the end-of-
training and end-of-tcst questionnaires covered the same four lessons
as listed in Table 12. At the end of the training, 40* of the trainees
felt that there should have been more time devoted to training on as¬
sembly and disassembly of cabling. At the end of the test, 53% felt
that more time should have been given to training on operator main¬
tenance and troubleshooting.
Discussion
The purpose of this subordinate evaluation was to appraise the
adequacy of the operator staff user training in preparing the trainees
to perform within Bystem specifications during the testing phases of
FM 222. The evaluation indicated that the overall objective of the
training was attained; however, the lack of sufficient time and equip¬
ment adversely affected training quality.
COMMANDER AND PRINCIPAL STAFF TRAINING EVALUATION
On the items dealing with effectiveness of instruction, the
trainee median rating was Effective and the evaluators rated the lec¬
ture session, instructor's ability, and the total lesson as Effective.
The majority of trainees stated that the time devoted to each lesson
was adequate for learning; however, the evaluators rated the practical
exercise session as borderline. The lesson objectives for all lessons
were judged to have been accomplished.
Although this evaluation indicated that the lesson objectives were
attained and that the training of the commanders and principal staff
was satisfactory overall, the problem of insufficient practical exer¬
cise, as noted in the other training phases, was also noted in this
phase of training.
ANALYSIS OF DEMOGRAPHIC VARIABLES
Data from all the enlisted trainees in the ENSIT and FRENSIT
classes who completed the final examination were analysed in a step¬
wise regression analysis. The sample site was 48 subjects. Table 3
lists the 24 biographical measures and aptitude scores Included in the
analysis as independent variables. The dependent variable was the
sum of the normalized scores from the written and performance portions
of the final examination.
Hesulta
Table 15 presents the results of the stepwise regression analysis.
Of the 24 independent variables, 4 were found to have significant, posi¬
tive correlations with the examination score. The ccsnbination of all
four variables accounted for a significant proportion of the total
variance in the examination scores (F ■ 11.72, df ■ 4/42, p < .001).
The value of the coefficient of determination R2 equaled .5275, which
means that approximately 53* of the variance of the examination scores
was accounted for by the four attribute variables in combination — the
CT. primary MOS , CO, and SC.
Table 15
Summary of Stepwise Regression Analysis for Data from
MIOD Operators in ENSIT and FRENSIT Classes
Variable entered
df
F to enter
Increase in R*
GT (general technical)
1/42
21.15
.3198
Primary MOS score
1/42
8.26
.1075
CO (combat)
1/42
4.44
.0536
SC (surveil lance/comm)
1/42
4.15
.0466
Value of R2
••
••
.5275
The last column in Table 15 gives the proportions of variance due
to the individual variables. These results show that when the four
variables are taken together the GT score is the most powerful pre¬
dictor and accounts for about 32* of the variance. The primary MOS
score accounts for about 11* of the variance, and the combat (CO) and
surveillance and communications (SC) scores account for about 5* each
of the variance. It appears that all four attribute variables are
meaningful predictors of MIOD operator training proficiency. Since
the effort and expense are virtually equivalent for obtaining any one
of these scores, or all four, it would be advantageous to use them all
to maximize predictive power.
25
Discussion
This analysis suffers frcan one major weakness: The sample size
may not have been large enough to insure high reliability of the re¬
sults. (The method of multiple regression analysis is known for its
low reliability, and that problem is minimized by using large samples.)
Ker linger and Pedhazur^ noted that some authors recommend that the
sample size be at least 30 subjects per attribute variable. In this
analysis, the advice calls for a sample size of 120 rather than the 48
that was used. Since our sample may not have been sufficient, the re¬
sults should be viewed as preliminary findings that have uncovered four
attribute variables that show promise as predictors of aptitude for
TOS operator staff user training.
^Ker linger , F. N. , fc Pedhazur, E. J. Multiple Regression in Behavioral
Research. New York: Holt, Rinehart, and Winston, 1973.
26