Limitations of Analytic Results

Any study of this nature is necessarily limited in several
respects. First of all, the survey approach used here is but one of
several that can be used to inform decisions about extending the
measurement of analytical abilities. Tucker’s (1985) results provide
useful information from different perspectives--those of cognitive
psychologists and philosophers. Other approaches that might also be
informative include the methods of cognitive psychology, which could
be used not only to supplement but also to extend the survey results
reported here. These methods would seem especially appropriate
because they relate more directly to actual skills and abilities than
to perceptions.
Second, the diversity that characterizes graduate education
renders the results of this study incomplete. Some clues have been
gained as to similarities and differences among a limited sample of
graduate fields. However, the substantial differences found among
fields are a source of concern, since we cannot be certain whether or
not some other sample of fields might exhibit even greater variation.
Finally, as several survey respondents pointed out, many of the
reasoning skills about which we asked are expected to, and do, improve
as the result of graduate study. In some sense these skills may
represent competencies that differ from, say, the verbal skills
measured by the GENEG eneral Test in the respect that these analytical
skills may develop much more rapidly. A question of interest, then,
is how to accommodate the measurement of these skills in the context
of graduate admissions testing, which currently focuses on the
predictive effectiveness of abilities that are presumed to develop
slowly over a significant period of time.
Future Directions
The study suggested several possible future directions. Because
of the substantial variation among fields, one possibility would
involve extending the survey to include additional fields of graduate
study. Some refinements could now be made on the basis of past
experience. For example, ratings of the frequency with which skills
are used, as well as the frequencies of errors and critical incidents,
could probably be omitted without much loss of information. OII the
other hand, it would seem desirable to add categories allowing ratings
of the differential importance of various reasoning skills at
different stages of graduate education, ranging from entry level to
dissertation writing.
Finally, based on the reasoning skills identified as most
important, criterion tasks might be developed against which the
validity of the current GRE analytical measure could be gauged. This
strategy would make especially good sense for those important skills
that may not be measurable in an operational test like the GRE General
Test, but which might correlate highly with the abilities now measured
by the test. One specific possibility would be the development of
rating forms, which could be used by faculty to rate the analytical
abilities of their students. These ratings could then be used as a
criterion against which GRE analytical scores could be judged.