By Michael M. Reischman
engineers start talking metrics, they may be referring to a different
way to determine research trends.
needs to be a better way to quantify trends in engineering research.
The trends, when taken at face value, can be used by legislators,
prospective students, and parents as an indicator of the potential
of a particular discipline or college. The health and vitality of
the engineering research enterprise has been brought into question
over the past two years in studies done by the National Research
Council's Board on Science, Technology, and Economic Policy. And
some of the results are very dramatic. For example, between 1993-99,
NRC found that federal research funding for mechanical engineering
and electrical engineering declined by 40 percent and 31 percent,
respectively. But engineering research administrators around the
country question these and other results from the studies. In fact,
upon examination of the data sources, some enlightening observations
can be made.
data most commonly used to establish engineering research trends
in various disciplines comes from three sources. And therein lies
the problem. The most troublesome source is the NSF-generated federal
obligation survey. Data for this survey comes from an analysis made
by funding agency program managers and represents their assessment
of where the money goes. The other two data sources are research
expendituresone is based on NSF surveys of university financial
offices; the other comes from ASEE's annual survey of engineering
colleges. The federal obligation data, unfortunately the source
for NRC's last two studies, is the most widely quoted and the basis
for the supposed precipitous declines in research funding for mechanical
and electrical engineering. Neither NSF nor ASEE expenditure data
demonstrates similar results. In fact, expenditure data indicates
that the research enterprise at most colleges and engineering departments
is growing at a rate of about 3 to 5 percent per year.
administrators uniformly recognize that the trends portrayed by
expenditure data are more representative of reality than the obligation
data trends. Four factors contribute to the differences, the first
of which is that the growth in interdisciplinary research is not
reflected in the assignment of obligated funds to traditional disciplines.
Another problem is that the fiscal year data for obligations and
expenditures don't always match up. A third factor is that obligation
data refers almost entirely to research funds, whereas expenditures
include all R&D funds. And finally, the categorization of obligated
research funds within funding agencies can be inconsistent.
use of metrics to measure success or failure can be valuablebut
only if the results are interpreted correctly. In the case of obligation
data, harm is done when the public is misled into thinking that
engineering research is declining in some disciplines. Student recruitment,
at both the graduate and undergraduate level, can be affected as
a result. A parent or a bachelor's degree recipient would look apprehensively
at a discipline where federal research funding is seriously decreasing.
Likewise, misleading metrics can affect legislators' funding decisions.
are valuable because they establish a standard that permits comparisons
of like entities. But the measurement is no better than the data.
Research Council stands squarely behind the use of multiple metrics
to characterize the engineering research enterprise. A sampling
of meaningful metrics includes: graduate student enrollments, graduate
degrees, and research expenditures and obligations at the federal,
state, and industrial levels. The data to elucidate trends in these
and other metrics is readily available. It is simply imprudent to
not use it to the fullest extent possible.
M. Reischman is a member of the Engineering Research Council.
He can be reached at email@example.com.