 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
 |
- By Michael M. Reischman
When engineers start talking metrics, they may be referring to a different way to determine research trends.
There needs to be a better way to quantify trends in engineering research. The trends, when taken at face value, can be used by legislators, prospective students, and parents as an indicator of the potential of a particular discipline or college. The health and vitality of the engineering research enterprise has been brought into question over the past two years in studies done by the National Research Council's Board on Science, Technology, and Economic Policy. And some of the results are very dramatic. For example, between 1993-99, NRC found that federal research funding for mechanical engineering and electrical engineering declined by 40 percent and 31 percent, respectively. But engineering research administrators around the country question these and other results from the studies. In fact, upon examination of the data sources, some enlightening observations can be made.
The data most commonly used to establish engineering research trends in various disciplines comes from three sources. And therein lies the problem. The most troublesome source is the NSF-generated federal obligation survey. Data for this survey comes from an analysis made by funding agency program managers and represents their assessment of where the money goes. The other two data sources are research expenditures—one is based on NSF surveys of university financial offices; the other comes from ASEE's annual survey of engineering colleges. The federal obligation data, unfortunately the source for NRC's last two studies, is the most widely quoted and the basis for the supposed precipitous declines in research funding for mechanical and electrical engineering. Neither NSF nor ASEE expenditure data demonstrates similar results. In fact, expenditure data indicates that the research enterprise at most colleges and engineering departments is growing at a rate of about 3 to 5 percent per year.
Research administrators uniformly recognize that the trends portrayed by expenditure data are more representative of reality than the obligation data trends. Four factors contribute to the differences, the first of which is that the growth in interdisciplinary research is not reflected in the assignment of obligated funds to traditional disciplines. Another problem is that the fiscal year data for obligations and expenditures don't always match up. A third factor is that obligation data refers almost entirely to research funds, whereas expenditures include all R&D funds. And finally, the categorization of obligated research funds within funding agencies can be inconsistent.
The use of metrics to measure success or failure can be valuable—but only if the results are interpreted correctly. In the case of obligation data, harm is done when the public is misled into thinking that engineering research is declining in some disciplines. Student recruitment, at both the graduate and undergraduate level, can be affected as a result. A parent or a bachelor's degree recipient would look apprehensively at a discipline where federal research funding is seriously decreasing. Likewise, misleading metrics can affect legislators' funding decisions.
Metrics are valuable because they establish a standard that permits comparisons of like entities. But the measurement is no better than the data. ASEE'sEngineering Research Council stands squarely behind the use of multiple metrics to characterize the engineering research enterprise. A sampling of meaningful metrics includes: graduate student enrollments, graduate degrees, and research expenditures and obligations at the federal, state, and industrial levels. The data to elucidate trends in these and other metrics is readily available. It is simply imprudent to not use it to the fullest extent possible.
Michael M. Reischman is a member of the Engineering Research Council.
He can be reached at mreischman@asee.org.
More...
