Index


Chemical and Biological Defense: Program Planning and Evaluation Should
Follow Results Act Framework (Letter Report, 08/16/1999,
GAO/NSIAD-99-159).

Pursuant to a congressional request, GAO examined the extent to which
the Department of Defense (DOD) has applied the Government Performance
and Results Act's outcome-oriented principles to the Chemical and
Biological (CB) Defense Program, focusing on whether: (1) CB Defense
Program goals are explicit and measurable; (2) the CB Defense Program
has performance measures that assess outcomes and impacts rather than
outputs and activities; and (3) organizations executing the CB Defense
research, development, testing, and evaluation (RDT&E) activities have
incorporated Results Act principles in their program planning and
evaluation.

GAO noted that: (1) DOD's CB Defense Program in general, and its RDT&E
activities in particular, have not incorporated key Results Act
principles, as evidenced by the fact that the goals of the program are
vague and unmeasurable and do not articulate specific desired impacts;
(2) in the absence of explicit and measurable goals, it is difficult to
assess whether the program has been successful in achieving its goals;
(3) the performance measures of CB Defense Program RDT&E emphasize
activities rather than impacts; (4) the program is not being evaluated
according to its impact on the defensive or operational capabilities of
U.S. forces, either individually or collectively; (5) CB Defense Program
planners use roadmaps to track program progress toward meeting chemical
and biological defense goals; (6) these goals frequently take the form
of advanced concept technology demonstrations; (7) however, the
demonstration of a new defensive technology or capability is not a
measure of the program's impact or contribution to the military's
ability to survive, fight, and win in chemical and biological
environments; (8) CB Defense Program research and development
organizations have incorporated Results Act principles inconsistently;
(9) only one organization had adopted the Results Act planning and
evaluation tools; and (10) the remaining research and development
organizations cited either the utilization of equivalent planning tools
or the unique challenges of evaluating research and development
activities as reasons why they had not or could not adopt the Results
Act processes.

--------------------------- Indexing Terms -----------------------------

 REPORTNUM:  NSIAD-99-159
     TITLE:  Chemical and Biological Defense: Program Planning and
	     Evaluation Should Follow Results Act Framework
      DATE:  08/16/1999
   SUBJECT:  Military research and development
	     Performance measures
	     Biological warfare
	     Strategic planning
	     Chemical warfare
	     Defense capabilities
	     Program evaluation
IDENTIFIER:  DARPA Chemical and Biological Defense Program

******************************************************************
** This file contains an ASCII representation of the text of a  **
** GAO report.  This text was extracted from a PDF file.        **
** Delineations within the text indicating chapter titles,      **
** headings, and bullets have not been preserved, and in some   **
** cases heading text has been incorrectly merged into          **
** body text in the adjacent column.  Graphic images have       **
** not been reproduced, but figure captions are included.       **
** Tables are included, but column deliniations have not been   **
** preserved.                                                   **
**                                                              **
** Please see the PDF (Portable Document Format) file, when     **
** available, for a complete electronic file of the printed     **
** document's contents.                                         **
**                                                              **
** A printed copy of this report may be obtained from the GAO   **
** Document Distribution Center.  For further details, please   **
** send an e-mail message to:                                   **
**                                                              **
**                    <info@www.gao.gov>                        **
**                                                              **
** with the message 'info' in the body.                         **
******************************************************************
Rev-LG logo.eps GAO United States General Accounting Office

Report to Congressional Requesters

August 1999 CHEMICAL AND BIOLOGICAL DEFENSE

Program Planning and Evaluation Should Follow Results Act
Framework

GAO/NSIAD-99-159

Page 1 GAO/NSIAD-99-159 Chemical and Biological Defense United
States General Accounting Office Washington, D. C. 20548 National
Security and

International Affairs Division

B-282699 Letter August 16, 1999 The Honorable Robert C. Byrd
Ranking Minority Member Committee on Appropriations United States
Senate

The Honorable Carl Levin Ranking Minority Member Committee on
Armed Services United States Senate

Since the Persian Gulf War, Members of Congress have raised
concerns regarding the adequacy of technology used by the
Department of Defense (DOD) to detect, identify, prepare for, and
protect troops against chemical and biological (CB) weapons. 1 In
1993, the National Defense Authorization Act for Fiscal Year 1994
(P. L. 103- 160) directed the Secretary of Defense to take actions
designed to improve the Department's CB defense capabilities,
including coordination and integration of all CB defense programs
into

what is now the CB Defense Program. More recently, concerns that
terrorists might move beyond using conventional weapons to
chemical or biological devices led Congress to authorize the
federal government to

improve domestic capabilities to respond to such incidents. With
the initiation of these domestic preparedness programs in fiscal
year 1997, federal research and development efforts to develop
nonmedical CB defense technology expanded considerably, and they
continue to grow. 2 According to the White House, the President's
fiscal year 2000 budget request includes over $10 billion to
combat terrorism. Almost $1.4 billion is for programs specifically
aimed at terrorist threats from chemical, biological,
radiological, or nuclear weapons, an amount which exceeds the

1 See Chemical and Biological Defense: Emphasis Remains
Insufficient to Resolve Continuing Problems (GAO/NSIAD-96-103,
Mar. 29, 1996) and Chemical Weapons: DOD Does Not Have a Strategy
to Address Low- Level Exposures (GAO/NSIAD-98-228, Sept. 23,
1998). 2 Nonmedical technologies refer to technologies for
detecting, identifying, protecting against, or decontaminating
personnel and equipment of chemical and biological agents. By
contrast, examples of medical research and development include the
development of prophylactics such as vaccines, medical diagnostics
for determining exposure to chemical or biological agents, and
therapeutic drugs or procedures for countering the effects of
exposure.

Letter

B-282699 Page 2 GAO/NSIAD-99-159 Chemical and Biological Defense
funding of less than $1 billion for military programs to counter
chemical and biological threats. In 1993 Congress enacted the
Government Performance and Results Act (commonly referred to as
the Results Act). The legislation was designed to have agencies
focus on the performance and results of their programs rather than
on program activities and resources, as they had traditionally

done. Congress sought to shift federal management and oversight
from its preoccupation with program staffing, activity levels, and
tasks completed to program results that is, to the real difference
that federal programs make in people's lives. Congressional
reports and administrative guidance indicate that programs such as
the CB Defense Program should follow the Results Act's outcome-
oriented principles, including the establishment of general goals
as well as quantifiable, measurable, outcome- oriented performance
goals and related measures.

As you requested, we examined the extent to which DOD has applied
the Results Act's outcome- oriented principles to the CB Defense
Program, focusing in particular on research, development, testing,
and evaluation

(RDT& E) activities that lead to new technologies and defensive
capabilities. Specifically, we assessed whether (1) CB Defense
Program goals are explicit and measurable, (2) the CB Defense
Program has performance measures that assess outcomes and impacts
rather than outputs and activities, and (3) organizations
executing the CB Defense

RDT& E activities have incorporated Results Act principles in
their program planning and evaluation. A companion report Chemical
and Biological Defense: Coordination of Nonmedical Chemical and
Biological Research and Development Programs (GAO/NSIAD-99-160,
Aug. 16, 1999) examines coordination on nonmedical CB defense
research and development programs. Results in Brief DOD's CB
Defense Program in general, and its RDT& E activities in
particular, have not incorporated key Results Act principles, as
evidenced by the fact that the goals of the program are vague and
unmeasurable and do not articulate specific desired impacts.
Program planners do not explain, for example, the meaning of goals
such as denying military

advantage or allowing U. S. forces to operate largely unimpeded by
chemical and biological attacks. In the absence of explicit and
measurable goals, it is difficult to assess whether the program
has been successful in achieving its goals.

Letter

B-282699 Page 3 GAO/NSIAD-99-159 Chemical and Biological Defense
The performance measures of CB Defense Program RDT& E emphasize
activities rather than impacts. The program is not being evaluated
according to its impact on the defensive or operational
capabilities of U. S. forces, either individually or collectively.
CB Defense Program planners use roadmaps to track program progress
toward meeting

chemical and biological defense goals. These goals frequently take
the form of advanced concept technology demonstrations. 3 However,
the demonstration of a new defensive technology or capability is
not a measure of the program's impact or contribution to the
military's ability to survive, fight, and win in chemical and
biological environments. For example, these

technology demonstrations may still need additional engineering
and manufacturing development or product and concept development,
as well as successful operational testing, before production
begins and warfighters are equipped. CB Defense Program research
and development organizations have incorporated Results Act
principles inconsistently. Only one organization has adopted the
Results Act planning and evaluation tools. The remaining research
and development organizations cited either the utilization of
equivalent planning tools or the unique challenges of evaluating
research and development activities as reasons why they had not or
could not adopt the Results Act processes. We are recommending
that the Secretary of Defense take actions to develop a
performance plan for the CB Defense Program based on the

outcome- oriented management principles embodied in the Results
Act. Background Consistent with the National Defense Authorization
Act for Fiscal Year 1994, 4 the Secretary of Defense assigned
responsibility for the overall coordination and integration of the
CB Defense Program to a single office, the Deputy Assistant to the
Secretary of Defense for Counterproliferation and Chemical/
Biological Defense. The office is responsible for approving

all planning, programming, and budgeting documents, ensuring 3
Advanced concept technology demonstrations assess the military
utility of mature technologies and their capabilities in realistic
operational scenarios. CB defense capabilities that have been
explored through these technology demonstrations include the
capability to (1) provide early warning of remote biological
warfare agents; (2) detect, warn, dewarn, identify, protect, and
decontaminate air bases and seaports against biological attack;
and (3) integrate biological and chemical detection and early
warning capability at an air base or seaport. 4 P. L. 103- 160,
sec. 1701.

B-282699 Page 4 GAO/NSIAD-99-159 Chemical and Biological Defense
coordination between the medical and nonmedical CB defense
efforts, and overseeing management oversight in accordance with
the law. Several organizations within DOD conduct RDT& E
activities on behalf of the CB Defense Program. These include the
Defense Advanced Research Projects Agency (DARPA), the Soldier and
Biological Chemical Command, and the

Joint Program Office for Biological Defense. In addition, CB
defense research and development is conducted at the national
laboratories of the Department of Energy (DOE), funded both by DOD
and DOE. The program addresses five defensive capabilities, three
of which are nonmedical: contamination avoidance, protection, and
decontamination, 5 as well as medical chemical defense and medical
biological defense. These

areas comprise the framework that DOD uses to formulate nonmedical
CB Defense Program requirements. When doctrinal, training, or
organizational solutions cannot satisfy warfighters' needs in
these areas, DOD seeks new equipment through the research,
development, and acquisition cycle. 6 CB defense funding is
divided between the program's two primary activities: RDT& E and
procurement. Of the CB Defense Program budget of $717 million
proposed for fiscal year 2000, $340 million (47 percent) would be
for RDT& E and the remaining $377 million (53 percent) for
procurement.

The Results Act is the primary legislative framework through which
agencies, at all levels, are required to set strategic goals,
measure performance, and report on the degree to which goals are
met. The outcome- oriented principles of the Results Act, which
Congress anticipated would be institutionalized and practiced at
all organizational levels in federal agencies, include (1)
establishing general goals and quantifiable, measurable, outcome-
oriented performance goals and related measures; (2) developing
strategies for achieving the goals, including

strategies for overcoming or mitigating major impediments to goal
achievement; (3) ensuring that goals at lower organizational
levels align with and support general goals; and (4) identifying
the resources that will be required to achieve the goals. 5
Contamination avoidance includes detecting, avoiding, and
bypassing contaminated areas; protection consists of individual
and collective protection; decontamination is the restoration of
combat power after a CB attack. 6 DOD categorizes RDT& E into five
budget activities: basic research (6. 1 account), applied research
(6. 2), advanced technology development (6. 3), demonstration/
validation (6. 4), and engineering and manufacturing development
(6. 5). DOD refers to activities 6. 1 to 6.3 as Defense Science
and Technology.

B-282699 Page 5 GAO/NSIAD-99-159 Chemical and Biological Defense
In its guidance on Results Act implementation, the Chief Financial
Officers Council advised agencies that to comply with the spirit
and intent of the act, the goals and measures used at lower
organizational levels should be linked with the agency's strategic
goals. The Quadrennial Defense Review (QDR) serves as DOD's
overall strategic

planning document. The QDR directs DOD organizations at all levels
to review their strategic and mission objectives in order to
ensure that they link to the goals and objectives of the QDR and
that Results Act performance plans indicate progress toward
meeting QDR goals. DOD implementing guidance states that the
goals, objectives, measures of success, quantifiable performance
measures, and program outcome evaluations of subordinate
organizations should be linked to the DOD corporate goals
articulated in the QDR and made operational in the DOD performance
plan. DOD does not routinely link its performance measures to
specific organizational units or individuals, which have
sufficient flexibility, discretion, and authority to accomplish
desired results. DOD's performance plan for fiscal year 2000 does
not specifically discuss the CB Defense Program. (CB Defense
Program RDT& E activities are aggregated with those of other
modernization activities to support DOD's second corporate goal to
prepare now for an uncertain future by pursuing a

focused modernization effort that maintains U. S. qualitative
superiority in key warfighting capabilities.) Congressional
reports and administrative guidance indicate that programs such as
the CB Defense Program should follow the outcome- oriented
principles of the Results Act. In our assessment of the adherence
of the program to the act, we only evaluate of the first of four
principles establishing general goals and quantifiable,
measurable, outcome- oriented performance goals, and related
measures. 7 7 Tasks required for implementing the first principle
include (1) identifying the organization's mission

and long- term strategic goals, (2) describing how the
organization's annual performance goals are related to its long-
term goals, (3) specifying annual performance goals for each
program activity, (4) identifying the performance measures the
organization will use to assess its progress, and

(5) describing how data will be verified and validated.

B-282699 Page 6 GAO/NSIAD-99-159 Chemical and Biological Defense
Goals of the CB Defense Program Are

Vague and Unmeasurable and Do Not Articulate Specific Desired
Impacts

Although DOD has taken the initial and necessary step of
articulating CB Defense Program goals, the goals are not
articulated in a manner consistent with Results Act principles.
The stated goals are vague and unmeasurable, and they fail to
articulate specific desired impacts. A Results Act framework
requires that managers define a related set of long- term
strategic goals, annual agency goals, and measurable performance
goals for each program. The five CB Defense Program goals are to
1. deter CB weapon use by denying military advantage to an enemy

through a combination of avoidance, protection, decontamination,
and medical support capabilities, allowing U. S. forces to operate
largely unimpeded by chemical and biological attacks and their
subsequent effects;

2. address the most probable CB weapon threats that could be
encountered in regional conflicts and field capabilities to the
forces required for two major theater wars; 3. ensure the CB
weapon Threat Evaluation Projection drives CB defense research,
development, and acquisition programs;

4. emphasize a joint service approach to CB defense research and
development, and acquisition; and 5. complete critical RDT& E and
acquisition of improved CB detection, identification, and warning
systems; individual and collective protection systems; and medical
support and decontamination systems.

Measuring the first goal is unachievable, determining a deterrence
effect is problematic, and attributing the specific rationale for
the deterrence is unrealistic. The second, third, and fourth goals
address the size, focus, and coordination of the program not
program outcomes. Together, these goals direct that the program be
sufficiently large to address the needs resulting from two major
theater wars; sufficiently focused to address the likely validated
threats; and sufficiently coordinated to capitalize on
efficiencies and other benefits of joint requirements
determination, research, development, and acquisition. The
objective of the fifth goal is measurable but speaks to program
outputs without addressing program

outcomes or impacts (such as decreased defensive vulnerabilities
or increased operational capabilities). The completion of RDT& E
or procurement cannot be assumed to result in a positive impact on
the

B-282699 Page 7 GAO/NSIAD-99-159 Chemical and Biological Defense
defensive posture or operational flexibility of U. S. forces.
While the completion of these activities may well generate
benefits for U. S. troops, in the absence of valid, reliable
measures, the contributions of RDT& E or procurement cannot be
determined. CB Defense Program Performance Measures Emphasize
Activities Rather Than Outcomes and Impacts The CB Defense Program
is not evaluated on the impact of its activities on the defensive
or operational capabilities of U. S. forces, either individually

or collectively. CB Defense Program planners use roadmaps, Defense
Technology Objectives (DTO) and advanced concept technology
demonstrations (ACTD) to assess progress toward goals. Program
planners collectively prepare a number of strategic plans they
describe as in the spirit of the Results Act, if not specifically
for the purpose of assessing outcomes and impacts. For example,
DOD's Nuclear, Biological, and Chemical (NBC)

Defense Annual Report to Congress 8 and the Joint Service NBC
Defense Research, Development, and Acquisition Plan are updated
annually and include detailed metrics and time lines reflecting
the performance of the program (such as the demonstration of a new
technology). Roadmaps track program progress toward DTOs that,
when achieved, DOD claims will

create new operational capabilities. A number of DTOs are ACTDs,
and plans state that technology demonstrations provide a means for
the rapid field testing of technical options to solve operational
needs. CB Defense Program roadmaps explicitly link the completion
of DTOs and ACTDs with an increase in the demonstrated warfighting
capabilities of U. S. forces. In addition, CB Defense Program
planners cited ongoing programmatic peer reviews, such as
Technology Area Reviews and Assessments (TARA), as additional
means to measure progress toward meeting program goals. We do not
concur that the conduct of an ACTD or a peer review of ongoing
work measures the impact of the CB Defense Program on the
warfighter. Both measures have limitations that make them
inappropriate for

appraising progress toward achieving program objectives. ACTDs
represent a means for rapidly introducing new technologies and
reducing the time from the start of a program to the system's
initial operational capability. However, the demonstration of a
new technology may not by itself result in the effective and safe
deployment of a military capability in 8 Submitted to Congress
annually pursuant to 50 U. S. C. 1523.

B-282699 Page 8 GAO/NSIAD-99-159 Chemical and Biological Defense
support of the warfighter. Moreover, as we previously reported,
DOD has not always emphasized the need to complete concept and
product development or testing before production, thus increasing
the risk of approving ACTDs in support of CB defense that include
immature technologies and then prematurely starting production. 9
Technology Area Review Assessments Are Not a

Measure of Program Impact TARAs are peer reviews conducted by the
Director, Defense Research and Engineering on each of DOD's 12
science and technology programs one

being, CB defense. TARAs address progress toward achieving DTOs
and form the basis of the Results Act rating for DOD's performance
in science and technology. 10 However, the application of the TARA
to generate performance measures of DOD's science and technology
programs such as CB defense is limited by several factors. First,
the scope of the TARA Results Act ratings is limited because TARAs
only address DTOs. Funding for DTOs comprises less than 50 percent
of total funding for applied and advanced technology research and
development. Thus, the Results Act ratings do not capture the
majority of the CB Defense Program's RDT& E activities. Second,
the focus of TARAs is on budgets, schedules, and technical
performance. TARAs do not measure technology transition from the
laboratory to the battlefield. Lastly, TARAs do not measure

improvements in the ability of U. S. troops to survive, fight, and
win in a CB environment. CB Defense Program Research and
Development Organizations Have Incorporated Results Act Principles
Inconsistently

The organizations that execute or contribute to the research and
development goals of the CB Defense Program vary in their use of
the Results Act principles to plan and assess their activities. 9
Defense Acquisition: Advanced Concept Technology Demonstration
Program Can Be Improved (GAO/NSIAD-99-4, Oct. 15, 1998).

10 In fiscal year 1999, there were 23 chemical and biological
defense DTOs, many in the form of ACTDs.

B-282699 Page 9 GAO/NSIAD-99-159 Chemical and Biological Defense
The Soldier and Biological Chemical Command Is the Only RDT& E
Organization to Systematically Apply Results Act Principles

The Soldier and Biological Chemical Command has demonstrated that
the Results Act principles can be integrated into the planning and
evaluation process of an organization conducting research and
development for the CB Defense Program. 11 In 1996, the Chemical
and Biological Defense Command (which in fiscal year 1999 merged
with the Soldier Systems Command to form the Soldier and
Biological Chemical Command) developed a Business Planning Guide
aligned with the Results Act guidelines. The guide linked planning
guidance of higher headquarters (the Army Materiel Command); the
Army's Planning, Programming, Budgeting, and Execution System; and
the Results Act. The guide stated that the Results Act is the
basis for the three products that result from our business
planning process outlined in this document: a strategic plan, an
annual performance plan, and an annual report. Subsequently, the

Chemical and Biological Defense Command issued strategic plans for
the fiscal years 1997 2003 and fiscal years 1998 2004 time frames,
annual performance plans for fiscal years 1997 and 1998, and an
annual report for fiscal year 1997.

The Chemical and Biological Defense Command strategic plan is
driven by and linked with the overarching planning architectures
of DOD, the Army, and the Army Materiel Command. The Chemical and
Biological Defense

Command's strategic planning model directly links the attainment
of its vision with the development of goals and enabling
strategies followed by the execution of the strategies and
measurement of performance. Separate measures were developed to
assess goal achievement as well as progress toward goal
achievement. Specifically, the plan identified Chemical and
Biological Defense Command goals, strategies for achieving the
goals, and measures of goal achievement. Two of the Command's six
goals address their potential contributions to the

CB Defense Program. They are to  research, develop, acquire, and
field NBC defense, smoke, and obscurant materiel that meets
warfighter requirements and reduces acquisition costs and
timeliness (i. e., produces products faster and at lower costs)
and  become the organization of choice for chemical, biological,
and smoke/ obscurant research, development, and technology
services.

11 We did not assess the comprehensiveness, quality, or
effectiveness of the command's effort.

B-282699 Page 10 GAO/NSIAD-99-159 Chemical and Biological Defense
The performance plan identifies performance measures for each
Command goal and performance goals for each strategy. The
performance measures address both accomplishments and progress
toward accomplishments. Examples of quantitative measures of
research and development accomplishments include (1) the
percentage of new NBC systems that meet NBC survivability
requirements, (2) the percentage of nonexempt acquisitions
receiving waivers from performance specifications, and (3) the

percentage of Soldier and Biological Chemical Command science and
technology programs transitioning to joint service and Army
development programs with user validation through modeling,
wargames, or similar methods. Soldier and Biological Chemical
Command officials explained that the Command is still evaluating
some of the fundamental dilemmas in applying Results Act
principles. For example, the Command has yet to agree on what the
right measures are. The identification of measures in the research
and development component of the Soldier and Biological Chemical
Command (and CB defense in general) has been an ongoing challenge
and continues to evolve.

Other CB Defense RDT& E Organizations Have Not Applied Results Act

Principles The remaining RDT& E organizations cited a variety of
reasons for not

incorporating the Results Act's principles in their program
planning or evaluation systems. 12 The two most common reasons
cited were that current DOD planning processes were equivalent to
those of the act, resulting in plans that were in the spirit of
the Results Act, and that the unique nature of RDT& E activities
did not lend itself to the act's performance measurement and
evaluation. CB Defense Program Planning and Evaluation

DOD officials explained that the Office of the Secretary of
Defense, NBC Defense Steering Committee, conducts planning
consistent with Results Act principles for the CB Defense Program
by issuing and executing the Program Objectives Memorandum. They
claimed that memorandum- related planning documents, including
reports that address different aspects of the CB Defense Program
(such as the Joint Operational Concept; the Research, Development,
and Acquisition Plan; the Joint Logistics Support Plan; and the
Joint Modernization Plan) as well as the 12 Of the remaining
principal federal organizations that conduct RDT& E on CB defense
topics, only

DOE's CB Nonproliferation Program has developed a strategic plan,
and none have developed a performance plan.

B-282699 Page 11 GAO/NSIAD-99-159 Chemical and Biological Defense
NBC Defense Annual Report to Congress, constitute the equivalent
of a strategic plan. Moreover, CB Defense Program managers stated
that DOD's Planning, Programming, and Budgeting System is
equivalent to the system required by the act and that therefore no
substantive changes are necessary

to comply with the spirit of the legislation. 13 Measurement of
RDT& E Activities

There is no consensus on the appropriateness of applying
performance measures to RDT& E activities. While the Soldier and
Biological Chemical Command has developed and applied measures of
research and development outcomes, other organizations conducting
CB Defense Program RDT& E have not. Neither DARPA, DOE, nor the
Joint Program Office for Biological Defense have taken the
initiative to develop a performance plan. In its strategic plan,
DOE included a 5- year roadmap for developing, demonstrating, and
delivering technology that would lead to major improvements in
preparedness and capabilities. The Joint Program

Office cites the conduct of ACTDs as measures of its performance.
DARPA officials maintain that the nature of their RDT& E
activities do not lend themselves to the application of
performance measurement. DARPA conducts leading- edge research
where the risks of failure are high and the probability of success
is low. Its mission is to pursue long- term, far- reaching, and
high- risk/ high- payoff technology for military systems in the
distant future. DARPA officials argued that developing useable
metrics that are measurable, relevant, and timely for technology
anticipated 10 or more years into the future is impossible.
Moreover, they stated that goals and expectations are set at the
project level and cannot be aggregated at the program level.
Therefore, according to the officials, it would be inappropriate
to develop programwide or agencywide measures of success or
performance. Nonetheless, DARPA did try to develop a performance
contract and submitted a draft to the Defense Management Council
in early 1998. It also argued, however, that it did not fit the
mold of most DOD agencies and should therefore be exempt from the
act's requirements. DARPA drafted performance metrics addressing
its research operations as well as administrative efficiency. The
performance metrics proposed for the research portion of its
activities consisted of a series of assessments 13 The DOD
Comptroller has noted that the Results Act is related to, but
distinct from, DOD's Planning, Programming, and Budgeting System,
and has stated that Results Act planning and program evaluations
need to be integrated with DOD's Planning, Programming, and
Budgeting System.

B-282699 Page 12 GAO/NSIAD-99-159 Chemical and Biological Defense
performed by an independent panel of experts. 14 In December 1998,
the Defense Management Council notified DARPA that it was exempt
from the

requirement. The draft performance contract was never finalized or
implemented. Congress has recognized that successful
implementation on the Results Act in science agencies would not
come quickly or easily. Nonetheless, research organizations have
concluded that the Results Act can or should be applied. The
Research Roundtable, a group of federal researchers and managers
representing a cross section of departments and agencies,
concluded in 1995 that the results of a research program's
performance can be measured. The Army Research Laboratory was
designated as a pilot project for performance measurement under
the act and ultimately outlined an evaluation approach that made
use of three pillars: metrics, peer review, and customer feedback.
In 1999, the Committee on Science,

Engineering, and Public Policy of the National Academy of
Sciences, the National Academy of Engineering, and the Institute
of Medicine reported on the results of their work on the issue of
measuring and evaluating

research in compliance with the act's requirements. The committee
concluded that both applied research and basic research programs
supported by the federal government can be evaluated meaningfully
on a regular basis.

Conclusions The CB defense research and development outcomes and
impacts are not being systematically measured. The CB Defense
Program lacks both quantifiable performance measures and
measurable objectives. In the absence of measures of program
impacts and measurable objectives, progress toward achieving
program goals cannot be determined. Program planning consists of a
series of roadmaps leading to specific equipment items. Managers
cite activity measures and technology demonstrations as measures
of the program's contribution. These planning and programming

steps are appropriate and necessary, but they are insufficient for
quantifying outcomes and impacts. Current measures do not assess
the incremental changes attributable, in whole or in part, to the
CB Defense 14 The Defense Science Board was the proposed panel. It
would have been tasked to review the portfolio of DARPA projects
to assess projects with regard to (1) relevance to warfighters,
(2) ratio of technology investments versus system development, (3)
level of risk, (4) ratio of new versus continuing efforts, and (5)
level of service and commercial sector participation.

B-282699 Page 13 GAO/NSIAD-99-159 Chemical and Biological Defense
Program that improve warfighters' ability to survive, fight, and
win in a CB environment. The Results Act outcome- oriented
principles have not been widely applied

by either CB Defense Program planners or executing organizations.
The utilization of these principles can enable managers and those
overseeing the program to quantify the relative success of the
program and of component projects in satisfying requirements
across different activities (e. g., point detection, early
warning, warning and reporting, and modeling). Impact measures can
provide a planning tool to allocate finite CB Defense Program
resources among competing sets of unmet requirements.
Recommendations We recommend that the Secretary of Defense take
actions to develop a

performance plan for the CB Defense Program based on the outcome-
oriented management principles embodied in the Results Act. The
plan should be agreed to and supported by the relevant RDT& E
organizations and incorporated in DOD's NBC Defense Annual Report
to Congress. Specifically, the plan should

 establish explicit and outcome- oriented goals linked to
warfighters' ability to survive, fight, and win in a CB
environment;  identify quantitative or qualitative performance
measures that can be used to assess progress toward goal
achievement;  describe how performance data would be validated;
describe how RDT& E activities of participating DOD and non- DOD

organizations are coordinated to achieve program goals; and
identify human capital, financial, and resource challenges or
external factors that limit the ability of the program to achieve
its goals. Agency Comments In written comments on a draft of this
report, DOD concurred with our

recommendation. DOD stated it will develop a strategic plan more
closely aligned with the tenets of the Results Act and publish the
plan in the next DOD NBC Defense Annual Report to Congress. DOD's
comments are

reprinted in appendix I. Our scope and methodology are explained
in appendix II, and CB Defense Program RDT& E organizations are
described in appendix III.

B-282699 Page 14 GAO/NSIAD-99-159 Chemical and Biological Defense
As agreed with your offices, unless you publicly announce its
contents earlier, we plan no further distribution of this report
until 7 days after its issue date. At that time, we will send
copies to the Honorable William S. Cohen, Secretary of Defense and
other congressional committees. We will also make copies available
to others on request.

If you have any questions regarding this report, please contact me
or Sushil K. Sharma at (202) 512- 3092. Key contributors to this
assignment were Jeffrey Harris and Weihsueh Chiu. Kwai- Cheung
Chan Director, Special Studies and Evaluations

Page 16 GAO/NSIAD-99-159 Chemical and Biological Defense

Contents Letter 1 Appendix I Comments From the Department of
Defense

18 Appendix II Scope and Methodology

20 Appendix III Chemical and Biological Defense

Program Research, Development, Test, and Evaluation Organizations
22

Figures Figure III. 1: Primary Planning and Executing
Organizations and Programs of DOD's Chemical and Biological
Defense Program Research, Development, Test, and Evaluation 23

Abbreviations

ACTD Advanced Concept Technology Demonstration CB chemical and
biological DARPA Defense Advanced Research Projects Agency DOD
Department of Defense DOE Department of Energy DTO Defense
Technology Objective NBC Nuclear, Biological, and Chemical QDR
Quadrennial Defense Review RDT& E research, development, testing,
and evaluation TARA Technology Area Review Assessment

Contents Page 17 GAO/NSIAD-99-159 Chemical and Biological Defense

Page 18 GAO/NSIAD-99-159 Chemical and Biological Defense

Appendix I Comments From the Department of Defense Appendi x I

Appendix I Comments From the Department of Defense

Page 19 GAO/NSIAD-99-159 Chemical and Biological Defense

Page 20 GAO/NSIAD-99-159 Chemical and Biological Defense

Appendix II Scope and Methodology Appendi x I I

The scope of our study was limited to the nonmedical research,
development, testing, and evaluation (RDT& E) activities of the
Department of Defense's (DOD) Chemical and Biological (CB) Defense
Program. This study does not address any classified programs or
projects. To determine whether the program goals were explicit and
measurable and whether its performance measures assessed outcomes
and impacts rather than outputs and activities, we reviewed the
legislative record, interviewed agency officials, and analyzed
program documents. To understand the requirements of the
Government Performance and Results Act, we reviewed the
legislation as well as implementation guidance issued by the
Office of Management and Budget, DOD Comptroller, and the General
Accounting Office. 1 We queried representatives of the Office of
the Secretary of Defense's Nuclear, Biological and Chemical (NBC)
Defense Steering Committee and the executing organizations (i. e.,
Joint Program

Office, Soldier and Biological Chemical Command, Defense Advanced
Research Projects Agency (DARPA)) regarding their familiarity
with, and utilization of, the Results Act's performance
measurement principles. We reviewed strategic and performance
plans implementing the act as well as documents characterized by
DOD officials as complying with the spirit of the legislation. 2
To determine whether organizations executing the CB Defense RDT& E
activities have incorporated Results Act principles in their
program planning and evaluation, we interviewed program officials,
examined program documents, observed program review meetings, and
attended scientific conferences on CB defense technologies.
Program documents we examined included program budgets, strategic
and performance plans, annual reports, internal program planning
documents, program briefings, and proceedings of program review
meetings. We also observed the 1999

DOD Technology Area Review and Assessment of chemical and
biological defense. Scientific conferences we attended included
the 1998 Joint Workshop on Standoff Detection for Chemical and
Biological Defense and the 1998 Scientific Conference on Chemical
and Biological Defense 1 See Executive Guide: Effectively
Implementing the Government Performance and Results Act (GAO/GGD-
96-118, June 1996); Agencies' Annual Performance Plans Under the
Results Act: An Assessment Guide to Facilitate Congressional
Decisionmaking (GAO/ GGD/ AIMD- 10. 1. 18, February 1998); and The
Results Act: An Evaluator's Guide to Assessing Agency Annual
Performance Plans (GAO/ GGD- 10. 1. 20, April 1998).

2 These include the DOD NBC Defense Annual Report to Congress
(submitted pursuant to 50 CFR 1523), the Joint NBC Modernization
Plan, the Joint Service NBC Defense Research, Development, and
Acquisition Plan, and the Joint Service NBC Defense Logistics
Plan.

Appendix II Scope and Methodology

Page 21 GAO/NSIAD-99-159 Chemical and Biological Defense

Research. We also obtained proceedings from these and other
scientific conferences. To respond to all three objectives, we
contacted the following organizations: DARPA, Arlington, Virginia;
Defense Threat Reduction Agency, Dulles, Virginia; DOD Inspector
General, Washington, D. C.; Department of Energy, Washington, D.
C.; Director, Defense Research and Engineering, Washington, D. C.;
Dugway Proving Ground, Dugway, Utah; Edgewood Chemical and
Biological Center, Aberdeen Proving Ground, Maryland; Joint
Program Office for Biological Defense, Falls Church, Virginia;
National Domestic Preparedness Office, Washington, D. C.; National
Ground Intelligence Center, Charlottesville, Virginia;
Nonproliferation and National Security Office, Department of
Energy, Washington, D. C.; Office of Science and Technology
Policy, White House, Washington, D. C.; Office of the Secretary of
Defense, Washington, D. C.; U. S. Army Soldier and Biological
Chemical Command, Aberdeen Proving

Ground, Maryland; and U. S. Army Training and Doctrine Command,
Fort Monroe, Virginia. We conducted our review from November 1998
to April 1999 in accordance with generally accepted government
auditing standards.

Page 22 GAO/NSIAD-99-159 Chemical and Biological Defense

Appendix III Chemical and Biological Defense Program Research,
Development, Test, and Evaluation Organizations Appendi x I I I

The CB Defense Program is overseen by the Office of the Secretary
of Defense's Nuclear, Biological, and Chemical Defense Steering
Committee, which is comprised of the Directors of the Defense
Threat Reduction Agency and Defense Research and Engineering as
well as their top officials responsible for CB defense. The
steering committee funds research and development at numerous
laboratories in DOD, Department of Energy, and private industry. 1
As illustrated in figure III. 1, key research and development
organizations in the execution of the program include the SBCCOM,
JPO- BD, DARPA. 1 The CB Defense Program addresses nonmedical
research and development in the areas of chemical detection,
biological detection, individual protection, collective
protection, decontamination, modeling and simulation, core science
and technology, and basic research. In addition, core science and
technology includes threat assessment and aerosol technology; and
basic research includes aerosol science, chemistry and toxicology,
and analytical chemistry.

Appendix III Chemical and Biological Defense Program Research,
Development, Test, and Evaluation Organizations

Page 23 GAO/NSIAD-99-159 Chemical and Biological Defense

Figure III. 1: Primary Planning and Executing Organizations and
Programs of DOD's Chemical and Biological Defense Program
Research, Development, Test, and Evaluation

SBCCOM is organized around two integrated business areas, one of
which is research, development, and acquisition. Nearly half of
SBCCOM research, development and acquisition funding supports the
CB Defense Program. SBCCOM is engaged in the full range of
research and development encompassing both biological and chemical
systems. Department of Defense

Department of Energy

Defense Threat Reduction

Agency Director of Defense

Research and Engineering

DATSD Counterproliferation

and CB Defense Director of

CB Defense Directorate

CB Nonproliferation Program CB Nonproliferation

Program Department

of Energy Labs JPO- BD SBCCOM

$$

DARPA NBC Defense Steering Committee

CB Defense Program DATSD = Deputy Assistant to the Secretary of
Defense SBCCOM = Soldier and Biological Chemical Command JPO- BD =
Joint Program Office for Biological Defense DARPA = Defense
Advanced Research Projects Agency

= Funding flows = Informal coordination

Appendix III Chemical and Biological Defense Program Research,
Development, Test, and Evaluation Organizations

Page 24 GAO/NSIAD-99-159 Chemical and Biological Defense

SBCCOM business areas include chemical detection, biological
detection, decontamination, protection, and supporting science and
technology. JPO- BD was created in 1994 to manage the biological
warfare agent detection program. The office monitors emerging
technologies for advanced development, demonstration, and upgrades
of fielded biological detection systems. The DARPA Biological
Warfare Defense Program is an applied research program established
by the National Defense Authorization Act for Fiscal Year 1997 (P.
L. 103- 160, as amended) to fund an applied research program

supporting revolutionary new approaches to biological warfare
defense. The Biological Warfare Defense Program pursues high-
risk, high- potential technologies from the demonstration of
technical feasibility through the development of prototype
systems.

DOE's CB Nonproliferation Program was established in 1997 in
response to the Defense Against Weapons of Mass Destruction Act of
1996, which appropriated $17 million to DOE to conduct research
and development to develop new means for detecting the presence,
transportation, production, and use of weapons of mass destruction
and related materials and technologies. According to DOE, the
purpose of this appropriation was to ensure the full engagement of
DOE national laboratories in responding to the threat posed by CB
weapons to U. S. civilians. DOE funds research and development,
from basic research to fieldable prototypes, in pursuit of
advanced technologies that can enable first responders to more
effectively prepare and respond to the use of CB agents.

(713039) Let t er

Ordering Information The first copy of each GAO report and
testimony is free. Additional copies are $2 each. Orders should be
sent to the following address, accompanied by a check or money
order made out to the Superintendent of Documents, when necessary,
VISA and

MasterCard credit cards are accepted, also. Orders for 100 or more
copies to be mailed to a single address are discounted 25 percent.

Orders by mail: U. S. General Accounting Office P. O. Box 37050
Washington, DC 20013

or visit: Room 1100 700 4th St. NW (corner of 4th and G Sts. NW)
U. S. General Accounting Office Washington, DC

Orders may also be placed by calling (202) 512- 6000 or by using
fax number (202) 512- 6061, or TDD (202) 512- 2537.

Each day, GAO issues a list of newly available reports and
testimony. To receive facsimile copies of the daily list or any
list from the past 30 days, please call (202) 512- 6000 using a
touchtone phone. A recorded menu will provide information on how
to obtain these lists. For information on how to access GAO
reports on the INTERNET, send an e- mail message with info in the
body to: info@ www. gao. gov or visit GAO's World Wide Web Home
Page at: http:// www. gao. gov

United States General Accounting Office Washington, D. C. 20548-
0001

Official Business Penalty for Private Use $300

Address Correction Requested Bulk Rate

Postage & Fees Paid GAO Permit No. GI00

*** End of document. ***