United States General Accounting Office
          __________________________________________________________________
          GAO                         Report to the Chairman, Legislation
                                      and National Security Subcommittee,
                                      Committee on Government Operations,
                                      House of Representatives
 
          __________________________________________________________________
 
          July 1990                   STRATEGIC DEFENSE SYSTEM
 
                                      Stable Design and Adequate Testing
                                      Must Precede Decision to Deploy
 
 
          Some of the information in this report--e.g., pictures, charts,
          and tables--could not be shown in ASCII text format.  If you wish
          to obtain a complete report, call GAO report distribution at
          202/275-6241 (7:30 a.m.-5:30 p.m. EST) or write to GAO, P.O. Box
          6015, Gaithersburg, MD 20877.
 
 
           _________________________________________________________________
 
 
 
                 B-239476
 
                 July 6, 1990
 
 
                 The Honorable John Conyers, Jr.
                 Chairman, Legislation and
                 National Security Subcommittee
                 Committee on Government Operations
                 House of Representatives
 
                 Dear Mr. Chairman:
 
                 This report discusses why the Strategic Defense Initiative
                 Organization will not be able to support currently scheduled
                 full-scale development or deployment decisions on any part
                 of Phase I of the Strategic Defense System.  The
                 organization will not be able to support these decisions
                 until it has solidified the system architecture, conducted
                 integrated system-level testing in real time involving
                 actual pieces of hardware and software, and until the
                 Department of Defense adheres to technical and programmatic
                 oversight, commensurate with such a system.
 
                 As arranged with your office, unless you publicly announce
                 the contents earlier, we plan no further distribution of
                 this report until 30 days from the date of this letter.  We
                 will then send copies to other appropriate congressional
                 committees; the Director, Strategic Defense Initiative
                 Organization; the Director, Office of Management and Budget;
                 the Secretary of Defense; and other interested parties.
 
                 This report was prepared under the direction of Samuel W.
                 Bowlin, Director for Defense and Security Information
                 Systems, who may be reached at (202) 275-4649.  Other major
                 contributors are listed in appendix IV.
 
                 Sincerely yours,
 
 
 
 
                 Ralph V. Carlone
                 Assistant Comptroller General
 
 
 
 
          1
 
 
          ____________________________________________________________________
          EXECUTIVE SUMMARY
 
 
 
 
          ____________________________________________________________________
          PURPOSE
 
                          The Department of Defense plans to spend between $69
                          and $87 billion to develop and deploy the first of a
                          multiple phase Strategic Defense System designed to
                          shield the United States from a massive Soviet
                          nuclear ballistic missile attack.  At the request of
                          the Chairman, Legislation and National Security
                          Subcommittee, House Committee on Government
                          Operations, GAO reviewed the Strategic Defense
                          Initiative Organization's (SDIO's) test and
                          evaluation plans.  These plans are to support a 1993
                          executive decision by the President on deploying
                          Phase I and a 1994-1995 acquisition decision by the
                          Department of Defense to enter full-scale
                          development.  Specifically, GAO was asked to
                          determine
 
                          --              what test and evaluation
                                          requirements were established for
                                          making development and deployment
                                          decisions for the first phase, and
 
                          --              whether SDIO's approach will meet
                                          those requirements and support a
                                          decision on deployment.
          ____________________________________________________________________
          BACKGROUND
 
                          The Strategic Defense System is being designed to
                          destroy thousands of incoming ballistic missiles and
                          warheads.  This will require detecting the missiles
                          and warheads, discriminating them from hundreds of
                          thousands of decoys, precisely tracking the
                          missiles and warheads, and then destroying them--all
                          within about 35 minutes or less, and despite the
                          enemy's concerted effort to defeat the system.  The
                          Strategic Defense System is, by far, the most
                          complex, technologically challenging system ever
                          attempted.  As currently envisioned, a
                          sophisticated communications system will link
                          together hundreds, or possibly thousands, of
                          computer-operated components in space and on the
                          ground.  The system will consist of several
                          subsystems (elements), which include surveillance
                          satellites, space- and ground-based weapons, and
                          ground-based subsystems to command and control the
                          satellites and weapons.
 
 
                                                  2
 
 
 
                          The Strategic Defense System is being developed in
                          several phases.  Phase I will be built using
                          existing technologies; later phases would use more
                          exotic technologies, such as lasers and particle
                          beams.  On September 17, 1987, the Secretary of
                          Defense directed that Phase I enter the concept
                          demonstration and validation stage.  This stage is
                          important because enough information must be
                          developed to show that the system is feasible before
                          a decision is made to enter full-scale development.
                          Because Phase I cannot be demonstrated outside of an
                          actual ballistic missile attack, much of this
                          information can be generated only through computer
                          simulation and modeling.
 
                          In 1988, SDIO formally adopted a design
                          (architecture) for Phase I.  However, in January
                          1990 SDIO changed the design dramatically by
                          incorporating a new space-based weapon, called
                          Brilliant Pebbles.   The idea behind Brilliant
                          Pebbles is that thousands of small, relatively
                          autonomous weapons would be deployed to intercept
                          Soviet intercontinental ballistic missiles.
 
          ____________________________________________________________________
          RESULTS IN BRIEF
 
                          The President is scheduled to make a decision in
                          1993 on deploying Phase I.  His options include
                          deploying, delaying, or cancelling Phase I.  Public
                          Law 99-145 states that a Strategic Defense System
                          may not be deployed in whole or in part unless the
                          President certifies to the Congress that the system
                          can fulfill its mission.  SDIO may be able to
                          justify a decision to delay or cancel Phase I;
                          however, SDIO will not be able to support an
                          informed decision to deploy it.  To do so requires a
                          stable design, sufficient testing and evaluation,
                          and according to SDIO, minimum funding levels.
                          However, it is highly unlikely any of these
                          conditions will be met.
 
                          The addition of Brilliant Pebbles in January 1990
                          has reduced, changed, or eliminated the need for
                          some of the space-based elements in the 1988 design.
                          At this point, it is uncertain exactly what Phase I
                          will consist of in terms of elements or what
                          functions they will perform.  Additionally, while
                          SDIO initially required real-time integrated system-
                          level testing prior to a presidential deployment
                          decision and a full-scale development decision, SDIO
                          officials now state that such detailed tests will
                          not be conducted prior to either decision.
                          Furthermore, according to SDIO, because of fiscal
                          year 1990 funding cuts, the full-scale development
 
                                                  3
 
 
                          decision has been delayed until 1994-1995, and
                          research efforts have been scaled back, further
                          reducing the amount of information that will be
                          available for a presidential deployment decision.
                          SDIO officials believe they will be able to support
                          a presidential deployment decision with less
                          information than was originally desired, but state
                          that this will be done at increased risk.
 
                          Phase I has not received the scrutiny and oversight
                          that Defense initially envisioned.  The Defense
                          Acquisition Board did not conduct its scheduled 1989
                          review, nor has it assessed how Brilliant Pebbles
                          affects the design and test and evaluation
                          requirements.  The lack of effective agency
                          oversight has contributed to the failure of other
                          automated weapons systems, none of which matches the
                          scale and technological complexity of Phase I.
                          Consequently, any executive decision in 1993 to
                          deploy Phase I would be premature and fraught with
                          high risk.
          ____________________________________________________________________
          PRINCIPAL FINDINGS
 
          SDI Design and
          Test Plans in
          State of Flux
 
                          Several problems are hampering SDIO's design and
                          test plans.  First, Brilliant Pebbles radically
                          changed the design and sent the program into a state
                          of flux.  SDIO has not yet solidified the role of
                          Brilliant Pebbles or what elements will be in the
                          final design.  This causes problems because without
                          a stable design detailed Strategic Defense System
                          requirements cannot be determined.  Setting
                          requirements is especially important for Phase I
                          where weapon, sensor, and communication systems are
                          interdependent.  Changing a requirement for a
                          weapon system could significantly affect sensor and
                          communications systems as well.  Further, an
                          unstable design increases the probability that
                          system requirements will not be adequately
                          determined and sufficient testing will not be
                          performed to ensure that the system works.
                          Nevertheless, Defense has requested $265 million for
                          full-scale development of one of the sensor
                          elements, the Boost Surveillance and Tracking
                          System, for fiscal year 1991 even though Brilliant
                          Pebbles may eliminate or reduce the requirements for
                          this system.
 
                          Second, SDIO does not plan to conduct integrated
                          system-level testing by the scheduled 1993
                          presidential decision.  SDIO-funded studies and test
                          plans have cited the importance of running real-time
 
                                                  4
 
 
                          integrated system-level tests, which combine
                          computer simulation and actual hardware and
                          software prototypes with human intervention.
                          Furthermore, most of the system tests and
                          evaluations so far have been based on the pre-
                          Brilliant Pebbles design.  Consequently, much of the
                          testing and analyses may no longer be relevant and
                          will have to be repeated.
 
                          Also, according to SDIO, its ability to support an
                          informed deployment decision depended on a minimum
                          funding level of $4.6 billion for fiscal year 1990.
                          However, funding was cut by 20 percent.  According
                          to SDIO, such a cut will reduce the amount of
                          research performed, delay SDIO's full-scale
                          development decision by 2 years, and reduce the
                          amount of information available to the President.
                          Nevertheless, SDIO officials have stated that they
                          will be able to support a presidential decision on
                          the program even though less information will be
                          available.
 
          Original Program
          Oversight Reduced
 
                          Because of the complexity, uniqueness, and cost of
                          Phase I, the Defense Acquisition Board initially was
                          to follow a highly structured oversight system of
                          annual reviews looking at cost growth, changing
                          requirements, and test and evaluation results.
                          However, the October 1989 program review scheduled
                          by the Defense Acquisition Board did not occur.
                          According to Defense officials, the review was
                          cancelled because the fiscal year 1990 budget had
                          not yet been finalized and because Defense was
                          reassessing the amount of oversight needed for the
                          Strategic Defense System.  Even though a Defense
                          Acquisition Board review is scheduled for June 15,
                          1990, major changes to the Phase I design and
                          significant reductions to test and evaluation
                          requirements have already occurred without the
                          Board's review and evaluation.
 
          ___________________________________________________________________
 
          RECOMMENDATIONS
          TO THE SECRETARY
          OF DEFENSE
 
                          GAO recommends that the Secretary of Defense advise
                          the President to defer a decision to deploy any
                          element of the Phase I system until SDIO has
                          stabilized the design and has demonstrated the
                          effectiveness of the system through integrated
                          system-level tests in real time, using system
                          hardware and software prototypes with human
 
                                                  5
 
 
                          intervention.  Further, the Secretary should ensure
                          that required oversight by the Defense Acquisition
                          Board be followed.  The Board should more closely
                          monitor system design, development, testing, and
                          evaluation.  GAO is making other recommendations in
                          chapter 4.
 
          RECOMMENDATION TO
          THE CONGRESS
 
                          GAO recommends that the Congress not fund full-scale
                          development for any element of Phase I, including
                          $265 million for the Boost Surveillance and
                          Tracking System in fiscal year 1991, until SDIO has
                          stabilized the design and has demonstrated the
                          effectiveness of the system through integrated
                          system-level tests in real time, using system
                          hardware and software prototypes with human
                          intervention.
          ____________________________________________________________________
          AGENCY COMMENTS
 
                          As requested by the Chairman's office, GAO did not
                          obtain official agency comments on a draft of this
                          report.  However, GAO discussed the information
                          contained in this report with SDIO officials and has
                          incorporated their comments where appropriate.
 
 
 
                                                  6
 
 
                                    C o n t e n t s
                                    ---------------
                                                                         Page
                                                                         ----
          EXECUTIVE SUMMARY                                                2
 
          CHAPTER
 
             1        INTRODUCTION                                         9
 
                      Genesis/Evolution of the SDI Program                 9
 
                      Objective of a Strategic Defense System             10
 
                      Description of Phase I                              11
 
                      Importance of Concept Demonstration and             12
                      Validation and Defense Acquisition Oversight
 
                      Constraints Affecting SDIO's Approach to            13
                      Concept Demonstration and Validation
 
                      Objectives, Scope, and Methodology                  14
 
 
             2        UNSTABLE ARCHITECTURE AND TEST PLANS WILL NOT       17
                      SUPPORT A PRESIDENTIAL DEPLOYMENT DECISION IN 1993
 
                      Brilliant Pebbles Sends Phase I and Strategic       17
                      Defense System into State of Flux
 
                      SDIO Will Not Perform Integrated System-Level       20
                      Testing Prior to 1993
 
                      SDIO Asserts Funding Cuts Will Reduce Information   22
                      Needed For an Informed Decision in 1993
 
             3        PROGRAM MANAGEMENT OVERSIGHT                        24
                      HAS BEEN REDUCED
 
                      Oversight For Phase I More Stringent Than           24
                      Typical Acquisition
 
                      1989 Annual DAB Review Did Not Occur; Decisions     25
                      Have Been Made Without DAB Review and Approval
 
                      Congress Cuts Funding for Operational Testing       26
                      and Evaluation Until Architecture is Solidified
 
 
 
                                                  7
 
 
 
             4        CONCLUSIONS AND RECOMMENDATIONS                     28
 
 
                      Recommendations to the Secretary of                 30
                      Defense
 
                      Recommendations to the Congress                     30
 
 
 
          APPENDIXES
 
                      Appendix I:  Department of Defense Major            31
                      Systems Acquisition Stages
 
                      Appendix II:  Strategic Defense System Elements     33
                      of Phase I
 
                      Appendix III:  Test Beds Supporting Demonstration   34
                      and Validation of the Strategic Defense System
 
                      Appendix IV:  Major Contributors to This Report     35
 
 
          FIGURES   (These figures are not included because they could not be
                     shown as ASCII text.)
 
                      Figure 1.1:  Phases of a Ballistic Missile Attack   11
 
                      Figure 1.2:  Phase I Architecture Incorporating     12
                                   Brilliant Pebbles
 
 
                                     ABBREVIATIONS
                                     -------------
          DAB         Defense Acquisition Board
          GAO         General Accounting Office
          IMTEC       Information Management And Technology Division
          SDI         Strategic Defense Initiative
          SDIO        Strategic Defense Initiative Organization
 
 
 
                                                  8
 
 
                                       CHAPTER 1
                                       ---------
                                     INTRODUCTION
                                     ------------
          The Strategic Defense Initiative (SDI) is a program to develop and
          deploy a Strategic Defense System to shield the United States
          against a Soviet nuclear ballistic missile attack.  Research and
          development for the SDI program began in 1984, and in 1987 a
          Strategic Defense System was approved for acquisition.  This
          system is to be developed in multiple phases.  Phase I, the
          subject of this report, is in the concept demonstration and
          validation stage of the Department of Defense's acquisition
          process.#1
 
          The Strategic Defense System will be an expensive undertaking.
          The Strategic Defense Initiative Organization (SDIO) cost
          estimates range from $69.6 to $87 billion for completion of Phase
          I development and deployment--with an additional $10.1 billion for
          Phase I operation and support until deployment is completed.
          While the cost of follow-on phases is unknown, SDIO estimates
          that, in addition to the Phase I costs, approximately $26.4
          billion will be needed for fiscal years 1990 through 1994 to
          pursue non-Phase I research and development.  GAO has recently
          reported to the Congress that the above figures are, at best,
          optimistic.#2
 
          GENESIS/EVOLUTION OF THE SDI PROGRAM
          ------------------------------------
          On March 23, 1983, President Reagan called for a comprehensive
          scientific research effort to develop a system that would render
          nuclear ballistic missiles impotent and obsolete.  In January
          1984, SDI was established as a research and technology development
          program, and in April 1984, Defense formally chartered the
          Strategic Defense Initiative Organization as the agency responsible
          for managing Defense's efforts.
 
          After several years of research, SDIO decided in 1987 to develop
          and deploy the system in phases.  SDIO felt that working in phases
          would allow SDIO to be prepared for an early deployment, if such a
          decision were made, and to respond to changing threats.  Phase I of
          the system is to be based on currently available technologies,
          while later phases are to incorporate technologies that are
          expected to be available in the future, such as neutral particle
          beams and lasers.  Phase I is not intended to be a "total defense",
          but is being designed to destroy a certain percentage of some
          intercontinental and sea-launched ballistic missiles (the exact
          percentages are classified).  Creating a full strategic defense
 
 
         1The Department of Defense's major system acquisition process is
          supposed to provide a single approach to designing, developing,
          implementing, and maintaining major weapons systems.  (The five
          stages of the acquisition process are discussed in appendix I.)
 
         2Strategic Defense Initiative: Funding Needs Through Completion of
          Phase I System (GAO/NSIAD-90-79FS, Jan. 29, 1990).
 
                                                  9
 
 
          capability will require the deployment of the follow-on phases.
          Therefore, a critical aspect of the first phase is the capability
          to evolve and support future phases.  Further, the system is not
          being designed to destroy sea-and air-launched cruise missiles.#3
          Other costly systems independent of the SDI program will be needed
          to defend against these weapons.
 
          From the beginning, SDIO has had one overall goal--to conduct a
          vigorous research and technology program that would provide the
          basis for an informed decision regarding the deployment of a
          Strategic Defense System.  The current Phase I program is intended
          to support an executive decision on deployment by the President in
          1993 and an acquisition decision on full-scale development by
          Defense in 1994 or 1995.  According to SDIO test and evaluation
          officials, the President will have a range of options including
          deploying, delaying, or cancelling Phase I.  However, if the
          President decides to deploy the system, Phase I development will
          not be consistent with Defense's prudent acquisition policies
          specifying that deployment decisions be made after full-scale
          development.
 
          OBJECTIVE OF A STRATEGIC DEFENSE SYSTEM
          ---------------------------------------
          The Strategic Defense System is based on a layered defense
          concept; the system is supposed to intercept a missile or a
          deployed warhead, as it travels toward its target.  The basic
          concept is that separate sensor, weapon, and command and control
          systems (SDIO refers to these separate systems as elements) would
          be in space and on the ground.  The system would be tied together
          by a complex communications network and a sophisticated computer-
          based command and control element.  During an attack, the system
          would have to function in an extremely hostile environment,
          including nuclear explosions and a concerted effort by the enemy
          to destroy the Strategic Defense System.
 
          The threat the system would be facing, along with the environment
          in which it would be operating, creates a unique and demanding
          challenge.  First, the space-based elements of the Strategic
          Defense System would have to be able to detect and begin tracking
          thousands of missiles almost immediately after launch, and then
          intercept and destroy some of them.  Those missiles that are not
          destroyed would release warheads, along with decoys intended to
          confuse the system, forcing the system to track hundreds of
          thousands of objects.  Also at this point, space-based elements of
          the system would have to distinguish the warheads from the decoys,
          and intercept and destroy some of the warheads.  The system would
          need to continue discriminating and tracking the surviving warheads
          so that the ground-based elements could intercept more warheads
          before they hit the United States.  These functions would have to
          be tightly coordinated and performed in less than 35 minutes--all
          this with nuclear warheads exploding, anti-satellite weapons
          attacking the system, and the enemy trying to disrupt
 
 
         3Cruise missiles are guided missiles that have terrain-seeking
          radar and fly at moderate speed in low altitude.
 
                                                 10
 
 
          communications and computer operations.  Figure 1.1 describes the
          phases of a ballistic missile attack.
 
          Figure 1.1:  Phases of a Ballistic Missile Attack
 
          (This figure is not included because it could not be shown as
          ASCII text.)
 
          Source:  Adapted from SDI Technology Survivability and Software,
          Office of Technology Assessment, May 1988.
 
 
 
          DESCRIPTION OF PHASE I
          ----------------------
          In 1988, a Phase I design, made up of seven elements, was approved
          by the Defense Acquisition Board.  The design consisted of two
          space-based sensors (Boost Surveillance and Tracking System and
          Space Surveillance and Tracking System); a space-based weapon
          (Space-Based Interceptor); two ground-based sensors (Ground
          Surveillance and Tracking System and Ground-Based Radar); and a
          ground-based weapon (Ground-Based Interceptor), all managed by a
          command and control system (Command Center Element).  (Appendix II
          describes each element.)  The six surveillance and weapon elements
          would be highly interdependent and rely heavily on the command and
          control element, along with a complex communications system.  The
          individual elements would work together as an integrated system to
          detect, track, discriminate, and destroy ballistic missiles and
          their warheads.  At the heart of the system is a large,
          distributed, real-time computer software system which, by some
          estimates, could have 40-100 million lines of code.  The complexity
          and risk of this undertaking cannot be overstated.  We have
          reported repeatedly on Defense systems that were far less complex,
          contained far fewer lines of code, but have yet to perform as
          intended.#4
 
          In January 1990, SDIO decided to include a new weapon concept,
          called Brilliant Pebbles, in Phase I.  Brilliant Pebbles involves
          several thousand individual interceptors orbiting the earth in
          order to detect and destroy a target by smashing into it at high
          speed.  Brilliant Pebbles is supposed to improve system
          survivability and reduce costs by dispersing thousands of space-
          based interceptors that are smaller and more autonomous than the
          Space-Based Interceptor.  The Space-Based Interceptor would house
          a number of interceptors, but unlike Brilliant Pebbles, it would
          rely on other satellites for tracking, targeting, and
 
 
         4Military Space Operations:  Operational Problems Continue With
          the Satellite Control Computer System (GAO/IMTEC-89-56, Aug. 8,
          1989); Space Defense:  Management and Technical Problems Delay
          Operations Center Acquisition (GAO/IMTEC-89-18, Apr. 20, 1989);
          Attack Warning:  NORAD's Communications System Segment Replacement
          Program Should Be Reassessed (GAO/IMTEC-89-1, Nov. 30, 1988);
          Military Space Operations:  Shuttle and Satellite Computer Systems
          Do Not Meet Performance Objectives (GAO/IMTEC-88-7, Aug. 5, 1988).
 
                                                 11
 
          communications.  By making interceptors autonomous, they would no
          longer need to rely on other satellites to perform these
          functions.  Further, because each interceptor would work
          independently, the system's survivability would be increased, in
          principle, because the loss of any one interceptor would not
          greatly affect the system's overall effectiveness.  Figure 1.2
          shows Phase I with the six elements and Brilliant Pebbles.
 
          Figure 1.2:  Phase I Architecture Incorporating Brilliant Pebbles
 
          (This figure is not included because it could not be shown as
          ASCII text.)
 
          SDIO funded three Brilliant Pebbles research and development
          studies during 1989 and 1990.  Studies completed by the Defense
          Science Board and the JASONs--a group of scientists who
          periodically provide technical support to the Department of
          Defense--determined that the Brilliant Pebbles concept was
          technically feasible.  Along with these two studies, Defense
          issued a Space-Based Architecture Study which reviewed the four
          space-based elements of Phase I--Space Surveillance and Tracking
          System, Boost Surveillance and Tracking System, the Space-Based
          Interceptor, and Brilliant Pebbles--and defined and justified a
          recommended space-based architecture.  The Space-Based
          Architecture Study recommended that research continue on both
          Brilliant Pebbles and the Space-Based Interceptor and that
          modified versions of the Boost Surveillance and Tracking System
          and the Space Surveillance and Tracking System sensors be
          included.
 
          SDIO has decided to eliminate the Space-Based Interceptor from
          Phase I, claiming that Brilliant Pebbles will cut costs by 20
          percent, that is, from $87 to $69.6 billion.  However, it is
          unclear whether the costs for all Phase I requirements have been
          incorporated into the $69.6 billion.  Further, neither figure
          includes the costs for operating and maintaining the system.
 
          IMPORTANCE OF CONCEPT DEMONSTRATION AND VALIDATION
          --------------------------------------------------
          AND DEFENSE ACQUISITION OVERSIGHT
          ---------------------------------
          On September 17, 1987, the Secretary of Defense directed that
          Phase I of the Strategic Defense System enter the concept
          demonstration and validation stage of Defense's major systems
          acquisition process.  SDIO's basic acquisition strategy is to
          complete Phase I while researching later phases of a Strategic
          Defense System.  After successfully completing concept
          demonstration and validation, Phase I could enter full-scale
          development.
 
          According to the Defense Systems Management College, concept
          demonstration and validation may be the most critical of all
          acquisition stages.  The primary purpose of this stage is to
          validate the feasibility of Strategic Defense System concepts and
          technologies, and show that the risk is low enough to enter full-
          scale development.  According to the College, the Secretary of
          Defense's decision to place a major system in full-scale
 
                                                 12
 
          development is extremely important.  During full-scale
          development, the system is built, tested, and ready for full-rate
          production.  Not only will development consume enormous resources,
          but major systems, at this point, take on a life of their own and
          are seldom cancelled.
 
          To help Defense gain sufficient design and development
          information, major system acquisitions are reviewed by the Defense
          Acquisition Board (DAB).  The purpose of the DAB, which is chaired
          by the Under Secretary of Defense for Acquisition, is to oversee
          major Defense acquisitions.  The DAB reviews each acquisition stage
          to ensure that every program is ready to proceed into more advanced
          stages of development or production.  Each stage of the acquisition
          process, as well as each milestone decision, is to be supported by
          test and evaluation.  The purpose of test and evaluation is to help
          ensure the timely development, production, and fielding of systems
          that meet users' requirements and perform as intended.  The DAB
          relies on information from test and evaluation to determine whether
          a system is ready to enter full-scale development.  The DAB reviews
          also ensure that plans for later stages follow sound acquisition
          management practices.
 
          The complexity, cost, and uniqueness of the Strategic Defense
          System has prompted the need for a more detailed program review
          process.  For example, the DAB developed a guidance document
          called the Implementor, which provides a framework for additional
          oversight.  Although formal DAB reviews are usually only required
          at major milestones, the Implementor recommends yearly DAB
          reviews.  Further, SDIO must submit additional documentation to
          the DAB to ensure program goals are being met.
 
          CONSTRAINTS AFFECTING SDIO's APPROACH
          -------------------------------------
          TO CONCEPT DEMONSTRATION AND VALIDATION
          ---------------------------------------
          Constraints exist which affect how the Strategic Defense System
          can be demonstrated.  Although the 1972 Anti-Ballistic Missile
          Treaty permits research, it limits the development, testing, and
          deployment of different types of strategic defense systems and
          components.  According to SDIO's 1989 report to the Congress on
          SDI compliance with the treaty, research includes conceptual
          design and laboratory testing.  Development occurs after research
          but precedes full-scale testing of systems and components designed
          for actual deployment.  The development, testing, and deployment of
          launchers, interceptors, and radars are restricted.  Also, the
          extent to which the system can demonstrate its ability to interact
          in a nuclear environment is constrained by limits on nuclear
          testing.
 
          Because the Strategic Defense System cannot be tested in its
          operational environment, many system capabilities must be
          demonstrated through computer modeling and simulation.  For
          example, in simulations, software models would mimic the behavior
          of sensors and weapons and be used in place of the actual elements
          to evaluate system performance.  A ballistic missile attack from
          launch to impact must also be simulated in software to prompt the
          element models to respond.  Eventually, some of these tests would
 
                                                 13
 
          involve actual prototypes#5 of weapons and sensor hardware and
          software interacting in the simulation.  This type of integrated
          system-level testing would be used to evaluate the performance of
          elements within the context of the entire Strategic Defense System.
          Thus, the ability to simulate the interaction of the systems'
          hundreds of computers with their many millions of lines of software
          code becomes of paramount importance in demonstrating whether the
          system can perform its mission.
 
          Early in the SDI program, SDIO recognized the need for integrated
          system-level testing prior to a full-scale development decision.
          Defense policies state that models or simulations can support a
          full-scale development decision and that prototyping is one of the
          most powerful tools available for determining system feasibility
          and capability.  Consequently, SDIO initially planned to build a
          simulation framework to conduct tests using software and hardware
          prototypes.  SDIO studies and contractor documents have supported
          the need for such test capabilities before a full-scale development
          decision.  However, because SDIO believed building such a
          simulation framework would be very complex, it explored other
          approaches for demonstrating and validating the Phase I system.
 
          SDIO decided instead to use multiple test beds.#6  Under this
          approach, each test bed would address critical strategic defense
          issues such as system performance, command and control,
          communications, and sensing and tracking.  The system performance
          test bed, known as the System Simulator, would be at the core of
          this test environment.  The System Simulator, using computer
          models of individual elements, would perform end-to-end, system-
          level tests in which all aspects of Phase I's performance would be
          evaluated while under a simulated ballistic missile attack.
          Although the System Simulator, or some other simulation framework,
          might evolve to include real-time integrated hardware-, software-,
          and human-in-the-loop capabilities, SDIO has no plans to accomplish
          this prior to either the presidential decision on deployment or the
          full-scale development decision.
 
          OBJECTIVES, SCOPE, AND METHODOLOGY
          ----------------------------------
          On June 5, 1989, the Chairman, Legislation and National Security
          Subcommittee, House Committee on Government Operations, requested
          that we assess the Strategic Defense Initiative Organization's
          program activities, test and evaluation requirements, and
          decisions on the development and deployment of Phase I of the
          Strategic Defense System.  Specifically, the objectives of our
          review were to determine (1) what test and evaluation requirements
 
 
         5A prototype is an original or model on which a later item is
          formed or based.  A prototype is usually built during the concept
          demonstration and validation stage and tested prior to the full-
          scale development decision.
 
         6Test beds are sites, facilities, or activities used for testing
          models or prototypes.  SDIO refers to its network of geographically
          dispersed test facilities as the National Test Bed.
 
                                                 14
 
 
          were established for making development and deployment decisions
          for the first phase of the Strategic Defense System and (2) whether
          the current program approach will meet those requirements and
          support a decision on whether to deploy.
 
          To address our objectives, we interviewed a wide range of
          officials involved in the Strategic Defense Initiative.  SDIO
          representatives included: (1) the Phase I System Engineer, as well
          as other Engineering Office officials, (2) test and evaluation
          officials, (3) Brilliant Pebbles Program Office officials, and (4)
          National Test Bed Program Office officials.  We discussed technical
          issues with members of the Phase One Engineering Team, which acts
          as a consultant to SDIO.  Team members are experts in ballistic
          missile defense and include physicists, software and electrical
          engineers, and simulation analysts.  We also met with officials
          from Martin Marietta, the prime contractor for the National Test
          Bed, and officials from General Electric Company, the system
          engineer and integration contractor for the Strategic Defense
          System.  General Electric has the responsibility for defining Phase
          I demonstration and validation requirements, including those for
          test and evaluation.  We met with Army and Air Force officials
          involved in major SDI research and development as well as Defense
          Acquisition Board officials.  Additionally, we consulted with
          ballistic missile defense experts at Sandia and Los Alamos National
          Laboratories.
 
          Our approach was to obtain a thorough understanding of the
          underlying engineering concepts for the Strategic Defense System
          by (1) obtaining and analyzing key system design and requirements
          documents, (2) interviewing and consulting with experts from the
          ballistic missile defense community, and (3) visiting and
          assessing research and development sites.  We reviewed pertinent
          management, technical, and contract documents provided by SDIO,
          SDIO contractors, the Air Force, the Army, Los Alamos National
          Laboratory, and the Defense Science Board.  We analyzed and
          compared system documents such as the 1988 System Description
          Document, the 1990 System Description Document, the Brilliant
          Pebbles System Description Document, and the Space-Based
          Architecture Study.  Furthermore, we analyzed and compared test
          and evaluation documents including the 1987 and 1989 Test and
          Evaluation Master Plan, the Phase I Integrated Test Plan, the
          System Simulator requirements document, the technical
          specifications and software for the current System Simulator, the
          Brilliant Pebbles Integrated Test Plan, and the Brilliant Pebbles
          Test and Evaluation Master Plan.  Through our analyses we were
          able to identify the chronology of events that led to SDIO's
          current test and evaluation approach in support of decisions on
          deployment and full-scale development.  We did not, however,
          validate test and evaluation results.
 
          Our work was conducted at SDIO Headquarters, Washington, D.C.; the
          National Test Bed Joint Program Office, Falcon Air Force Base,
          Colorado Springs, Colorado; Electronic Systems Division, Boston,
          Massachusetts; the Army Strategic Defense Command, Huntsville,
          Alabama; Sandia National Laboratory, Albuquerque, New Mexico; Los
          Alamos National Laboratory, Los Alamos, New Mexico; the Phase One
 
                                                 15
 
 
          Engineering Office, Crystal City, Virginia; the Defense
          Acquisition Board, Washington, D.C.; General Electric Corporation,
          Blue Bell, Pennsylvania; and Martin Marietta Corporation, Colorado
          Springs, Colorado.
 
          We discussed the facts in this report with SDIO officials and have
          incorporated their comments where appropriate.  However, in
          accordance with the requester's wishes, we did not obtain official
          agency comments on a draft of the report.  We conducted our work
          from June 1989 through May 1990, in accordance with generally
          accepted government auditing standards.
 
 
 
                                                 16
                                       CHAPTER 2
                                       ---------
                    UNSTABLE ARCHITECTURE AND TEST PLANS WILL NOT
                    ---------------------------------------------
                  SUPPORT A PRESIDENTIAL DEPLOYMENT DECISION IN 1993
                  --------------------------------------------------
          The President is scheduled to make a decision in 1993 on deploying
          Phase I of the Strategic Defense System.#7  Public Law 99-145
          states that a Strategic Defense System may not be deployed in whole
          or in part unless the President certifies to the Congress that the
          system can survive, meaning that it can maintain enough
          effectiveness during a war to fulfill its mission.  However, by
          1993 SDIO will not have conducted integrated system-level tests
          designed to demonstrate that the entire Phase I system will work as
          planned, so it will not be able to support an informed deployment
          decision by the President.
 
          The recent inclusion of Brilliant Pebbles has fundamentally
          changed the 1988 Phase I architecture by potentially reducing or
          eliminating the need for two sensors and one weapon component of
          the system.  This change has significantly destabilized the
          architecture, resulting in a dramatic restructuring of Phase I.
          According to SDIO test and evaluation officials, the new Phase I
          architecture will not be solidified until 1991 thus decreasing the
          level of system testing that can be performed by 1993.
          Furthermore, according to SDIO's Director, because of fiscal year
          1990 funding cuts, research will be scaled back, reducing the
          information available on which to make a deployment decision in
          1993.
 
          BRILLIANT PEBBLES SENDS PHASE I AND
          -----------------------------------
          STRATEGIC DEFENSE SYSTEM INTO STATE OF FLUX
          -------------------------------------------
          In January 1990, Brilliant Pebbles was formally incorporated into
          the Phase I design.  However, exactly what role Brilliant Pebbles
          will play in the Phase I architecture and even what pieces of
          Phase I will be deployed is uncertain and consequently, has put
          the design of the Strategic Defense System into a state of flux.
          The result is a destabilized architecture and a major
          restructuring of SDIO's program.
 
          Stable Integrated Architecture
          ------------------------------
          Needed for System Development
          -----------------------------
          The SDIO Phase I System Engineer has stated that his first goal in
          support of the presidential deployment decision is to create "a
          complete and coherent system design with definition of all elements
          and inter-element interfaces."#8  It is important in systems
 
 
         7SDIO is conducting the SDI program to support a decision by the
          President prior to the completion of his current term.  Much of the
          specific program guidance and direction, as well as the timing of
          the presidential decision is classified.
 
         8Interfaces are the internal and external communication paths
          within and outside of the system.
 
                                                 17
 
 
          development to have a stable, integrated architecture before
          detailed system requirements can be determined.  A stable
          architecture provides the blueprint for design and development of
          each element.  An unstable architecture causes confusion and
          increases the probability that system requirements and integrated
          test objectives will not be met, and that the system will not
          perform as intended.
 
          The Strategic Defense System is a "system-of-systems" and
          accordingly, the elements are all interdependent.  A requirement
          change for one element may affect all the others.  For example, in
          the 1988 Phase I architecture, the Boost Surveillance and Tracking
          System and the Space Surveillance and Tracking System had a
          requirement to combine their tracking data to provide targeting
          information for the Space-Based Interceptor.  This combined data
          would also be transmitted to the ground-based elements to help them
          identify targets.  The interfaces must be precisely defined to
          support such data sharing.  If the Space Surveillance and Tracking
          System's mission changes, then all the elements must reflect the
          change as well.  Accordingly, to ensure integrated system
          development, each element developer must be aware of all the other
          elements and the respective interfaces when designing each system.
 
          Brilliant Pebbles Destabilizes Phase I Architecture
          ---------------------------------------------------
          Major program restructuring is going on to incorporate Brilliant
          Pebbles and stabilize the architecture.  Brilliant Pebbles is now
          the focal point of SDIO's efforts.  Its effect on the Phase I
          architecture is far-reaching.
 
          As discussed in chapter 1, the 1988 Phase I architecture was made
          up of seven highly integrated elements.  Command and control
          relied on rapid communication and data sharing among elements.
          For example, during a battle, the Boost Surveillance and Tracking
          System would begin tracking boosters and would be one of several
          systems to alert national decision-makers.  The Boost Surveillance
          and Tracking System would then transfer the information to the
          Space Surveillance and Tracking System, which would begin tracking
          and pass intercept information to the Space-Based Interceptor.  At
          the same time, battle information would be sent to the ground-based
          weapons and sensors.  The Ground Surveillance and Tracking System
          and the Ground-Based Radar would continue tracking and the Ground-
          Based Interceptor would destroy the warheads before they reenter
          the earth's atmosphere.  Throughout the battle, command and control
          information would be centralized at the ground-based Command
          Center Element.
 
          The addition of Brilliant Pebbles fundamentally changed this
          highly-integrated approach to strategic defense.  Brilliant
          Pebbles provides for several thousand self-contained interceptors
          that can detect and destroy missiles independently of the other
          Phase I elements.  As a consequence, interfaces, sensing and
          tracking capabilities, communications, and the numbers of elements
          needed--both in terms of types and constellations (numbers of
          satellites)--will have to be reassessed.  For example, the Boost
          Surveillance and Tracking System may not be required as a Phase I
 
                                                 18
 
 
          element, and Brilliant Pebbles will add thousands of independent
          interceptors to the system.  Furthermore, Brilliant Pebbles will
          eliminate the need for the Space-Based Interceptor, could reduce
          tracking requirements placed on the Space Surveillance and Tracking
          System, and could expand the mission requirements for the Ground
          Surveillance and Tracking System.  Unlike the Space-Based
          Interceptor, whose battle management functions were highly
          dependent on ground-based command and control instructions,
          Brilliant Pebbles, after being turned on by the commander, could be
          autonomous.
 
          Phase I Architecture Remains Uncertain
          --------------------------------------
          Defense officials have stated that the elements that will make up
          the Phase I architecture have not yet been solidified.  However,
          several scenarios have been given.  An Office of the Secretary of
          Defense official stated that a space-based system made up of the
          Boost Surveillance and Tracking System and Brilliant Pebbles could
          be initially deployed as the Phase I architecture, followed by the
          Phase I ground-based elements.  Another scenario is that the Boost
          Surveillance and Tracking System and Brilliant Pebbles would be
          deployed with a terminal interceptor#9 (the High Endoatmospheric
          Interceptor), which is not currently included in the Phase I
          design.
 
          In April 1990, the Director of SDIO told the Congress that because
          of Brilliant Pebbles potential for independent operation, the Phase
          I architecture may not require the Boost Surveillance and Tracking
          System and the Space Surveillance and Tracking System or may
          require less capability from them.  However, in May 1990, SDIO
          officials stated that the Phase I architecture will include
          Brilliant Pebbles and all the 1988 Phase I architecture elements,
          with the exception of the Space-Based Interceptor.  The
          differences among such scenarios and the uncertain architecture
          point to the instability of basic concepts underlying the Strategic
          Defense System.
 
          Unless the architecture is stabilized and the respective elements
          are developed, designed, and tested as an integrated system, SDIO
          will not know whether the system will work as planned.  The
          process of interconnecting the Phase I elements so that together
          they can effectively accomplish the strategic defense mission is
          called system integration.  Successful system integration should
          result in the design and delivery of complete elements that will
          work in concert with the other Phase I elements.  Two of the
          elements, however, are preceding the rest of the system.  Because
          of presidential guidance, SDIO has expedited the research and
          development of Brilliant Pebbles.  Consequently, Brilliant Pebbles
          has been placed in an accelerated acquisition program.  The Boost
          Surveillance and Tracking System, which may or may not be needed if
          Brilliant Pebbles is successful, is approaching a full-scale
          development decision.  Defense has requested $265 million for
 
 
         9Terminal interceptors are intended to destroy warheads after they
          have entered the earth's atmosphere.
 
                                                 19
 
          full-scale development of the Boost Surveillance and Tracking
          System in fiscal year 1991.  The Boost Surveillance and Tracking
          System is a multi-missioned satellite that supports other Defense
          programs.  Defense documents show that SDIO will provide $1.3
          billion (or 99 percent) of the system's funding through fiscal year
          1991.  The Boost Surveillance and Tracking System's specific
          functions for the Strategic Defense System are to detect missile
          launches, acquire and track boosters, and assess the number of
          boosters and warheads that have been destroyed.  If Brilliant
          Pebbles and the Boost Surveillance and Tracking System are allowed
          to advance before the requirements of the other elements are
          clearly defined, significant integration problems and costs could
          emerge.
 
          Phase I System Test Plans and
          -----------------------------
          Results May Be Moot
          -------------------
          SDIO test and evaluation officials claim that including Brilliant
          Pebbles in Phase I has destabilized the architecture and that a
          firm design will not be available before 1991.  For example, they
          have stated that a Preliminary Systems Requirements Review that was
          scheduled for May 1990 will be deferred for at least a year.
          Furthermore, SDIO test and evaluation officials are rewriting test
          plans to incorporate Brilliant Pebbles.  Because most of the test
          and evaluation efforts have been based on the 1988 Phase I
          architecture, much of the data and analysis may no longer be
          relevant.  For example, if, as mentioned above, the Space
          Surveillance and Tracking System's mission is changed and it no
          longer provides tracking information, then all of the modeling and
          simulation done so far is useless because all the interfaces and
          data paths have changed.  Any change to one element causes this
          rippling effect across the system.
 
          Further, system-level tests and the resulting requirements have
          not included Brilliant Pebbles.  Accordingly, system tests must be
          rewritten and actual simulations rerun to include them.  Because
          the other Phase I elements have not included Brilliant Pebbles in
          their tests, analyses, and development, element test plans and
          simulations must also be reassessed.
 
          SDIO WILL NOT PERFORM INTEGRATED SYSTEM-
          ----------------------------------------
          LEVEL TESTING PRIOR TO 1993
          ---------------------------
          SDIO-funded studies and test plans cite the importance of
          conducting integrated system-level tests in real time, using actual
          system hardware and software with human intervention.  For
          example, the Advanced Simulation Framework Study recommended that
          component hardware and software models at various geographic
          locations be tested and evaluated to see how they work in real-time
          operations.  Moreover, SDIO's Integrated Test Plan notes the
          importance of such tests before making a decision on full-scale
          development.  According to the plan, integrated system testing
          minimizes costs by integrating and building upon existing
          experiment assets, data, and results, and sharing a common test
          bed.  However, an integrated system-level test in real time using
          hardware and software and a human-in-the-loop will not be
 
                                                 20
 
 
          demonstrated prior to the planned 1993 decision on deployment.
          SDIO officials feel they will be able to support a presidential
          deployment decision with less information than was originally
          desired, but state that this will be done at increased risk.
 
          Value of Integrated System-Level Testing
          ----------------------------------------
          Integrated system-level testing will help confirm whether the
          individual elements and the strategic defense commander can
          successfully interact in real time.  Actual hardware and software
          in the test environment will provide significantly more information
          about system performance than a model.  Such testing would put
          system components in a realistic system environment for detecting
          problems early during design and development.  SDIO officials
          decided that such testing will be performed in the System Simulator
          (or some other simulation framework) and will provide the basis for
          system design studies and end-to-end validation of overall system
          performance.  Simply put, the System Simulator will increase SDIO's
          confidence that when the independently designed and developed
          elements become operational, they will integrate effectively into a
          Strategic Defense System.
 
          In 1988 SDIO established the Stellar Task Force to address the
          question of how best to evolve to such an integrated system test
          environment.  While the task force fully supported the use of the
          System Simulator for system-level tests, it concluded that trying
          to start off using only one test environment was too complex an
          undertaking.  It recommended breaking the problem into pieces and
          developing several test beds.#10  By using several test beds, the
          system could be broken into units, each focusing on a different
          part of the system.  For example, a communications test bed would
          enable SDIO to conduct a series of tests on all of the elements'
          abilities to effectively communicate.
 
          The task force's rationale for this approach was that detailed
          test results from each test bed could be entered into the System
          Simulator's data bases for more accurate system-level testing.
          The elements will be represented at the System Simulator by
          software models rather than actual element hardware and software.
          SDIO does plan to use several test beds and then evolve to
          integrated system-level testing at the System Simulator,
          incorporating actual element hardware and software in a real-time
          environment.  However, integrated testing will not occur before the
          President's scheduled 1993 decision.
 
          System-Level Test Results Based
          -------------------------------
          on Immature Element Models
          --------------------------
          A stable architecture with defined system and element requirements
          is required before software models can be developed to validate
 
 
        10These test beds are the Surveillance and Tracking Test Bed, the
          Communications Network Test Bed, the Pilot Command Center, the
          Gaming Test Bed, and the System Simulator, all of which are
          described in appendix III.
 
                                                 21
 
          each of the elements and the system as a whole.  However, even
          prior to the inclusion of Brilliant Pebbles, the SDIO Phase I
          System Engineer had stated that detailed system requirements for
          all Phase I elements will probably not be set until 1994 or 1995.
          Therefore, element software models cannot sufficiently demonstrate
          how the various elements will work until detailed requirements are
          defined.  As things stand, test results available in 1993 will be
          based on limited software representations of the elements.  For
          example, Air Force officials have stated that they have not been
          provided detailed ballistic missile defense requirements for the
          Boost Surveillance and Tracking System.  Without such requirements,
          system developers cannot develop an accurate software
          representation for system-level testing.  Consequently, it is
          unclear whether detailed requirements will be determined, a model
          developed, system testing completed, and results analyzed in
          sufficient time to support the 1993 deployment decision.
          Therefore, not only will system-level tests not be conducted in
          real time or be integrated with pieces of hardware and software,
          but the tests will be based on simulations involving immature,
          possibly inaccurate element models.
 
          SDIO ASSERTS FUNDING CUTS WILL REDUCE INFORMATION
          -------------------------------------------------
          NEEDED FOR AN INFORMED DECISION IN 1993
          ---------------------------------------
          According to SDIO, an informed executive decision on deployment is
          contingent, in part, on minimum funding levels.  At one time, this
          decision was to occur in the same time frame as Defense's formal,
          full-scale development decision.  In most major acquisitions, a
          deployment decision is made after full-scale development is
          completed and the system has moved into the next stage.  (See app.
          I.)  According to SDIO's Director, an informed decision on
          deployment is contingent on funding levels of $4.6 billion for
          fiscal year 1990 and $33 billion over fiscal years 1990-1994.  He
          stated that any funding reduction would reduce confidence in the
          deployment decision and would have increasingly serious
          consequences for the SDI program.  For example, the Director stated
          that a 20 percent funding cut would reduce the confidence in making
          a decision on deployment due to cutbacks in research.  He further
          stated that emerging concepts, especially Brilliant Pebbles, would
          not be fully explored.  Hence, the space-based architecture would
          not be completely defined, and initial system development and
          deployment schedules would be delayed at least 2 years.
 
          The Congress reduced SDIO's fiscal year 1990 budget request by 20
          percent (from $4.6 billion to $3.6 billion) because of
          congressional concern for overall fiscal constraints and SDIO's
          major uncertainties for fiscal year 1990.  Consequently, SDIO
          initiated a major replanning and restructuring strategy to
          identify program priorities and impacts, and SDIO delayed its full-
          scale development decision for 2 years.  Many contracts were
          reduced, delayed, or cancelled, resulting in scaled-back
          demonstration and validation activities that, according to SDIO,
          will provide less information than was originally envisioned to
          support an informed 1993 deployment decision.  For example, the
          identification of system-level demonstration and validation
          requirements in support of system-level tests will not be
 
                                                 22
 
 
          established until fiscal year 1992; and by 1993, testing on command
          and control functions will not be as thorough as originally
          planned.  Further, because the new Phase I architecture will not be
          defined until 1991, the elements will be less developed and their
          designs less detailed, therefore, less able to support system-level
          testing.  Finally, the system's communications network will not be
          tested for real-time operations.  Nevertheless, SDIO officials have
          stated that they will be able to support a presidential deployment
          decision but with increased risk.
 
 
 
                                                 23
 
                                       CHAPTER 3
                                       ---------
                             PROGRAM MANAGEMENT OVERSIGHT
                             ----------------------------
                                   HAS BEEN REDUCED
                                   ----------------
          SDI program decisions have been made without the formal review and
          approval of the Defense Acquisition Board (DAB), increasing the
          risk that program objectives will not be achieved.  Defense,
          recognizing the complexity, uniqueness, and cost of the Strategic
          Defense System, required additional DAB oversight and directed
          additional monitoring of SDIO's operational test and evaluation
          activities.  Despite the more stringent oversight requirements, the
          1989 annual DAB review did not occur as planned.  Consequently,
          fundamental changes to the Phase I architecture and reductions in
          test and evaluation requirements have occurred without formal DAB
          review and oversight.
 
          OVERSIGHT FOR PHASE I MORE STRINGENT
          ------------------------------------
          THAN TYPICAL ACQUISITION
          ------------------------
          As discussed in chapter 1, the DAB oversees major Defense
          acquisition programs.  As part of its oversight responsibility, the
          DAB conducts formal reviews at major program milestones to ensure
          that programs are ready to proceed into more advanced stages of
          development and that proposed program plans for later stages follow
          sound acquisition management practices.  Among the issues typically
          considered in a DAB milestone review are cost, requirements
          growth, schedule delays, threat assessment, acquisition strategy,
          and test and evaluation plans and results.  As part of the Defense
          acquisition process, DAB reviews are to help reduce the risks
          associated with major systems acquisitions.
 
          Phase I, however, is not a typical acquisition.  The complexity,
          cost, and uniqueness of the Strategic Defense System has prompted
          the need for a more stringent acquisition process.  For example,
          in 1987 the DAB developed a guidance document called the
          Implementor which provides a framework for additional oversight.
          Although formal acquisition reviews by the DAB are usually only
          required at major milestones, the Implementor expanded the DAB's
          oversight responsibility to include yearly reviews, and lists
          additional documentation SDIO must submit to the DAB to ensure
          program goals are being met.
 
          In addition to the increased DAB oversight, Defense also decided in
          1987 to establish an oversight group under its Director for
          Operational Test and Evaluation to monitor SDIO's test and
          evaluation activities.  Each stage of the acquisition process, as
          well as each milestone decision, is to be supported by test and
          evaluation.  The purpose of test and evaluation is to help ensure
          the timely development, production, and fielding of systems that
          meet the users' requirements and perform as intended.  The DAB
          relies on information from test and evaluation activities to
          determine whether a system is ready to enter the next stage of
          development.
 
 
                                                 24
          1989 ANNUAL DAB REVIEW DID NOT OCCUR; DECISIONS
          -----------------------------------------------
          HAVE BEEN MADE WITHOUT DAB REVIEW AND APPROVAL
          ----------------------------------------------
          The Implementor the DAB imposed on the SDI program states that the
          Phase I program baseline must be reviewed annually by the DAB.
          These annual reviews were to include cost, schedule, technical
          performance, and operational assessments.  The annual DAB review
          scheduled for October 1989 was delayed until December and later
          cancelled.  According to Defense officials, the review was
          cancelled because the fiscal year 1990 budget had not yet been
          finalized, and because Defense was reassessing the amount of
          oversight needed for the Strategic Defense System acquisition.  The
          review has been rescheduled for June 15, 1990.  Further,
          significant issues have not been formally reviewed by the DAB as to
          their impact on the SDI program.  These issues include the October
          1989 Space-Based Architecture Study's conclusions and
          recommendations and the dramatic effect of incorporating Brilliant
          Pebbles into Phase I.
 
          The Space-Based Architecture Study was initiated to review the
          feasibility and consequences of incorporating Brilliant Pebbles
          into Phase I with special emphasis on the space-based elements--the
          Space-Based Interceptor, the Boost Surveillance and Tracking
          System, and the Space Surveillance and Tracking System.  The study
          concluded that Brilliant Pebbles was feasible, but recommended that
          research continue on both Brilliant Pebbles and the Space-Based
          Interceptor.  Additionally, the study concluded that if Brilliant
          Pebbles were incorporated, major changes should be made to system
          requirements that could significantly affect program cost and
          schedule.  For example, the study concluded that (1) the Boost
          Surveillance and Tracking System's ballistic missile defense
          requirements could be reduced; (2) the Space Surveillance and
          Tracking System is an essential asset for Phase I but may need to
          be modified; and (3) the Ground Surveillance and Tracking System's
          capability will have to be increased.  While SDIO officials have
          considered the study's results, the DAB has not formally reviewed
          the study and determined its consequences on the program's design,
          cost, and schedule.
 
          Under Defense's current approach, major changes to Phase I have
          occurred without high-level Defense oversight and management
          review.  Although the study recommended that research continue on
          both space-based weapons, SDIO decided to include Brilliant Pebbles
          in the Phase I architecture.  This fundamental change to the
          program baseline was made without formal DAB review and oversight.
          As noted in chapter 1, this addition will have far-reaching impacts
          that will ripple across the Strategic Defense System; system
          interfaces, sensing capabilities, communications, and element
          requirements will all change significantly.
 
          SDIO has also reduced the testing requirements for both the System
          Simulator and the Pilot Command Center without DAB review.  SDIO
          initially established stringent System Simulator requirements that
          would demonstrate an end-to-end, real-time system test with
          hardware-, software-, and human-in-the loop in support of a full-
          scale development decision.  However, the System Simulator is now
          only required to demonstrate an end-to-end simulation using
          software representations of the system elements instead of actual
 
                                                 25
 
          pieces of hardware and actual system software operating in real
          time.  Further, the Pilot Command Center, SDI's test bed to
          demonstrate battle management and command, control, and
          communications functions in real time, was to include actual sensor
          and weapon hardware-in-the-loop testing.  Current test plans do not
          include actual sensor and weapon interfaces.  Since these
          significant reductions in demonstration and validation
          requirements have not been subject to DAB review, high-level
          Defense Department officials have not passed judgment on whether
          this proposed approach will provide sufficient information to make
          an informed deployment decision.
 
          CONGRESS CUTS FUNDING FOR OPERATIONAL TESTING AND EVALUATION
          ------------------------------------------------------------
          UNTIL ARCHITECTURE IS SOLIDIFIED
          --------------------------------
          The Congress has shown long-standing interest in how major weapon
          systems perform and the adequacy and timeliness of operational test
          and evaluation.  As early as 1971, the Congress enacted legislation
          requiring Defense to provide the Congress with data on operational
          test and evaluation results of major weapon systems before
          committing major production dollars.
 
          In 1983, Congress enacted additional legislation creating the
          Office of the Director of Operational Test and Evaluation.
          Defense directed this office to provide an independent assessment
          of system-level testing for the Strategic Defense System.  This
          office is to provide independent oversight, coordinate the military
          services' planning and execution of operational tests, and
          objectively report on test results to Defense and congressional
          decision-makers.
 
          The Congress expects independent oversight and objective reporting
          of operational test results before it is willing to commit to
          production decisions.  However, the Congress, in November 1989,
          expressed concern about the constantly changing design of the
          Strategic Defense System and, given these changes, questioned what
          value the Office of Operational Test and Evaluation could provide
          to the Strategic Defense System at that time.  According to the
          1990 Authorization Act conferees,#11
 
               ...the need for a Strategic Defense Initiative
               operational test and evaluation activity is premature in
               view of the shifting SDI architectures, plans and
               priorities.  Accordingly, the conferees agree to reduce
               the operational test and evaluation authorization by
               $12.415 million from the requested amount and direct the
               Department of Defense not to obligate any appropriated
               funds for such an activity until it has been formally
               authorized in law.
 
 
 
 
        11Authorizing Appropriations for Fiscal Year 1990 for Military
          Activities of the Department of Defense, House of Representatives
          Report No. 101-331, 101st Congress, 1st Session (Conference Report).
 
                                                 26
 
 
          We agree that such testing is premature given the fluid and
          constantly changing nature of the Strategic Defense System
          architecture.  However, once the architecture is set, early
          system-level operational test and evaluation will help ensure that
          the acquisition process, which includes system-level test plans and
          methodologies, is not only monitored but also assessed.
          Furthermore, such oversight will help ensure that complex system-
          level test results, justifying a full-scale development decision,
          will be understood.
 
 
 
 
                                                 27
 
 
                                       CHAPTER 4
                                       ---------
                            CONCLUSIONS AND RECOMMENDATIONS
                            -------------------------------
          The Strategic Defense Initiative Organization is engaged in one of
          the most complex and technologically challenging efforts ever
          undertaken.  Tens of billions of dollars will be needed to develop
          and deploy the first phase of a Strategic Defense System.  The
          first phase is designed to protect the United States from only a
          portion of a massive Soviet nuclear ballistic missile attack.
 
          The Phase I architecture of the Strategic Defense System remains in
          a state of flux, primarily because Brilliant Pebbles, a new space-
          based weapon, was added to the system.  The importance of a stable
          architecture cannot be overstated.  An architecture that is not
          well defined runs the risk that system requirements will not be met
          and that the system will not perform as intended.  Unless the
          individual Strategic Defense System elements are designed,
          developed, and tested as an integrated system, SDIO will not know
          whether the system will work as intended.
 
          Brilliant Pebbles has greatly changed the Phase I architecture and
          highlights the instability of SDIO's design for the Strategic
          Defense System.  Instead of the highly integrated, interdependent
          architecture envisioned for Phase I, Brilliant Pebbles is to
          provide more autonomy and require less interdependence among
          elements.  As a result, element interfaces, surveillance and
          tracking capabilities, communications, and the numbers of elements
          will change.  According to the Phase I System Engineer, even before
          the addition of Brilliant Pebbles to the architecture, detailed
          system requirements for all Phase I elements would probably not be
          completely defined until 1994 or 1995.  Nevertheless, Defense has
          requested $265 million for full-scale development of the Boost
          Surveillance and Tracking System in fiscal year 1991.  While the
          system is being developed to meet other Defense missions, we
          believe that a decision to enter full-scale development for the
          Boost Surveillance and Tracking System before the Phase I
          architecture had been solidified and its ballistic missile defense
          requirements defined would be premature.  Such decisions in other
          Defense programs have led to significant cost increases, schedule
          delays, and performance shortfalls.
 
          Phase I system-level test and evaluation activities will be
          significantly affected as well.  System-level testing and
          evaluation are critical to making decisions on the continued
          development and deployment of a Strategic Defense System.
          However, the majority of system test and evaluation efforts so far
          have been based on an earlier Phase I architecture that did not
          include Brilliant Pebbles.  Consequently, much of the current test
          data and analyses will not be applicable to the new architecture,
          and many of the tests will likely have to be redesigned and
          repeated.  Even though initial SDIO test plans required that real-
          time integrated system-level tests be performed prior to a full-
          scale development decision and a presidential deployment decision,
          SDIO officials now claim such tests will not be conducted prior to
          either decision.  We believe SDIO's initial plans could
 
                                                 28
 
 
          significantly reduce the risks involved in making critical
          development and deployment decisions for such a costly and
          technically complex system.
 
          The President is scheduled to make a decision in 1993 on deploying
          Phase I of the Strategic Defense System.  The President will have a
          range of options including deploying, delaying, or cancelling the
          Phase I system.  Public Law 99-145 states that the system may not
          be deployed unless the President determines that the system is
          sufficiently capable of fulfilling its mission.  Given that the
          architecture continues to change, the program is falling behind
          schedule, and the amount of test and evaluation data has been
          reduced, we do not believe that SDIO will be able to give the
          President enough information to support a 1993 decision to deploy
          Phase I.
 
          We are not alone in raising concerns about the Strategic Defense
          System's changing architecture.  The Congress has been troubled by
          the constantly changing design of the Phase I system and has
          questioned the value of operational test and evaluation oversight
          at this time.  The Congress was so concerned that for fiscal year
          1990 it not only reduced SDI program funding by 20 percent (from
          $4.6 billion to $3.6 billion), but also cut funding for the Office
          of Operational Testing and Evaluation, citing such activities as
          premature in view of the shifting Strategic Defense System
          architecture, plans, and priorities.
 
          The federal government has repeatedly found that major
          acquisitions such as the Strategic Defense System are extremely
          difficult and complex undertakings, fraught with high risks.
          Indeed, many have floundered.  A principal element needed to
          successfully design and develop highly integrated systems is the
          commitment and oversight of agency leaders.  To provide that
          oversight they must have the information they need--such as program
          cost, requirements growth, schedule delays, acquisition strategy,
          and test and evaluation plans and results--to effectively guide the
          acquisition effort.
 
          High-level oversight of the SDI program, however, is not
          occurring.  The Defense Acquisition Board, which oversees major
          Defense acquisition programs, has not reviewed major changes to the
          Phase I architecture and to the system's test and evaluation
          requirements.  The DAB's 1989 annual program review was cancelled.
          According to Defense officials, the review was cancelled because
          the fiscal year 1990 budget had not yet been finalized, and because
          Defense was reassessing the amount of oversight needed for the
          Strategic Defense System acquisition.  We do not believe these
          reasons justify cancelling scheduled oversight reviews considering
          the dramatic, fundamental changes in the Phase I design and the
          significant reductions in system-level test and evaluation
          requirements.  As a result, significant changes, such as the
          addition of Brilliant Pebbles to the Phase I architecture and the
          results of the October 1989 Space-Based Architecture Study, have
          not been subject to formal review.
 
 
 
                                                 29
 
 
          RECOMMENDATIONS TO THE SECRETARY OF DEFENSE
          -------------------------------------------
          We recommend that the Secretary of Defense advise the President to
          defer a decision to deploy any element of the Phase I system until
          SDIO has stabilized the architecture and has demonstrated the
          effectiveness of the system through integrated system-level tests
          in real time, using system hardware and software prototypes with
          human intervention.  Additionally, the Secretary should ensure
          that required oversight by the Defense Acquisition Board be
          followed.  The Board should more closely monitor system design,
          development, testing, and evaluation.  Finally, when the program's
          architecture is stable, the Secretary should request that the
          Congress reinstate funding for the Director of Operational Test and
          Evaluation to provide an independent system-level assessment for
          the Strategic Defense System.
 
          RECOMMENDATION TO THE CONGRESS
          ------------------------------
          We recommend that the Congress not fund full-scale development for
          any element of Phase I until SDIO has stabilized the architecture
          and has demonstrated the effectiveness of the system through
          integrated system-level tests in real time, using hardware and
          software prototypes with human intervention.  This would include
          not providing $265 million for the Boost Surveillance and Tracking
          System in fiscal year 1991.  However, if Defense needs the Boost
          Surveillance and Tracking System for another mission, independent
          and separate from the Strategic Defense System, it should be
          justified and funded to meet that mission and should no longer be
          considered an element of the Strategic Defense System.
 
 
 
 
                                                 30
 
          APPENDIX I                                               APPENDIX I
 
 
                                 DEPARTMENT OF DEFENSE
                                 ---------------------
                           MAJOR SYSTEMS ACQUISITION STAGES
                           --------------------------------
          Mission area analysis and program initiation generally precede the
          five Department of Defense acquisition stages.  Defense components
          continually analyze their assigned mission areas to identify
          deficiencies (needs) and determine if new systems or major
          upgrades to existing systems are necessary.  These analyses often
          result in recommendations to initiate new acquisition programs
          through the validation of a need to correct the deficiency.  Once a
          need has been identified and validated and Defense initiates an
          acquisition program, the program enters the concept formulation
          stage.
 
          CONCEPT FORMULATION STAGE
          -------------------------
          In this stage, potential requirements and alternative approaches to
          satisfy the need are identified and evaluated.  Various types of
          analyses considering trade-offs among performance, life-cycle cost,
          and  schedule are conducted to select among possible concepts to
          satisfy the need.  Once a concept has been selected, it is
          presented to Defense for approval.
 
          DEMONSTRATION AND VALIDATION STAGE
          ----------------------------------
          In this stage, feasibility and desirability of the selected
          requirements and the system concept are further analyzed, generally
          using techniques like computer simulation, hardware prototyping,
          development test and evaluation, operational test and evaluation,
          or a combination of test methods.  When the feasibility of the
          concept has been convincingly demonstrated and validated, the
          program enters the full-scale engineering and development stage.
 
          FULL-SCALE DEVELOPMENT STAGE
          ----------------------------
          In this stage, the system, including every item necessary for its
          logistic and operational support, is designed, fabricated, and
          tested.  At the conclusion of this stage, the system is ready to be
          produced.
 
          FULL-RATE PRODUCTION AND INITIAL DEPLOYMENT STAGE
          -------------------------------------------------
          During this stage the proposed system is built and released to the
          user.  At this point, the system becomes operational.
 
          OPERATIONS SUPPORT STAGE
          ------------------------
          This stage immediately follows deployment and extends until the
          system is removed from Defense inventory.  Two major Defense
          reviews are conducted in this stage.  The first takes place 1 to 2
          years after deployment to determine if operational readiness and
          support objectives are being achieved and maintained.  The second,
          occurring 5 to 10 years after deployment, evaluates system
 
                                                 31
 
 
          APPENDIX I                                               APPENDIX I
 
 
          capabilities and assesses whether major upgrades are needed or if
          the system should be replaced.
 
 
 
 
                                                 32
 
 
          APPENDIX II                                             APPENDIX II
 
 
                               STRATEGIC DEFENSE SYSTEM
                               ------------------------
                                  ELEMENTS OF PHASE I
                                  -------------------
                                      General      Specific
          System element              function     functions
          --------------              --------     ---------
          Boost Surveillance          Sensor       Detect missile launches;
          and Tracking                             acquire and track
          System                                   boosters; assess kills
 
          Space Surveillance          Sensor       Acquire and track warheads
          and Tracking                             and satellites; assess
          System                                   kills
 
          Ground                      Sensor       Track warheads and decoys;
          Surveillance and                         discriminate warheads from
          Tracking System                          decoys; assess kills
 
          Ground-Based                Sensor       Acquire and track warheads
          Radar                                    and decoys; discriminate
                                                   warheads from decoys
 
          Space-Based                 Weapon       Destroy boosters and
          Interceptor                              warheads
 
          Ground-Based                Weapon       Destroy warheads
          Interceptor
 
          Command Center                           Human decision-making;
                                                   communications and
                                                   guidance for defense
                                                   system
 
 
 
                                                 33
 
 
          APPENDIX III                                           APPENDIX III
 
 
                        TEST BEDS SUPPORTING DEMONSTRATION AND
                        --------------------------------------
                      VALIDATION OF THE STRATEGIC DEFENSE SYSTEM
                      ------------------------------------------
          The Surveillance and Tracking Test Bed will evaluate tracking and
          discrimination function performance for accuracy, computational
          requirements, speed, robustness.  It will also validate
          surveillance algorithms and integrated system performance.
 
          The Communications Network Test Bed will validate communications
          network operations and management and validate security of the
          Strategic Defense System's communications design.
 
          The Pilot Command Center will validate command structure, hardware-
          in-the-loop, and demonstrate proof of concept for mobile and fixed
          physical configurations with selected software performance.
 
          The Gaming Test Bed will evaluate proposed command and control
          operational concepts, build operator and user confidence in
          hardware-in-the-loop decision tools to support the development of
          the Pilot Command Center.
 
          The System Simulator will provide the basic vehicle for system
          design studies and end-to-end validation of overall system
          performance.  It will also provide validate software and
          integration capabilities.
 
 
 
 
                                                 34
 
 
          APPENDIX IV                                            APPENDIX IV
 
 
                           MAJOR CONTRIBUTORS TO THIS REPORT
                           ---------------------------------
          INFORMATION MANAGEMENT AND TECHNOLOGY DIVISION, WASHINGTON, D.C.
          ----------------------------------------------------------------
          Michael T. Blair, Assistant Director
          Leonard J. Latham, Technical Advisor
          Sally M. Obenski, Site Senior
          Victoria L. Miller, Evaluator
          Teresa M. Schlee, Writer-Editor
 
          DENVER REGIONAL OFFICE
          ----------------------
          Barry A. Tidwell, Evaluator-in-Charge
          Yvonne J. Rodriguez, Evaluator
 
 
 
          (510419)
 
 
 
                                                 35
.