Report of the Panel on
Reducing Risk In Ballistic Missile Defense Flight Test Programs
27 February 1998

Report of the Panel on
Reducing Risk
In
Ballistic Missile Defense
Flight Test Programs
27 February 1998

This study was initiated by the sponsors to address risk in the flight test programs of BMDO's hit-to-kill (HTK) ballistic missile defense (BMD) systems. The four systems are the Theater High Altitude Area Defense (THAAD); the Patriot-3 System with its Patriot Advanced Capability-3 (PAC-3) missile; the AEGIS LEAP Interceptor (ALI) Program, which is a risk-reduction program within the Navy Theater Wide (NTW) Defense System; and BMDO's NMD program. This study was motivated by a series of flight test failures in some of these programs failures which indicated a high level of risk. These failures have significantly delayed the planned fielding of BMD systems.

The study group members examined current theater missile defense (TMD) programs to address the issues. However, the task was not to evaluate these programs. Rather, the task was to take lessons from these programs that could and should be applied to the NMD program.

Observations about the current state and future progress on these individual TMD programs are relevant to the findings of the study and are included here. We focused on those observations that are common to more than one program and that could, therefore, be important warning flags for the NMD program.

The study group was composed of members who have extensive experience in the development, testing, and operational employment of complex systems.

We applied this experience in examining current HTK BMD programs. The goal was to extract lessons from these programs and over complex programs and apply these lessons to the NMD program.

These tasks further defined the overall objective.

As previously indicated, to understand the historic T&E paradigm, the study group relied extensively on members' broad and extensive experience in dealing with complex development programs. The primary focus was on reviewing current relevant programs, relating the results of this review to history and experience, and applying the lessons to the NMD program.

We found that the processes and approaches to identify and preclude failure modes in BMD programs are not fundamentally different from the long-standing, sound design and management approaches used for other successful systems. However, the demands of the HTK end game require a higher level of fidelity in the ground simulations [both digital and hardware-in-the-loop (HWIL)] to surround the variables and uncertainties of the dynamics and target presentation.

This report presents specific findings on practices and preflight test options and on NMD program areas that require increased funding emphasis.

In keeping with the study group objectives, the Key Judgments are overarching study group views based on the aggregate of HTK programs examined.

Specific recommendations are provided for reducing the test risk in the NMD program. Findings that address the specific questions in the tasking are also provided. In addition, we reviewed practices from other complex programs. In this category, even though the report specifically discusses only the Peacekeeper Intercontinental Ballistic Missile (ICBM) program and the earlier Sprint Missile program, our experiences with other programs provide similar conclusions.

The sections on specific BMD programs highlight some of the sources of the lessons that we believe are particularly relevant to the NMD program.

This section begins with the Key Judgments. These Key Judgments emanated from what the study group regarded as the most overarching in the aggregate of the programs examined and the experiences of the study group members. These are also those most important as lessons for NMD. Following the Key Judgments, overarching recommendations are provided and then specific questions are addressedagain using the lessons from the programs examined and experiences of the study group members.

The study group's most fundamental finding and the finding most relevant to the NMD program is that the general planning and execution of the THAAD and LEAP programs are inconsistent with the difficulty of the task. These programs are pursuing very aggressive schedules, but these schedules are not supported by the state of planning and testing.

Specifically, the perceived urgency of the need for these systems has led to high levels of risk that have resulted in delayed deployments because of failures in their development test (DT) programs.

After more than a dozen flight tests, the most obvious and visible consequence of this approach is that we are still on "step one" in demonstrating and validating HTK systems. Failures having little to do with the kill vehicle (KV) performancewhere the technology should be in hand have precluded demonstrating that weapons systems are capable of reliably hitting a ballistic missile warhead. And even when this first step is achieved, these programs will skill have to go through steps two and three: demonstrating reliable HTK at a weapon system level and demonstrating reliable HTK against likely real-world targets.

Failures to date reflect inadequate design and fabrication discipline.

The "early capability" approach demands operational capability before system design is completed through the Engineering Manufacturing Development (EMD) phase. This approach is inconsistent with the complexity of the task and has, thus far, not accelerated operational capability. Instead, the added risk has produced little discernible benefit and has actually delayed operational capability.

The most convincing evidence of the risk pressures from this approach is found in the test plaming. This plannng is characterized by either less-than- minimal testing or highly compressed testing or both.

For THAAD, the original plan was 2 years to the first test flight and then almost a test flight per month for the next 2 years. Thus far, the response to failures has been to reduce the testing in an attempt to maintain the schedule. The NTW test schedule is not compressed, but the number of planned tests is not consistent with the task. The Patriot program, which, in most respects, is carefully planned and is building on a legacy of well-developed processes, also has been forced into the less-than-minimum test mode.

Current planning for the NMD test program is even more optimistic than the theater HTK programs.

As noted here, as a benchmark, the Peacekeeper programcertainly no more technically challenging than HTKresponded to intense schedule pressure with a clearly adequate and well-paced test program and delivered the required capability on schedule.

The study group was struck by the similarities among the challenges and the similarities among the likely solutions to flight test failures. However, we also found that lessons, approaches, and solutions used in one program were often not available for other programs with similar challenges.

BMD programs rely heavily on the "graybeard" community to transfer "tribal knowledge." While such groups can provide useful insights and advice, they cannot be a substitute for effective formal and informal cross-tell at the management and engineering levels among programs facing similar challenges. This activity needs to be a high-priority responsibility of BMDO.

The rush to failure in flight testing has been partially caused by a fundamental misunderstanding of the purpose of developmental testing. Some of these tests were treated as demonstrations of known capabilities where "fly to verify" was the purpose. In practice, the unknowns made them "fly to learn" experiences. The "demonstration mindset" was evident in flight tests conducted without complete component qualification and ground testing. One program office espoused the concept of "test a little, learn a lot." The drive for early capability based on minimum capability demonstration has been a factor in this "key demonstration" mentalitythat is, a single success is regarded as a large step forward and becomes the criteria for a key program decision, such as exercising an option to buy operational missiles. This approach and mindset are sharp departures from experience on successful flight test programs that have followed the practice of "learn a lot" and then "test to verify."

BMD programs need to pay more attention to reducing the uncertainties to only those issues that cannot be tested on the ground or adequately simulated. One example is that none of the infrared (IR) HTK programs (THAAD, ALI, and NMD) have exploited or plan to exploit existing high-fidelity scene generation capabilities to exercise their hardware to the maximum advantage.

Test planning needs to be very explicit in identifying the ground test and flight test needs for each key issue.

In general, the test programs are designed to provide a single shot in each operating regime. While back-up hardware is availablein most casesto repeat tests, the single-shot planning produces unrealistic test schedules and pressures to move on despite failures to achieve test objectives.

The study group was not surprised to find that accepthng higher risk is not accelerating fielded capability. The virtually universal experience of the study group members has been that high technical risk is not likely to accelerate fielded capability. It is far more likely to cause program slips, increased costs, and even program failure.

For NMD, the schedule and cost pressures inherent in the 3 + 3 formulation and the system requirements are inherently even more severe than those for the TMD programs that have experienced excessive flight test failures.

To succeed, the NMD program must meet a series of formidable challenges. The effort to meet these challenges must emanate from a clear set of requirements, consistent resource support (which includes an adequate number of test assets), well-defined milestones, and a rigorous test plan. The study group believes that current NMD program is not characterized by these features and is on a high-risk vector. It will benefit from the earliest possible restructuring to a more achievable set of goals.

The study group's initial attempt to distinguish between best practices and specific program recommendations became counterproductive. Hence, the following slides provide specific recommendations for implementing best practices in BMD programs. We have emphasized those practices most relevant to the NMD program.