Report of the Panel on
Reducing Risk In Ballistic Missile Defense Flight Test Programs
27 February 1998

These first sets of recommendations apply to all BMD programs, including NMD.

To reduce the pressure on the programs "to shoot" before progressing to a reasonable probability of success, BMDO must put its programs on realistic schedules before failures occur. These schedules should be consistent with those of past successful programs. Aggressive schedules should be allowed only if they can be justified by new processes or approaches that will support accelerated schedules.

For the near term, the BMDO programs should focus on demonstrating that HTK technology is viable and that HTK against simple targets can be achieved reliably. Only after this has been demonstrated can the programs continue to demonstrate that the proposed weapon system is operationally feasible. As noted in the Findings Section, some requirements on systems result from a desire to demonstrate operational capability before the system design and development have been completed. Relaxing these requirements would raise the probability of success in the first step of demonstrating HTK. Finally, the weapon system must demonstrate that it is robust to a first layer of countermeasures.

While all the programs have a test plan, most programs do not make use of the ground simulation and testing warranted by the difficulty of the in-flight task.

As noted in the Key Judgments, the flight test risk should be reduced to only those uncertainties that must be resolved in flight. Other issues should be resolved in ground testing. night testing should then be used for verification.

The attempt to achieve an early operational capability before EMD is workable for systems and capabilities that are reasonably well in hand. For complex, demanding tasks (i.e., HTK) that have yet to be demonstrated adequately, the drive for early capability is proving to be counterproductive.

The NTW program seems to have modified significantly the demand for early capability; however, the THAAD program continues to pursue this capability with undiminished zeal. The THAAD program should be relieved of this requirement, and the energy and resources should be channeled to the EMD program.

Program office briefings for all the HTK EMD programs stressed operational "urgency" as justification for accelerated, high-risk approaches. For some of these programs, the operational "urgency" generated by the Service seems out of proportion with the joint priorities and the program resources.

To ensure adequate HWIL test facilities and the best use of available facilities, BMDO should conduct a comprehensive review of current ground test capabilities, including capabilities that complement HWIL ( e.g., tethered systems and hover test facilities).

Programs should embrace a testing approach that provides for sequential, replicable, non-destructive ground tests, with simulations and ground test facilities providing the supporting capabilities. End-to-end system simulation is vital to reduce flight test risk, and it should include more use of HWIL testing for critical flight hardware.

KV ground testing should include realistic scene generation as part of a HWIL capability for testing end-to-end KV performance.

If BMDO finds that facilities are not adequate for providing such capabilities to the extent needed for supporting the BMD programs, BMDO should place a high priority on a coherent, near-term investment program to fill the gaps.

A continuing program should also be in place to upgrade capabilities as needed.

BMDO must take a more active role in ensuring adequate preparation for flight testing. This will require aggressive BMDO inithatdves and strong USD(A&T) support.

For example, BMDO should establish the driving philosophy indicated. here. The environmental specification should ensure that crithcal components wills have an adequate margin to deal with unexpected conditions.

A formal process is needed to ensure full certification of the system before each flight test. This process should include rigorous ground testing and software-in-the-loop (SWIL) and HWIL simulations.

Expecting the required development and testing for deployment readiness to be completed by the end of 2000 is unrealistic. The NMD program should be restructured now to provide for adequate, sequential development and testing.

While the 3 + 3 program is not a UOES program in the sense used in THAAD, it carries similar potential for interference with an orderly operational system program. In particular; the 3-year development program is driven by the need to be ready to deploy 3 years later. This has led to a highly compressed and less-than-minimum flight and ground test program.

In any case, these specific recommendations will reduce the NMD risk.

The philosophy of the 3 + 3 program calls for continued evolution of capabilities in the period between a successful readiness review and the actual decision to deploy a system. This approach requires a continuing and vigorous key technology development program to ensure that the program continues to evolve to meet the changing capability needs over time.

In addition to the emphasis in earlier Key Judgments and recommendations, KKV performance warrants extraordinary attention. As a minimum, BMDO should demand extensive testing of candidate KKV systems in the most capable facilitythe KHILS.

This section responds directly to specific questions. Some findings are repetitive points that have already made in the Key Judgments and the recommendations; however, the repetitive points tend to be those most in need of further emphasis.

The collective experience of the study group members suggests that a successful flight test program starts with a realistic schedule. There must be the time and the commitment for a system to pass through a disciplined design process and the painstaking intervening steps that make the system ready for each sequential flight test. Our collective experience also suggests that test dates have to be driven more by successful completion of events than by the calendar.

End-to-end flight readiness evaluations are essential. For BMD programs, these evaluations have too often been most evident and most thorough after a failure instead of before the flight test. These reviews consistently verified the validity of the historic flight test paradigm.

In addition to these historic paradigms that produced successful flight tests of complex systems, the characteristics of HTK programs demand increased emphasis on certain aspects of the paradigm (e.g., design margins, full qualification of components, careful analysis of critical functions and components, thorough ground end-to-end tests, and so forth). The challenge of HTK also warrants additional emphasis on HWIL testing and high-fidelity simulations.

The historic T&E paradigm is still valid and is essential to successful flight test programs, but it is not being adequately followed in BMD programs. This paradigm is affordable for HTK systems and is far less costly than the current, riskier approaches that produce flight test failures, program delays, and possible program failure.

The Peacekeeper program provides a prototypical example of a successful, high-priority development program. It is particularly noteworthy that there were intense pressures for early initial operational capability (IOC) for Peacekeeperfrom the user, from the DoD leadership, and from the National leadership.

Further, the Peacekeeper program was a new generation of a proven weapon (the ICBM), which, while pushing the state of the an, did not demand any fundamental technology not already demonstrated.

Even so, the program took 6 1/2 years from start to first flight and had a rigorous, disciplined, flight test program with adequate time between tests to analyze results before the next testa thoroughly sequential approach. The program reached IOC on time and with less than the planned number of test flights.

Highly relevant lessons also apply to BMD programs from a myriad of spacecraft programs: the need for very high reliability in critical systems, rigorous component and system qualification, and end-to-end ground checkout before the system is launched.

The Sprint Missile program, as an element of the SAFEGUARD system, is another example of a high-priority program that was executed under intense schedule pressures.

With this highly compressed test schedule, the first 10 tests were characterized by a high failure rate. However, this failure rate was made tolerable by the extensive planned series of tests that followed.

The study group heard repeated references to "random" failures. However, few, if any, of these failures were "random"a statistical matter. They were caused by poor design, test planning, and preflight testing deficiencies; poor fabrication; poor management; and lack of rigorous government oversight.

The tendency of the government and program managers to trivialize the causes of these costly failures, combined with the aggressive schedule discussed on the previous slide, has led to a "rush to failure."

We felt that the program manager~both government and contractor underestimated the degree of difficulty in achieving HTK The fact that the contractor simulations often predicted HTK 100 percent of the time gave us the impression that the contractors routinely underestimate the many things that can go wrong. We felt that this lack of appreciation for the complexity of the task continued after experience should have provided compelling evidence to the contrary.

Success in flight testing has depended on detailed, well-documented, integrated test planning vs. depending on a series of leap-ahead demonstrations.

In contest to this need, BMD HTK program planning has beenand much of it still ischaracterized by inadequate and compressed flight and ground testing. Much of the testing is in parallel, with inadequate time between tests to correct problems before the next test. This situation is exacerbated by inadequate provisions to repeat tests that fail.

To the extent that cost or other factors demand minimum test flights, the importance of the other aspects (design margins, component qualification, ground tests, preflight review, and so forth) becomes even more critical. Regardless of the approach, when developing new and unprecedented capabilities, flight test failures will occur. A prudent program will anticipate and account for this reality. Failures can be minimized with incremental, sequential testing; however, they cannot be eliminated.

The study group found significantly increased emphasis on ground testing following test failures. We also found an array of existing capabilities for simulation and ground testing, including concepts that approach end-to-end preflight testing. We did not find consistently coherent plans to make the best use of this array of capabilities nor a rigorous analysis of remaining deficiencies and programs to fill those gaps.

We found approaches that might fill the function that captive-carry fills for air-to-air and air-to-surface missiles: providing find-and-fix opportunities for system interfaces and critical functions. These approaches included tethered systems with dynamic scene generation and expanded hover testing. This capability is critical for reversing the record of failures in HI BMD programs. Again, we did not find the use or planned use of these approaches in BMD programs.

Since the THAAD program has produced the most flight test experience to date, it also provides the richest source of lessons learned for the NMD program.

Numerous reviews and the Integrated Product Team (IPI ) process have reported that initial design and fabrication were not subjected to adequate discipline and quality control Further, the THAAD program office has also a expressed these concerns at various times.

To a large extent, the NMD program will depend on an integration contractor for these functions. The NMD program will need a rigorous approach for ensuring that the early THAAD experience is not repeated in NMD. The 3 + 3 approach leaves no room for the failures of proven technology that can be very nearly eliminated with discipline and quality control.

As already suggested several times, schedule pressure can be a powerful opponent of rigorous design, fabrication, and test discipline. The 3 + 3 approach is likely to produce intense schedule pressures.

Even though an extensive array of simulation and test facilities are available, the study group was unable to find a comprehensive, coherent plan for their use and further development particularly for HWIL testing of critical components.

A comprehensive review of the largely Service-and contractor-developed capabilities is urgently needed to provide a coherent plan for current use and for a DoD investment plan to fill future needs.

We found that the THAAD and NMD programs were not making maximum use of existing facilities. The THAAD contractor felt that the seeker for FT-8 did not need to be subjected to the most rigorous level of testing, because of the large target size in FT-8. The NMD program was also not planning use of the KHILS Facility to team more about the two candidate seekers in its program.