TY - BOOK AU - National Academies of Sciences, Engineering, and Medicine TI - Necessary DoD Range Capabilities to Ensure Operational Superiority of U.S. Defense Systems: Testing for the Future Fight SN - DO - 10.17226/26181 PY - 2021 UR - https://nap.nationalacademies.org/catalog/26181/necessary-dod-range-capabilities-to-ensure-operational-superiority-of-us-defense-systems PB - The National Academies Press CY - Washington, DC LA - English KW - Conflict and Security Issues AB - Rigorous operational testing (OT) of weapon systems procured by the U.S. Department of Defense (DoD) is fundamental to ensuring that these sophisticated systems not only meet their stated requirements, but also perform under realistic operational conditions when faced by determined adversaries employing their own highly capable offensive and defensive weaponry. DoD's test and training range enterprise provides the geography, infrastructure, technology, expertise, processes, and management that make safe, secure, and comprehensive OT possible. The challenges facing the nation's range infrastructure are both increasing and accelerating. Limited test capacity in physical resources and workforce, the age of test infrastructure, the capability to test advanced technologies, and encroachment impact the ability to inform system performance, integrated system performance and the overall pace of testing. Necessary DoD Range Capabilities to Ensure Operational Superiority of U.S. Defense Systems assesses the physical and technical suitability of DoD test and evaluation ranges, infrastructure, and tools for determining the operational effectiveness, suitability, survivability, and lethality of military systems. This report explores modernization, sustainment, operations, and resource challenges for test and evaluation ranges, and makes recommendations to put the DoD range enterprise on a modernization trajectory to meet the needs of OT in the years ahead. ER - TY - BOOK AU - National Research Council TI - Improved Operational Testing and Evaluation: Better Measurement and Test Design for the Interim Brigade Combat Team with Stryker Vehicles: Phase I Report SN - DO - 10.17226/10710 PY - 2003 UR - https://nap.nationalacademies.org/catalog/10710/improved-operational-testing-and-evaluation-better-measurement-and-test-design PB - The National Academies Press CY - Washington, DC LA - English KW - Conflict and Security Issues KW - Behavioral and Social Sciences KW - Surveys and Statistics AB - The U.S. Army Test and Evaluation Command (ATEC) is responsible for the operational testing and evaluation of Army systems in development. ATEC requested that the National Research Council form the Panel on Operational Test Design and Evaluation of the Interim Armored Vehicle (Stryker) to explore three issues concerning the initial operation test plans for the Stryker/Interim Brigade Combat Team (IBCT). First, the panel was asked to examine the measures selected to assess the performance and effectiveness of the Stryker/IBCT in comparison both to requirements and to the baseline system. Second, the panel was asked to review the test design for the Stryker/IBCT initial operational test to see whether it is consistent with best practices. Third, the panel was asked to identify the advantages and disadvantages of techniques for combining operational test data with data from other sources and types of use. In this report the panel presents findings, conclusions, and recommendations pertaining to the first two issues: measures of performance and effectiveness, and test design. The panel intends to prepare a second report that discusses techniques for combining information. ER - TY - BOOK AU - National Research Council TI - Improved Operational Testing and Evaluation and Methods of Combining Test Information for the Stryker Family of Vehicles and Related Army Systems: Phase II Report SN - DO - 10.17226/10871 PY - 2004 UR - https://nap.nationalacademies.org/catalog/10871/improved-operational-testing-and-evaluation-and-methods-of-combining-test-information-for-the-stryker-family-of-vehicles-and-related-army-systems PB - The National Academies Press CY - Washington, DC LA - English KW - Conflict and Security Issues KW - Behavioral and Social Sciences KW - Surveys and Statistics AB - The U.S. Army Test and Evaluation Command (ATEC) is responsible for the operational testing and evaluation of Army systems in development. ATECrequested that the National Research Council form the Panel on Operational Test Design and Evaluation of the Interim Armored Vehicle (Stryker). The charge to this panel was to explore three issues concerning the IOT plans for the Stryker/SBCT. First, the panel was asked to examine the measures selected to assess the performance and effectiveness of the Stryker/SBCT in comparison both to requirements and to the baseline system. Second, the panel was asked to review the test design for the Stryker/SBCT initial operational test to see whether it is consistent with best practices. Third, the panel was asked to identify the advantages and disadvantages of techniques for combining operational test data with data from other sources and types of use. In a previous report (appended to the current report) the panel presented findings, conclusions, and recommendations pertaining to the first two issues: measures of performance and effectiveness, and test design. In the current report, the panel discusses techniques for combining information. ER - TY - BOOK AU - National Research Council TI - Industrial Methods for the Effective Development and Testing of Defense Systems SN - DO - 10.17226/13291 PY - 2012 UR - https://nap.nationalacademies.org/catalog/13291/industrial-methods-for-the-effective-development-and-testing-of-defense-systems PB - The National Academies Press CY - Washington, DC LA - English KW - Engineering and Technology KW - Conflict and Security Issues AB - During the past decade and a half, the National Research Council, through its Committee on National Statistics, has carried out a number of studies on the application of statistical methods to improve the testing and development of defense systems. These studies were intended to provide advice to the Department of Defense (DOD), which sponsored these studies. The previous studies have been concerned with the role of statistical methods in testing and evaluation, reliability practices, software methods, combining information, and evolutionary acquisition. Industrial Methods for the Effective Testing and Development of Defense Systems is the latest in a series of studies, and unlike earlier studies, this report identifies current engineering practices that have proved successful in industrial applications for system development and testing. This report explores how developmental and operational testing, modeling and simulation, and related techniques can improve the development and performance of defense systems, particularly techniques that have been shown to be effective in industrial applications and are likely to be useful in defense system development. In addition to the broad issues, the report identifies three specific topics for its focus: finding failure modes earlier, technology maturity, and use of all relevant information for operational assessments. ER - TY - BOOK AU - National Research Council TI - Reliability Growth: Enhancing Defense System Reliability SN - DO - 10.17226/18987 PY - 2015 UR - https://nap.nationalacademies.org/catalog/18987/reliability-growth-enhancing-defense-system-reliability PB - The National Academies Press CY - Washington, DC LA - English KW - Conflict and Security Issues KW - Behavioral and Social Sciences KW - Surveys and Statistics AB - A high percentage of defense systems fail to meet their reliability requirements. This is a serious problem for the U.S. Department of Defense (DOD), as well as the nation. Those systems are not only less likely to successfully carry out their intended missions, but they also could endanger the lives of the operators. Furthermore, reliability failures discovered after deployment can result in costly and strategic delays and the need for expensive redesign, which often limits the tactical situations in which the system can be used. Finally, systems that fail to meet their reliability requirements are much more likely to need additional scheduled and unscheduled maintenance and to need more spare parts and possibly replacement systems, all of which can substantially increase the life-cycle costs of a system. Beginning in 2008, DOD undertook a concerted effort to raise the priority of reliability through greater use of design for reliability techniques, reliability growth testing, and formal reliability growth modeling, by both the contractors and DOD units. To this end, handbooks, guidances, and formal memoranda were revised or newly issued to reduce the frequency of reliability deficiencies for defense systems in operational testing and the effects of those deficiencies. Reliability Growth evaluates these recent changes and, more generally, assesses how current DOD principles and practices could be modified to increase the likelihood that defense systems will satisfy their reliability requirements. This report examines changes to the reliability requirements for proposed systems; defines modern design and testing for reliability; discusses the contractor's role in reliability testing; and summarizes the current state of formal reliability growth modeling. The recommendations of Reliability Growth will improve the reliability of defense systems and protect the health of the valuable personnel who operate them. ER - TY - BOOK AU - National Research Council A2 - Francisco Samaniego A2 - Michael Cohen TI - Reliability Issues for DOD Systems: Report of a Workshop SN - DO - 10.17226/10561 PY - 2002 UR - https://nap.nationalacademies.org/catalog/10561/reliability-issues-for-dod-systems-report-of-a-workshop PB - The National Academies Press CY - Washington, DC LA - English KW - Conflict and Security Issues KW - Behavioral and Social Sciences KW - Surveys and Statistics AB - The final report of the National Research Council's (NRC) Panel on Statistical Methods for Testing and Evaluating Defense Systems (National Research Council, 1998) was intended to provide broad advice to the U.S. Department of Defense (DoD) on current statistical methods and principles that could be applied to the developmental and operational testing and evaluation of defense systems. To that end, the report contained chapters on the use of testing as a tool of system development; current methods of experimental design; evaluation methods; methods for testing and assessing reliability, availability, and maintainability; software development and testing; and validation of modeling and simulation for use in operational test and evaluation. While the examination of such a wide variety of topics was useful in helping DoD understand the breadth of problems for which statistical methods could be applied and providing direction as to how the methods currently used could be improved, there was, quite naturally, a lack of detail in each area.To address the need for further detail, two DoD agencies-the Office of the Director of Operational Test and Evaluation and the Office of the Under Secretary of Defense for Acquisition, Technology, and Logistics-asked the NRC's Committee on National Statistics to initiate a series of workshops on statistical issues relevant to defense acquisition. The aim of each workshop is to inform DoD about the methods that represent the statistical state of the art and, through interactions of the statistical and defense communities, explore their relevance for DoD application. ER - TY - BOOK AU - National Research Council A2 - Vijay Nair A2 - Michael L. Cohen TI - Testing of Defense Systems in an Evolutionary Acquisition Environment SN - DO - 10.17226/11575 PY - 2006 UR - https://nap.nationalacademies.org/catalog/11575/testing-of-defense-systems-in-an-evolutionary-acquisition-environment PB - The National Academies Press CY - Washington, DC LA - English KW - Conflict and Security Issues AB - The Department of Defense (DoD) recently adopted evolutionary acquisition, a dynamic strategy for the development and acquisition of its defense systems. Evolutionary defense systems are planned, in advance, to be developed through several stages in a single procurement program. Each stage is planned to produce a viable system which could be fielded. The system requirements for each stage of development may be specified in advance of a given stage or may be decided at the outset of that stage's development. Due to the different stages that comprise an evolutionary system, there exists a need for careful reexamination of current testing and evaluation policies and processes, which were designed for single-stage developments. The Office of the Under Secretary of Defense for Acquisition, Technology and Logistics (USD-AT&L) and the Director of Operational Testing and Evaluation (DOT&E) asked the Committee on National Statistics (CNSTAT) of the National Academies to examine the key issues and implications for defense testing from the introduction of evolutionary acquisition. The CNSTAT was charged with planning and conducting a workshop to study test strategies for the evolutionary acquisition. The committee reviewed defense materials defining evolutionary acquisition and interviewed test officials from the three major test service agencies to understand the current approaches used in testing systems procured through evolutionary acquisition. The committee also examined possible alternatives to identify problems in implementation. At the workshop that took place on December 13-14, 2004, the committee tried to answer many questions including: What are the appropriate roles and objectives for testing in an evolutionary environment?, Can a systematic, disciplined process be developed for testing and evaluation in such a fluid and flexible environment?, and Is there adequate technical expertise within the acquisition community to fully exploit data gathered from previous stages to effectively combine information from various sources for test design and analysis?. Testing of Defense Systems in an Evolutionary Acquisition Environment provides the conclusions and recommendations of the CNSTAT following the workshop and its other investigations. ER - TY - BOOK TI - Statistical Methods for Testing and Evaluating Defense Systems: Interim Report DO - 10.17226/9074 PY - 1995 UR - https://nap.nationalacademies.org/catalog/9074/statistical-methods-for-testing-and-evaluating-defense-systems-interim-report PB - The National Academies Press CY - Washington, DC LA - English KW - Conflict and Security Issues ER - TY - BOOK AU - National Research Council A2 - Michael L. Cohen A2 - John E. Rolph A2 - Duane L. Steffey TI - Statistics, Testing, and Defense Acquisition: New Approaches and Methodological Improvements SN - DO - 10.17226/6037 PY - 1998 UR - https://nap.nationalacademies.org/catalog/6037/statistics-testing-and-defense-acquisition-new-approaches-and-methodological-improvements PB - The National Academies Press CY - Washington, DC LA - English KW - Conflict and Security Issues AB - For every weapons system being developed, the U.S. Department of Defense (DOD) must make a critical decision: Should the system go forward to full-scale production? The answer to that question may involve not only tens of billions of dollars but also the nation's security and military capabilities. In the milestone process used by DOD to answer the basic acquisition question, one component near the end of the process is operational testing, to determine if a system meets the requirements for effectiveness and suitability in realistic battlefield settings. Problems discovered at this stage can cause significant production delays and can necessitate costly system redesign. This book examines the milestone process, as well as the DOD's entire approach to testing and evaluating defense systems. It brings to the topic of defense acquisition the application of scientific statistical principles and practices. ER - TY - BOOK AU - National Research Council A2 - Michael L. Cohen A2 - Duane L. Steffey A2 - John E. Rolph TI - Statistics, Testing, and Defense Acquisition: Background Papers SN - DO - 10.17226/9655 PY - 1999 UR - https://nap.nationalacademies.org/catalog/9655/statistics-testing-and-defense-acquisition-background-papers PB - The National Academies Press CY - Washington, DC LA - English KW - Conflict and Security Issues KW - Math, Chemistry, and Physics KW - Surveys and Statistics AB - The Panel on Statistical Methods for Testing and Evaluating Defense Systems had a broad mandate—to examine the use of statistics in conjunction with defense testing. This involved examining methods for software testing, reliability test planning and estimation, validation of modeling and simulation, and use of modem techniques for experimental design. Given the breadth of these areas, including the great variety of applications and special issues that arise, making a contribution in each of these areas required that the Panel's work and recommendations be at a relatively general level. However, a variety of more specific research issues were either brought to the Panel's attention by members of the test and acquisition community, e.g., what was referred to as Dubin's challenge (addressed in the Panel's interim report), or were identified by members of the panel. In many of these cases the panel thought that a more in-depth analysis or a more detailed application of suggestions or recommendations made by the Panel would either be useful as input to its deliberations or could be used to help communicate more individual views of members of the Panel to the defense test community. This resulted in several research efforts. Given various criteria, especially immediate relevance to the test and acquisition community, the Panel has decided to make available three technical or background papers, each authored by a Panel member jointly with a colleague. These papers are individual contributions and are not a consensus product of the Panel; however, the Panel has drawn from these papers in preparation of its final report: Statistics, Testing, and Defense Acquisition. The Panel has found each of these papers to be extremely useful and they are strongly recommended to readers of the Panel's final report. ER - TY - BOOK TI - Statistical Issues in Defense Analysis and Testing: Summary of a Workshop DO - 10.17226/9686 PY - 1994 UR - https://nap.nationalacademies.org/catalog/9686/statistical-issues-in-defense-analysis-and-testing-summary-of-a PB - The National Academies Press CY - Washington, DC LA - English KW - Math, Chemistry, and Physics KW - Surveys and Statistics ER - TY - BOOK AU - National Academies of Sciences, Engineering, and Medicine A2 - Robert Pool TI - Key Challenges for Effective Testing and Evaluation Across Department of Defense Ranges: Proceedings of a Workshop–in Brief DO - 10.17226/26150 PY - 2021 UR - https://nap.nationalacademies.org/catalog/26150/key-challenges-for-effective-testing-and-evaluation-across-department-of-defense-ranges PB - The National Academies Press CY - Washington, DC LA - English KW - Conflict and Security Issues AB - To protect itself from attacks by foreign forces, the United States relies upon its armed services, which in turn rely upon weapons and other systems to provide them with the tools they need to successfully neutralize adversaries' combat capabilities. Maintaining the armed services' warfighting advantage requires a steady stream of new and improved weapons and technologies. A crucial step for acquiring and using these assets is testing their effectiveness and suitability on Department of Defense (DoD) ranges. The DoD has testing ranges that span the globe, where new military technologies are tested based on real threats, tasks, and environments to ensure their combat readiness. These ranges are a vital aspect of the nation's defense, but will they be able to adequately test the increasingly complex military technologies of the future? Against this backdrop, the DoD's Office of the Director, Operational Test and Evaluation, requested that the Board on Army Research and Development of the National Academies of Sciences, Engineering, and Medicine assess the physical and technical suitability of DoD test and evaluation (T&E) ranges and infrastructure. As part of that task, the study committee convened a 2-day workshop on January 28-29, 2021, to gather information on the challenges facing the nation's military ranges. The workshop brought together experts from the military, industry, and government, who discussed the current status of T&E on military ranges and what will be required to ensure their effectiveness in coming decades. This Proceedings of a Workshop-in Brief summarizes the presentations and discussions that took place at the workshop. ER - TY - BOOK AU - National Research Council TI - Tactical Display for Soldiers: Human Factors Considerations SN - DO - 10.17226/5436 PY - 1997 UR - https://nap.nationalacademies.org/catalog/5436/tactical-display-for-soldiers-human-factors-considerations PB - The National Academies Press CY - Washington, DC LA - English KW - Conflict and Security Issues KW - Behavioral and Social Sciences AB - This book examines the human factors issues associated with the development, testing, and implementation of helmet-mounted display technology in the 21st Century Land Warrior System. Because the framework of analysis is soldier performance with the system in the full range of environments and missions, the book discusses both the military context and the characteristics of the infantry soldiers who will use the system. The major issues covered include the positive and negative effects of such a display on the local and global situation awareness of the individual soldier, an analysis of the visual and psychomotor factors associated with each design feature, design considerations for auditory displays, and physical sources of stress and the implications of the display for affecting the soldier's workload. The book proposes an innovative approach to research and testing based on a three-stage strategy that begins in the laboratory, moves to controlled field studies, and culminates in operational testing. ER - TY - BOOK AU - National Research Council TI - Assessment of Technologies Deployed to Improve Aviation Security: First Report SN - DO - 10.17226/9726 PY - 1999 UR - https://nap.nationalacademies.org/catalog/9726/assessment-of-technologies-deployed-to-improve-aviation-security-first-report PB - The National Academies Press CY - Washington, DC LA - English KW - Engineering and Technology KW - Transportation and Infrastructure AB - This report assesses the operational performance of explosives-detection equipment and hardened unit-loading devices (HULDs) in airports and compares their operational performance to their laboratory performance, with a focus on improving aviation security. ER - TY - BOOK AU - National Research Council TI - Evaluating Testing, Costs, and Benefits of Advanced Spectroscopic Portals for Screening Cargo at Ports of Entry: Interim Report (Abbreviated Version) SN - DO - 10.17226/12699 PY - 2009 UR - https://nap.nationalacademies.org/catalog/12699/evaluating-testing-costs-and-benefits-of-advanced-spectroscopic-portals-for-screening-cargo-at-ports-of-entry PB - The National Academies Press CY - Washington, DC LA - English KW - Conflict and Security Issues KW - Transportation and Infrastructure AB - To improve screening of containerized cargo for nuclear and radiological material that might be entering the United States, the Department of Homeland Security (DHS) is seeking to deploy new radiation detectors, called advanced spectroscopic portals (ASPs). The ASPs are intended to replace some or all of the current system of radiation portal monitors (called PVT RPMs) used in conjunction with handheld radioisotope identifiers (RIIDs) to detect and identify radioactive material in cargo. The U.S. Congress required the Secretary of Homeland Security to certify that ASPs will provide a 'significant increase in operational effectiveness' over continued use of the existing screening devices before DHS can proceed with full-scale procurement of ASPs for deployment. Congress also directed DHS to request this National Research Council study to advise the Secretary of Homeland Security about testing, analysis, costs, and benefits of the ASPs prior to the certification decision. This interim report is based on testing done before 2008; on plans for, observations of, and preliminary results from tests done in 2008; and on the agency's draft cost-benefit analysis as of October 2008. The book provides advice on how DHS' Domestic Nuclear Detection Office (DNDO) can complete and make more rigorous its ASP evaluation for the Secretary and the nation. ER - TY - BOOK AU - National Academies of Sciences, Engineering, and Medicine TI - Enabling DoD's Test Ranges and Infrastructure to Meet Threats and Operational Needs in the 21st Century: Unclassified Summary DO - 10.17226/26607 PY - 2022 UR - https://nap.nationalacademies.org/catalog/26607/enabling-dods-test-ranges-and-infrastructure-to-meet-threats-and-operational-needs-in-the-21st-century PB - The National Academies Press CY - Washington, DC LA - English KW - Conflict and Security Issues AB - The Department of Defense operates several ranges across all service branches to test the effectiveness of military systems in the land, sea, air, space, and cyberspace domains. These ranges and infrastructure represent a critical part of the DoD acquisition and systems development process. The DoD's Office of Test and Evaluation (DOT&E) has asked the Board on Army Research and Development to assess how effectively these ranges fulfill DOT&E's mission to determine operational effectiveness and lethality of systems currently under development. This study will specifically evaluate whether these ranges are prepared to simulate threats, countermeasures, and operations against near-peer adversaries. This publication is the unclassified version of the classified report. ER - TY - BOOK AU - Transportation Research Board AU - National Academies of Sciences, Engineering, and Medicine TI - Improving Safety-Related Rules Compliance in the Public Transportation Industry DO - 10.17226/14593 PY - 2011 UR - https://nap.nationalacademies.org/catalog/14593/improving-safety-related-rules-compliance-in-the-public-transportation-industry PB - The National Academies Press CY - Washington, DC LA - English KW - Transportation and Infrastructure AB - TRB’s Transit Cooperative Research Program (TCRP) Report 149: Improving Safety-Related Rules Compliance in the Public Transportation Industry identifies potential best practices for all of the elements of a comprehensive approach to safety-related rules compliance.The categories of best practices, which correspond to the elements of a safety-related rules compliance program, include screening and selecting employees, training and testing, communication, monitoring rules compliance, responding to noncompliance, and safety management.The report also outlines the features of a prototype safety reporting system for public transportation. ER - TY - BOOK AU - National Research Council TI - Review of Systemization of the Tooele Chemical Agent Disposal Facility SN - DO - 10.17226/5198 PY - 1996 UR - https://nap.nationalacademies.org/catalog/5198/review-of-systemization-of-the-tooele-chemical-agent-disposal-facility PB - The National Academies Press CY - Washington, DC LA - English KW - Environment and Environmental Studies AB - In 1993, at Tooele Army Depot, Utah, the Army completed construction of the Tooele Chemical Agent Disposal Facility (TOCDF), the first complete facility for destruction of lethal unitary chemical agents and munitions to be built in the continental United States. The TOCDF will employ the Army's baseline incineration system to destroy the depot's increment of the nation's aging unitary chemical stockpile. This book assesses Army changes and improvements to the TOCDF in response to recommendations contained in earlier reports of the committee. It assesses aspects of the facility's readiness for safe agent handling and destruction operations, its agent monitoring system, and its site specific risk assessment. ER - TY - BOOK AU - National Research Council TI - Detection of Explosives for Commercial Aviation Security SN - DO - 10.17226/2107 PY - 1993 UR - https://nap.nationalacademies.org/catalog/2107/detection-of-explosives-for-commercial-aviation-security PB - The National Academies Press CY - Washington, DC LA - English KW - Engineering and Technology KW - Conflict and Security Issues AB - This book advises the Federal Aeronautics Administration (FAA) on the detection of small, concealed explosives that a terrorist could plant surreptitiously on a commercial airplane. The book identifies key issues for the FAA regarding explosive detection technology that can be implemented in airport terminals. Recommendations are made in the areas of systems engineering, testing, and technology development. ER - TY - BOOK AU - National Research Council TI - Achieving Effective Acquisition of Information Technology in the Department of Defense SN - DO - 10.17226/12823 PY - 2010 UR - https://nap.nationalacademies.org/catalog/12823/achieving-effective-acquisition-of-information-technology-in-the-department-of-defense PB - The National Academies Press CY - Washington, DC LA - English KW - Conflict and Security Issues KW - Engineering and Technology AB - In the military, information technology (IT) has enabled profound advances in weapons systems and the management and operation of the defense enterprise. A significant portion of the Department of Defense (DOD) budget is spent on capabilities acquired as commercial IT commodities, developmental IT systems that support a broad range of warfighting and functional applications, and IT components embedded in weapons systems. The ability of the DOD and its industrial partners to harness and apply IT for warfighting, command and control and communications, logistics, and transportation has contributed enormously to fielding the world's best defense force. However, despite the DOD's decades of success in leveraging IT across the defense enterprise, the acquisition of IT systems continues to be burdened with serious problems. To address these issues, the National Research Council assembled a group of IT systems acquisition and T&E experts, commercial software developers, software engineers, computer scientists and other academic researchers. The group evaluated applicable legislative requirements, examined the processes and capabilities of the commercial IT sector, analyzed DOD's concepts for systems engineering and testing in virtual environments, and examined the DOD acquisition environment. The present volume summarizes this analysis and also includes recommendations on how to improve the acquisition, systems engineering, and T&E processes to achieve the DOD's network-centric goals. ER -