Testing computer software

Nobody knows technology like the Defense Advanced Research Projects Agency (DARPA). It has paved the way for countless innovations over the years, ranging from the Internet to unmanned aerial vehicles. As hardware is developed, prototypes can be performance-tested to make sure the development is on track. But what about the survivability of information systems? When it comes to information on the battlefield, how much robustness and security are enough? That is a question that DARPA sought to answer as part of its UltraLog project.

Ultratog Program Goals

Now in its fourth year of development, the UltraLog uses intelligent agent technology to show that largescale, distributed-logistics command and control systems can survive under wartime conditions and severe cyber- and kinetic attacks. At its core, UltraLog is a high-speed logistics planning and execution system.

When presented with an operations plan (OPLAN), the UltraLog system responds by building a logistics support plan comprising two primary components: detailed time-phased force and deployment data (TPFDD) and a sustainment plan. The TPFDD provides detailed information about what gets moved, conveyances, routes, and start and stop times. The sustainment plan provides information on projected demand, refill, inventory on hand, and potential inventory shortfalls.

During the execution of an OPLAN, the UltraLog system dynamically reworks the transportation plan and recalculates materiel requirements as the operational environment evolves. For example, UltraLog will recalculate and update the plan to account for added or deleted units, delays in or the early arrival of units, and any changes in operating tempo (OPTEMPO).

The project uses a host of survivability technologies to buttress the agent network. The ultimate survivability goal is to deliver 70 percent of the system's performance despite the loss of 45 percent of its infrastructure, such as loss of computers or reduced central processing unit (CPU) memory. Of course, less loss of infrastructure would be expected to yield higher performance.

Functional Assessment

The individual technologies of UltraLog were tested in a laboratory environment. But would an integrated suite be both survivable and militarily useful? To determine this, DARPA brought in a functional assessment team that was from outside of DARPA and independent of its developers.

The assessment was run against a model of the Army's V Corps deployed to Southwest Asia. The model was built at the DARPA Technology Integration Center in Arlington, Virginia. Over 1,000 networked intelligent agents were constructed representing V Corps units, a Future Combat Systems (FCS) unit of action, corps and theater support, and stateside supply and distribution. [An intelligent agent is a software program that uses some degree of intelligence to execute a task without user supervision.] The scenario involved a complex operation requiring a 180-day deployment for units of the 1st Armored Division originating in Europe and the United States.

During the course of the operation, six major OPLAN changes were introduced that required significant replanning. Among these changes, separate hostilities were initiated that required the short-fuse deployment of the FCS unit of action for a 72-hour combat cycle. In all, the scenario involved hundreds of military units, 28,000 major end items, and 33,000 personnel. UltraLog was the high-speed, survivable logistics planning and execution system for the scenario and was expected to rapidly generate high-quality TPFDD, respond to supply and transportation queries, perform real-time dynamic replanning during the execution of the operation, and produce detailed class I (subsistence), III (petroleum, oils, and lubricants [POLl), V (ammunition), and IX (repair parts) data--all while providing user-friendly graphical user interfaces.

The approach to the functional assessment was derived from standard military utility testing and evaluation methodologies. Two logistics measures of effectiveness were used--

* While the system is under cyber- or kinetic attack, does it provide useful warfighting information?

* Does the system produce an executable logistics plan?

The two logistics measures of effectiveness produced 5 logistics operational issues and 24 logistics measures of performance that assessed the system's continued ability to provide accurate, timely, and usable warfighting information. For the most part, standards were derived from the key performance parameters in the Global Combat Support System (GCSS) Capstone Requirements Document.

Testing Robustness and Security

Functional assessment experiments were designed to test the two key survivability features of robustness and security. The steps of the key robustness process in maintaining logistics functionality include detecting a missing or impaired agent (representing a military unit or a collection of military units), deciding how to react, and then remotely restarting the unit on a different computer and restoring all of its supply data on that computer. Another key process is maintaining the system's ability to function in the face of "service denial," which occurs when a cyberattack floods the computer network with so much communications traffic and superfluous data that the system is overloaded and grinds to a halt.

The key to functional logistics security is a secure execution environment with tamper-proof, secure mechanisms. The system needs to prevent unauthorized access as well as unauthorized operations by those who have access. The system also needs to prevent malicious information from being inserted. Security Red Teams launched 20 classes of cyberattacks against the network from both external and insider sources. [The performance of UltraLog's security function will be the subject of an article in the next issue of Army Logistician.]

The functional assessment of robustness ran over 170 experiments. These included--

* Degrading computer CPU resources by up to 75 percent. Excursions were run that degraded the entire system by 90 percent and selected units by 85 percent.

* Degrading memory (computer processing capacity) by up to 75 percent.

* Cutting communication links among various combat and support units.

* Degrading bandwidth along various communications links.

* Removing the logistics capability of support units in various supply chains. These were called, appropriately, "sledgehammer experiments."

But a true functional assessment is more than stresses caused by cyber- or kinetic attacks; the day-to-day logistics fog of war also must be included. To account for the disruptions created by the fog of war, a number of perturbations were injected simultaneously into the scenario. These included delaying deployments, changing unit OPTEMPO, and spiking demands for selected items of stock.

Typically, an experiment involved introducing a change to the OPLAN while simultaneously initiating one or more of the stresses listed above. For example, in one experiment, the arrival in theater of the 2d Brigade, 1st Armored Division, was delayed 5 days while communications were constrained to 56 kilobits per second between the already-deployed 1st Armored Division headquarters and its 1st Brigade and between the unit of action command element and the unit of action's 1st Combined Arms Battalion. Selected computers within the combined arms battalion also were disabled.

Another series of experiments focused on attacks against elements of the supply chain. In these, computers were knocked off the net for as many as seven support units. These included such units as the 102d POL Supply Company, the 592d Ordnance Company and the 47th Forward Support Battalion. As in the other experiments, perturbations also were introduced that forced the system to modify transportation and sustainment plans.

A series of logistics queries was used to measure how well the system performed in each experiment. For example, one transportation query was: Do the planned delivery dates and final delivery locations match the requested delivery dates and locations for the equipment and personnel of the 1-35th Armored Battalion, the 1-501st Aviation Battalion, and the 1-6th Infantry Battalion'? A representative sustainment query was: What is the total amount of JP8 fuel requested by selected combat units that consume JP8?