NASA’s Langley Research Center in Virginia is addressing runway and taxiway incursion threats by testing a variety of new technologies, one of the most interesting of which is a miniature head-up display that the pilot wears on his head and positions in front of the eye.

The display might not be the most elegant piece of equipment ever to grace the flight deck, but rarely must scientific experimentation focus solely on aesthetics. Unlike the integrated helmet and display sight systems used in the Army’s Apache attack helicopters, the head-worn display NASA is testing would mount to the side of the pilot’s headset in front of the right or left eye (depending on which is dominant). The goal is to provide the wearer with seemingly magical powers of vision, allowing the pilots to see a synthetic model of the world in every direction, even in zero-zero weather conditions.

During taxi, the color image in the display can include realistic-looking runways, taxiways, buildings and, perhaps at some point in the future, other aircraft and ground service vehicles linked via ADS-B (automatic dependent surveillance-broadcast). NASA thinks the concept could eventually be applied during all phases of flight, giving pilots an almost unlimited field of view that would allow them, for example, to peer straight down through the floor of the cockpit and see the computer-generated earth below.

Incursion Danger Still Real
For the time being, researchers are focusing on applying head-worn-display technology and other advanced concepts just to the taxi environment. And for good reason. Runway and taxiway incursions are near the top of the list of aviation dangers in the U.S., occurring with such regularity at major airports that many experts believe it’s only a matter of time before a major accident happens.

Several times in the last six months, transport-category airplanes have narrowly avoided disaster on the ground while taxiing or taking off. Last July a departing United Airlines 737 came within an estimated 300 feet of colliding with a 747 freighter on an intersecting runway at Chicago O’Hare International Airport, an incident blamed on controller error. Less than a week later two regional airliners nearly collided at Los Angeles International Airport when an America West jet strayed onto the active runway into the path of a departing SkyWest Embraer Brasilia. A warning by the tower and the quick reaction of the SkyWest pilot averted disaster as the airplanes missed each other by an estimated 150 feet. In yet another dramatic incident at LAX, a SkyWest pilot in early October had to slam on his brakes when a Gulfstream V strayed onto the runway. The SkyWest jet shuddered to a stop less than 100 feet from the business jet. Then, at the end of October, a loaded Lufthansa 747 “brushed” the wing of an empty Continental 757 at Newark, causing damage to both airplanes.

The rise in runway incursion incidents contrasts sharply with other traditional dangers such as CFIT (controlled flight into terrain) which have declined dramatically in recent years thanks to warning devices for the cockpit. The NTSB in November said it wants the FAA to address holes on runway safety the way it has for other trouble areas, with NTSB chairman Mark Rosenker calling the current situation “unacceptable.”

The FAA in 2003 issued an advisory circular (AC 120-57) for professional crews that focused on improving coordination and communication during taxiing. Clearly, that document hasn’t had the major impact on curbing incursions that the agency hoped it would. Now, the FAA Safety Team is holding a series of pilot workshops around the country aimed at educating crewmembers about ways to avoid some of the common pitfalls and traps at busy tower-controlled airports. NASA wants to take things a step further.

The researchers at NASA’s Langley center are exploring advanced technologies to help halt runway and taxiway incursions. In addition to the head-worn display, a series of simulator sessions involving professional pilots conducted in the last few months brought to the cockpit moving airport surface maps and head-up display (HUD) symbology developed to give pilots better information about their location on the airport, which in turn is supposed to enhance situational awareness.

“Any of those high-profile runway incursions that have occurred in the last six months, we have the technology to prevent them,” said Lance Prinzel, a NASA research psychologist who is helping to develop the new technologies. In all, NASA invited 15 groups of pilots to spend a full day each in the simulator, driving around Chicago O’Hare at night and in 700-foot RVR visibility in a simulated 757. The goal of the research, according to Prinzel, was to evaluate concepts that could provide “equivalent visual taxi operations” even in extremely low visibility.

On the day AIN visited NASA to observe one of the research sessions, Ed and Dave, seasoned 747 pilots with a major airline, were paired in the simulator for the standard eight-hour evaluation day. As we would all soon learn, driving around a busy airport at night or when the visibility drops can be hazardous duty.

A Day in the Sim
The crew routinely flies together on the route between Washington Dulles Airport and Frankfurt, Germany, so they know a thing or two about taxiing a big airplane on big airports. Both admitted it had been “a few years” since they’d flown into O’Hare and that their knowledge of the airport’s maze of taxiways, terminals and gates was fuzzy–but really, how hard could it be?

As it turns out, pretty hard. On this day, Ed, the captain on the tiller in the left seat, made a number of blunders, some of them major (as when he twice plowed into the side of other airplanes) and some minor (if getting lost at one of the U.S.’s busiest airports can be considered minor). To be fair, the NASA simulation crew threw its share of curve balls at the pilots. Still, it was clear that the technology intended to reduce errors for some reason was having the opposite effect.

During a break in the action, Ed related to the researchers that the head-worn display was drawing an undue amount of his attention inside the cockpit and that as a result he would periodically “lose situational awareness.” Also, trying to correlate the image in the head-worn display to the real world outside was at times quite difficult, he said.

Before discussing some of the head-worn display’s apparent shortcomings, it’s useful to describe what the device is. The prototype hardware used for the evaluation sessions is big and bulky. To put it on, the wearer has to don an official Tony Hawk skateboard helmet and then position the display (a small window of glass on an articulating arm that protrudes awkwardly from the side of the helmet) not over his eye, but rather over his eyebrow.

Why the eyebrow and not the eye itself? Because the display blocks the view of one eye, meaning the pilot can’t see out of that eye and therefore would lose depth perception. If the pilot wants to look through the display, he is expected to tilt his head forward and move just his eyeball up. The result is a pain in the neck, literally.

Before starting the simulator run, the sim operator must ensure that the display is properly aligned, or bore sighted, to the real world. This is accomplished by having the wearer stare straight ahead and aligning an X in the display with an X on the simulator visual screen. Directly behind the pilot’s head is an optical “head motion sensor” measuring four inches high and about 16 inches wide. A cable runs from the side of the pilot’s helmet to this sensor bar. It’s not the most convenient or space-friendly design, but Prinzel promised the production version of the hardware wouldn’t include the helmet, would have a smaller and lighter display and would automatically align itself with the real world without the aid of a NASA scientist.

Once the display is correctly aligned, the view through it is more or less in synch with the real world outside, allowing the pilot to look all around and see a video-game-like image of his surroundings. It’s like having a looking glass that gives the wearer Superman’s X-ray vision. The trouble is, the view in the display is rather jumpy and it can’t keep up with even normal side-to-side and up-and-down head movements. In only a short amount of time spent taxiing around O’Hare with this contraption strapped to one’s head, it becomes evident that the device would probably do more harm than good in a real airplane.

Potential Pitfalls of Head-worn Display
The biggest potential problem, and the one that the 747 captain complained most about, was in becoming so engrossed in the synthetic view inside the tiny display that one completely forgets to take a look outside the cockpit. Also, having one eye completely blocked by a video screen is almost as bad as having both eyes covered, not only because you lose depth perception but also because the brain can deal with only one complex image at a time. True, this is why the display is supposed to be worn slightly above the eye, but it’s tough to continuously transition between
it and the real world. Another unintended consequence of the head-worn device is that pilots apparently taxi much more slowly when using it, perhaps because of the display’s inherent latency.

This was one of the last crews to go through the sim session and it was clear, as Prinzel admitted, that the head-wearable display concept has some limitations that need to be overcome. He pointed out, however, that the prototype display was a $1,500 off-the-shelf model that didn’t provide the fidelity of a full production version, selling for perhaps as much as $100,000. The production display would also be much smaller and would simply clip to the side of the headset, making its use far more palatable.

U.S. soldiers and Apache helicopter pilots routinely use the technology, he reminded, meaning there could still be a place for the head-worn display in civil aviation someday. The keys seem to reside in solving the latency problem, developing a display that the pilot can partially see through and providing better resolution than the simple color VGA capability of this early prototype.

Several of the other runway-incursion-prevention technologies used in the evaluation were very well received by the 747 crew. An airport map showing the airplane’s GPS-derived position received kudos from both pilots, as did a magenta course line (presumably sent via secure datalink from ground control) that showed the airplane’s taxi route from its present position all the way to the gate. In fact, when presented with only this information and without the skateboard helmet getup, the pilots had no problem scooting around the airport.

Using the regular HUD with special symbology that alerted them to upcoming turns and even other aircraft, the pilots did well, although the limited field of view of the HUD clearly made the wearable display more attractive. Eventually NASA researchers think the head-worn-display concept can be refined to a point that it can be used in flight. Some are even imagining a day when pilots would pop in a wireless contact lens that would provide a host of flight-related information without the need for any “display” at all.