By: Larry Walker, May 2023; Context: Sperry Knowledge Systems Center.
Larry reported to Sperry Univac’s Development Center located in Roseville, Minnesota in the Fall of 1966. Larry knew nothing about Univac systems, nor did he know any individuals who worked in the Roseville Center. His assignment was to become familiar with Univac’s Exec 8 Operating System. Exec 8 was the flagship OS for Univac’s 1100 computer line and had been selected to be the foundation for the UAL reservation project.
Competitive with IBM’s OS 360, Exec 8 was a rich, full-function operating system. The programming staff responsible for its development and maintenance numbered between 500 and 600 people. A print-out of the full OS filled 4 large binders of regular computer print pages, containing over 4 million instructions that comprised the OS software.
Larry had to first learn the 1100 system Assembly Language and what each of the over 100 computer hardware instructions did. Then the OS printouts would begin to make some sense. Helpful individuals presented an overview of the OS software’s component parts, and Larry dug in to get a feel for what all this massive OS did.
He slowly, but surely, began to learn which individuals were the ‘go-to’ people for which parts of the OS. Way larger than CDC’s 3200 Master OS, Exec 8’s structure emerged. He learned to navigate the 4 binders of code and began to recognize when he had to take questions to the experts.
This process went on for a little over 3 months, a time-frame consistent with the expected arrival of their fourth child in February. During this time, the house was sold, and personal belongings were packed in preparation for departure to Chicago, site of the UAL project.
The day they were totally ready to depart, a blizzard struck Minnesota and was projected to hit Wisconsin and Chicago as well. Larry, Judy, 3 toddlers, plus the brand-new baby were all sitting on their suitcases wondering what to do. The wind was howling, snow was falling when Larry said, “Tell you what. We can’t stay here, no beds. Let’s get in the car, and if the weather is too bad to drive, we will pull into the first motel we see.”
That plan was acceptable, so they set off. Once they got to the freeway, an interesting perspective greeted them. The blizzard warning had scared all other drivers off the road, and the wind was blowing so hard that no snow whatsoever gathered on the roadway. So, Larry kept driving.
It turned out to be this way all the way to Chicago, where they arrived late in the evening. When they arrived at their targeted motel, the driveway was not plowed, so all they could do was rev up and plow in as far as they could get. Once planted in the snowbank, they moved into their motel room. The Chicago papers the next day reported: “The Great Snow. 24 inches of snow in 24 hours!”
Larry was reminded that none of Munich, Brussels, nor Paris had ever had this big a snow.
***
The UAL project was housed in United facilities about 2 miles from O’Hare Airport, the busiest airport in the world at that time. Reporting to work, Larry met the project team and was assigned to be a part of the executive software team, the team responsible for tying all the elements of the project together. He had a lot to learn.
Some of the project team had experience with Univac’s work at the NASA Goddard Space Center. These people had the real-time software experience essential to meeting the project’s high volume transaction numbers.
The computer facility for this project was massive to hold the hardware necessary to meet the ambitious project goals. There were 4 mainframe 1108 processing computers. This was the first commercial multi-processing system where all 4 CPU’s simultaneously operated out of a single core memory. Each unit had to meticulously ‘lock’ and ‘unlock’ any of the memory pieces that were used in common by the 4 units.
There were over a dozen Fastrand devices, massive storage units capable of holding the data for United’s fleet of passenger planes. Fastrands were over 5 feet long and about 4 feet high. They could hold large amounts of data, but access time to retrieve the data was ponderous and slow. Once United had all the Fastrand running, one of their operators noticed that this massive momentum had cracked the cement wall next to the Fastrand.
There were also between 15 to 20 magnetic tape units. Connections to the 5,000 CRT terminals nationwide involved over 100 communication lines running in parallel. The system was set up to poll (ask for input) each of these lines every 100 milliseconds. The blinking lights that showed this polling activity on all the lines was a fascinating light display to watch – a very random blinking of lights here and there. One day, one of the engineers was watching this and asked, “How often do your poll each of these lines?” The response was: “Every 100 milli-seconds.” He responded, “Well, I just did an estimate, and I believe transmission time of the poll to the West Coast would be about 120 milli-seconds. So, you would be sending a second poll before you get an answer to the 1st poll.” The value for the polling rate was upped to 120 milli-seconds – and – the blinking lightboard began to pulse at a regular rhythm – not as interesting, but a much more efficient use of resources.
There were also many disc units, which held less data, but had much faster retrieval times. Then new to Larry were the 432 drums, small capacity but 8 milli- second average retrieval time.
CDC did not have devices anything like the 432 drums. They had discs, but their average retrieval time was 35 milli-seconds – 4 times slower than the drums. These devices were one of the reasons Univac could undertake a project like the UAL project, and CDC could not. Realizing the importance of the 8 milli-second access time, United planned to move all the reservation records for today’s flights to the drums. This is where the high-level transaction volume would hit the most.
Somewhat later Larry uncovered the second hardware capability that Univac had that CDC did not have. Every one of the 5,000 CRT terminals was equipped with an Externally Specified Index (ESI) number. This self-identification greatly reduced the processing time to identify where each query was coming from. CDC did not have ESI either.
432 Drums and ESI-specified terminals were two key advantages for Univac. One consequence of this was that Univac had about ⅓ of the airline reservation systems in the world, and IBM had the remaining ⅔.
This was one of Univac’s strongest market areas.
***
The main software tasks for the project were two-fold. UAL programmers were responsible for all the reservations queries and related applications.
Another project innovation is that these programmers would write their applications in Fortran, a high-level language more efficient at generating code than Assembly Language. Fortran was strengthened with the addition of a library of common routines that could be re-used by multiple applications.
UAL also committed resources to establish and maintain their massive database. UAL allowed reservations up to a year in advance, and as by far the largest airline in the world, their database was large – and precious.This responsibility included moving today’s database to the 432 drums each and every day.
The Univac programmers were tasked with developing the real-time features that must be added to Exec 8 to meet the high-volume transaction rates demanded of this project. This software was named Transaction Interface Processor (TIP). The TIP software had to embed features and shortcuts in the very depths of Exec 8’s 4 million lines of code. The TIP programmers were being asked to streamline the very ponderous, large Exec 8 to become a sleek, trim high-volume real-time system.
Univac also had to provide the user interface that supported the Fortran applications being developed by UAL programmers. Univac had one Fortran expert charged with making both sides of this situation work.
Univac had between 40 and 50 staff onsite: project leadership, administration, programmers, and hardware maintenance engineers – major project staffing.
Larry was the newest member on the Executive Software team, the team responsible for the bulk of changes needed to Exec 8. Larry was the only member of the team who had been trained in Roseville on Exec 8, so he had no trouble fitting in.
Making changes to complex software that you had no part in writing is a challenging task – errors proliferate – and must be tracked down and eliminated. Larry’s role evolved to become a major ‘debugger’ (error finder). When an error occurred, the system operators would take a ‘dump’ (capture the complete system state at the time of failure – and print it out.These memory dumps created printouts about 5 inches thick – showing the state of all system queues, registers, active programs, etc. etc. Univac’s Fortran expert wrote a program that located all these key parameters and put them in one readily accessible place.
Larry’s cubicle became a warehouse for the multiple dumps taken every day. The expert dump software made life bearable, and Larry loved the challenge of solving the ‘puzzle’ presented by each error. The most challenging were those where a multiprocessing “lock” or “unlock” was mishandled. Failures of this kind meant two processors both were using a single piece of memory – but for different purposes.
Larry wrote a simple test transaction – all it did was to open itself 2 times – and exit. This exponential transaction growth quickly brought the entire system down – usually in less than a minute.
Turns out that the UAL high volume transaction processing environment quickly hit timing errors that the more typical ponderous batch processing seldom encountered. Univac Chicago’s continuous submission of unusual errors were met by disbelief in Roseville as their hundreds of lower volume users were never reporting such issues. Trust was low between the two sites.
Implementing the necessary changes took much longer than anticipated. Delivery schedules were missed and missed. At one point, the programming staff was put on 12-hour workdays, 7 days a week. This lasted over a year but was not enough to meet the committed schedules.
***
It took 3 years to insert all the needed changes into the Exec 8 foundation.
So – the project was ready for a ‘live’ demonstration – 5,000 CRT terminals manned by UAL ticket agents around the country would all be live. Excitement was high to see how well the system performed relative to its two major goals, 1 second response to every terminal, every time and 110,000 reservation transactions per hour were handled.
Detailed measurements were built into the software, so at the end of the demonstration, it would be possible to clearly see what the performance was. A UAL/Univac project leadership meeting was scheduled for the following day. Larry presented the performance measures. The main goal: 110,000 transactions per hour. Actual rate during the demonstration:
4,000 transactions per hour!
The room sat in stunned silence. Larry still remembers seeing the UAL project manager turn ‘white as a ghost’ – close to dying on the spot.
***
This is when ‘Truth’ hit the fan!
Previously, the Univac programming team had worked hard. Now, they entered the school of hard knocks.
Univac brought in experts from many of its previous airline projects to unlock the performance of the TIP system. Turned out, most of them could not in a short visit grasp the complexity of the TIP solution. The exception was Ed Mack who previously led implementation of the Eastern Airline reservation system. Ed visited multiple times over a 3-month period.
He helped the Univac team determine that the current implementation had exhausted 100% of core memory, 100% of input-output capacity, and 100% of CPU instruction processing capacity.
On his last visit he threw up his hands in dismay. His parting comment: “I kept looking for some major logic error in the design of this solution. Now that I fully comprehend how TIP works, I realize it is logically correct. It is just ungodly slow! I can’t imagine it can be fixed.”
The programming team realized that performance had never been a topic of conversation. Their entire focus had been on figuring out how to modify Exec 8, so it could handle transactions at all. During this review period, Larry got to see the project contract. To his amazement, the contract clearly called for a sleek, lean dedicated real time solution – not something grafted on top of the ponderous Exec 8!
Despite this Mack’s forecast and the realization that they were climbing a steep mountain, the programming team dug in even while realizing that every improvement had to be absolutely a reduction in use of every computing system resource – use less memory, use less input-out, use less CPU. Day by day they made progress. Some key improvements:
- While the switching software in Exec 8 took a minimum of 1,000 executed instructions, one day one of the programmers realized that the way the switching software used the “lock-unlock” instructions could lead to incredibly long queuing, unqueuing processes.
- Eventually, they realized that most of the locks were unnecessary as each CPU was focused on a single, unique query transaction. The team deleted the locks and performance leaped ahead. (Roseville refused to accept the logic of these changes.)
- The Fortran expert reviewed the application programs and discovered that some of the applications were making input-output requests inside of Fortran loops – for no good reason. Moving the requests outside of the loops provided another performance leap forward.
While this ‘leaps’ were essential, the rest of the missing performance had to come from painfully reviewing every piece of executive code to reduce the instruction counts, every use of input-output code to reduce instruction counts + serious review to determine if the IO request could be avoided completely, every use of core memory to reduce the amount of space required. And – no tradeoffs were allowed as every resource had been exhausted.
This painful focus on performance went on for a year. Time to demonstrate the progress with another live, nationwide test. The test would again be in the evening to minimize disruption of the airline operations. The test was launched, breaths were held, fingers were crossed. The system seemed to be purring ahead. Performance measures had been built in, so the watchers could get a glimpse. At the end of the test period, the average performance over the test period was reported: 90,000 transactions per hour!
UAL project team members were dancing on desktops. Project leaders were relieved. While still short of the 110,000/hour required, Univac also had promised to deliver their latest computer mainframe which was fast enough to make up the difference.
Larry went home satisfied that 4+ years of his life had not been wasted.
***
At 9:00 the next morning, Larry got a call from Univac management:
UAL had canceled the contract and demanded immediate removal of the system.
Crushed, Larry then had to call the other 7 members of his executive team to deliver the incomprehensible bad news.
Reflection:
Lou Thomas was the hands-on Univac UAL project manager. TIP was his visionary solution for the UAL contract. When the UAL contract was canceled, Lou took TIP to Univac’s Salt Lake City Division which built Univac’s communication devices. TIP became a successful real time processing add-on to Exec 8 where it competed with IBM’s CICS product, an add-on to their OS 360 operating system.