InfoWorld 100 September 18, 1995  1. Florida Power Corp.

"At a nuclear power plant, we're always preparing for the worst."

- Brook Julias, nuclear computer and control specialist

Client/server at core of nuclear plant

By Deborah Asbrand

Project at a glance:

Challenge:	Distribute real-time and archived data and replace an aging minicomputer system.
Solution:	A client/server system that integrated with the plant's 1,900 instruments, such as pumps and valves.
Benefit:	It's helping cut maintenance costs and providing information never before available for analyzing plant-system problems.
Surprise:	A positive relationship developed with the systems integrator.  In this project, the vendor's answer has been, "Sure, we can do that."
Key:	The biggest hurdle was the operating system.  The project required an OS that would work in real time and also coexist with the corporate TCP/IP network.  Windows NT fits the bill the closest.
Advice:	Choose an open system, so you can add bandwidth as needed.

strategy:  288      technology:  348      implementation:  364       total points:  1,000

Brook Julias and the MODCOMP Classic minicomputer began their respective careers at the Crystal River Nuclear Power Plant in 1981.  The relationship, however, has never been a warm one, and when the MODCOMP is retired at the end of this year, Julias will have been almost single-handedly responsible for its ouster.  In the minicomputer's place will be a client/server system that's as dramatic in the simplicity of its components as in the sweeping scope of its responsibility.

Using Windows NT and Intel-based PCs, Julias and the Crystal River plant have charged into the staid field of industrial processing and created a sophisticated system for ferrying the data collected from thousands of instruments into the hands of plant operators and engineers.

Crystal River earned its place at the top of this year's InfoWorld 100 not just for the elegance of its distributed architecture but for its sense of adventure - and its success - in tying the architecture so seamlessly to the plant's labyrinth of instruments.  And the point of the new client/server system is more than just modernization.  It's increased safety and efficiency.

"In this industry, you need to be careful that what you put in will work," says Julias, nuclear computer and control specialist in the plant's instrument and control section and the driving force behind the plant's conversion to client/server.

For years, the MODCOMP minicomputer has served faithfully as the process monitoring computer for the uranium fueled plant, located 7 miles north of Crystal River on Florida's west coast.  The 4,700 acre site also contains four coal plants, which, along with the nuclear plant, supply electricity to more than 1 million customers and make Florida Power Corp. one of the major energy producers in the Southeast.

The MODCOMP's charter mission was to collect 3,700 data points every few seconds.  After performing some massive number crunching, the minicomputer flashed the results on the control room monitors read by Nuclear Regulatory Commission (NRC) licensed operators.  Its assistance was reliable, albeit limited.  For starters, the minicomputer kept the data locked up tight, permitting few to view and study the information.  Further, the MODCOMP balked at sharing data over Crystal River's NetWare 3.1 LAN or Florida Power's corporate WAN, administered by the utility's headquarters 100 miles south in St. Petersburg.

Data gathering required legwork and lots of stamina, and little information was kept on permanent record.  Technicians could visit the control room to jot down current real-time statistics or to view the results of routine surveillance procedures and thumb through old fashioned logs filled in by an operator who had walked through the plant taking manual readings.  Databases of information and graphics were mere fantasy.  Security was nonexistent.

"The MODCOMP's security feature is that it's so obtuse no one knows how to use it," Julias says.

THE PILOT PROGRAM.  When the tab for a planned 1990 hard drive upgrade mounted to $500,000 - including the reworked operating system, firmware, software recompiling, and the verification required by the NRC - Julias became convinced that the plant's computing future lay in PC-based technology.  Later that year, he persuaded management to fund a $220,000 pilot project to test his idea.

In October 1991, a pilot application for archiving instrument data went live and was an immediate success, clearly demonstrating the advantages of jettisoning the plant's aging hardware in favor of client/server technologies.  The following spring, Crystal River put the project out for bid.  By the fall, Julias was working with the winning bidder, systems integrator Electronic Visions Inc. (EVI), in Rockledge, FL, to build a distributed PC network that would replace the MODCOMP and add new systems for collecting and archiving real-time data.

Julias, EVI Vice President Howard McGinnis, and Crystal River computer specialist Barry Baumgardner thought big as they planned the new system.  Called Plant Integrated Computer System, or PICS, it would consist not only of an Ethernet LAN connected to the engineers' and operators' desktops, but also of a separate LAN that would link a complex web of multiplexing (mux) computers.  The multiplexers would read and archive the 3,700 data points that the MODCOMP was tallying and run a real-time database that spit out split-second readings.  The new database, called the Safety Parameter Display System, or SPDS, would carry crucial data on plant conditions, such as the reactor cooling system's pressure and temperature.  It would read analog instruments five times each second and digital instruments once every millisecond.

NT POWER.  What would supply the horsepower for such a power hungry system?  Shrink wrapped networking products.

"We wanted to keep the service parts we needed to a minimum, and we didn't want to be limited in the vendors we could buy from," Julias says.  That meant off-the-shelf products.  It also meant taking the radical step of choosing Windows NT 3.5 over the typical industrial system foundation of Unix or a proprietary operating system.

"NT hasn't been around that long, so we don't have a lot of peers doing this," McGinnis says.  NT as the application server would accomplish Julias' goal of reducing network traffic by confining as much processing power as possible to the server.

Other elements of the systems were dictated by Florida Power's corporate standards.  The communications protocol had to be TCP/IP to be compatible with the utility's fiber-optic WAN, and the client side of the network had to be PCs running Windows 3.1.

Among the system's only esoteric components is the database.  SQL databases may be the rage, but Crystal River went with Cbase from Citadel Software Inc., in Brookville, Ind., a database management package that trades SQL's bells and whistles for better performance and control of the static data - such as alarm set points and conversion data - that PICS archives.

IT'S ALL IN THE DATA.  The heart of PICS is the data acquisition system.  It's a collection of diskless multiplexing computers that receive data from the plant's instruments and feed it back to the PICS LAN.  The multiplexing computers are the workhorses of the PICS system.  They acquire signals and convert them to a representation of temperature or pressure, for example, and then check for alarm conditions before relaying significant changes to the mux server.

"Without the multiplexers, nothing else happens," McGinnis says.

The multiplexing computers are outfitted with 486 processor boards that plug in to each unit's 8 bit STD bus.  The STD bus is to the PC bus as the American Heritage Dictionary is to a paperback book - heftier and much harder to dislodge.  Developers use the sturdier bus to reinforce industrial systems.

EVI packed enough intelligence into the multiplexing computers so that when each boots up, it knows to travel over the network to the main mux server and load its database.

The data acquisition system is split into two sections.  The 26 multiplexing computers that monitor the plant activities form one section.  The all important SPDS system that collects real-time data from the instruments wired into it forms the second section.

SPDS consists of three multiplexing computers equipped with cards that convert analog and digital signals into engineering units, such as degrees Fahrenheit, and then transmit the data to the control room for display.

Masterminding operations on each data acquisition section is a mux server.  The mux server's job is to form a bridge between the data acquisition system and the PICS network.  The mux server, a Dell Computer Corp.  Pentium with 32MB of memory, uses two network interface cards to route network traffic.  One card absorbs data from the multiplexing computers and then funnels it through a custom application, which formats the data and sends it to the PICS LAN through the other card.

BUILT TO LAST.  Like the plant's reactor, Crystal River's client/server system is reinforced to withstand just about any act of nature or humankind.

Redundancy is a major system feature.

"At a nuclear power plant, we're always planning for the worst," Julias says.  Each network has a primary and secondary mux server for redundancy.

The SPDS system, which contains the data operators need to complete an emergency shutdown of the plant, is integrated into PICS but designed to continue working on its own should the network crash.  SPDS also has a complete duplicate system for added assurance.

In addition, each of the other major applications - one supplied by the plant's fuel supplier to help run operations and the other for calculating the plant's thermal performance and monitoring efficiency - also runs on a two PC system setup to provide failover.

"We've gone in and started pulling plugs on the Windows machines and the system doesn't skip a beat," Julias says.

Crystal River has made sure PICS had a gradual phase-in.  The MODCOMP will not be officially retired until PICS is not only in working order but has also been verified and validated according to NRC guidelines.  At no time will the plant be without any reporting or alarm capabilities.

Original deployment estimates called for a costly three month plant closing during the PICS installation process.  Careful planning and design of the system helped avoid the plant closing and allowed it to stay online.  That's unique in the nuclear industry, McGinnis says.

"Most of the time transitions like these are slated to occur during scheduled shutdowns, and they rarely go off without hitches," he says.

"The graphical Windows interface is one of PICS' biggest benefits," says Steve Koleff, supervisor of technical support.  Employees are already coming up to speed on Windows through Florida Power's corporate network, so using PICS will come naturally to them, he says.

The new information is already changing the way the engineers work.

Relegated in the past to using outdated systems such as the passive tape based data monitoring system that sat exiled in an electrical equipment cabinet one floor beneath the plant's control room, the engineers can now study water tank levels and other data points from their desks.

"We've found an enormous amount of new uses for monitoring the data," says Walt Newman, senior operations engineer.  By clicking on a Windows icon, he can specify the data he wants, such as electrical output and reactor power.  After downloading the information into Microsoft Excel for analysis, he can compare the data and look for inefficiencies.
PC phobia has kept other engineers from learning to use the spreadsheet and thus the data.

"Engineers here are used to calculators and pencil and paper," Julias says.  "Getting them to use the new tools has been a battle."  So EVI's new front end includes a graphics application that lets users click on as many as 10 points in real time and see them displayed in graph form - with no need to understand cells or formulas.

PICS is also changing life in the control room.  There, operators have traditionally watched over the plant's alarms and reactor on RGB monitors mounted overhead and in the control board, all displaying cryptic, eye straining data points.

Soon, new 28 inch VGA monitors will replace the dreary RGB models with color graphics fed in from the PICS system.

PICS will also be an asset in what is the biggest event in the life of any of the 109 nuclear plants in the United States:  the refueling shutdown they undergo every other year.  During the outages, the plants stop generating power - and revenues.  Crystal River's refueling process typically takes 57 days - some plants require as long as 12 weeks - and Newman expects the PICS system to squeeze shutdown time even further.

"To make the plant as efficient as possible between fuelings, we can look at the individual components and the data," Newman says.

Always looking for backup, Crystal River has made sure that when the PICS network goes live at the end of this year, all of the manual logs and procedures remain intact - just in case.  Says Julias:  "The plant can still run without the computer, but it makes life a lot more difficult."

- Deborah Asbrand