Steward Observatory
University of Arizona
Tucson, AZ 85721
Proceedings of SPIE conference on Instrumentation in Astronomy VII, 1235, p. 486, (1990)
List of Figures:
| Figure 1 | Figure 2 | Figure 3 |
| Figure 4 | Figure 5 | Figure 6 |
| Figure 7 | Figure 8 | Figure 9 |
Abstract
The process for spincasting 8 meter borosilicate honeycomb mirrors
requires us to heat 14 tons of glass in a complex mold to 1170 C °
while spinning the entire furnace at 6.8 rpm. After casting, the
honeycomb blank must be cooled through the annealing temperature range
at 0.2 C ° per hour. The glass will be in the furnace for an eight
week period. We describe here the computer control system to read the
600 N-type thermocouples and control the 270 8-kilowatt heaters used
in the spincasting furnace. The control system uses a proportional --
integral -- derivative (PID) algorithm to regulate the furnace
temperature to a few degrees over the entire casting cycle.
Considerable design effort has gone into assuring that a component
failure or a control system error does not turn an 8 meter mirror into
an expensive patio ornament. Errors are avoided by four strategic
steps: fault avoidance, fault detection, fault containment and fault
recovery. Examples are provided in each of these categories. System
redundancy begins with three on-board 68000-family VME-bus computers
which control overlapping areas of the furnace. Redundancy extends
down through the temperature measurement and power control systems
with many modular, interleaved, and optically isolated subsystems.
Data logging and system monitoring are achieved with a Sun 3/280
workstation running IRAF in the control room. Rotation of the furnace
is controlled by two 40 HP DC servomotors with speed regulation to
0.1% The oven control system contains over 300 circuit cards,
dozens of subracks and power supplies, more than 3000 connectors of at
least 10 different types. there are more than 20 miles of wire and
cable, most with multiple conductors, ranging from fiber optics
thinner than a hair to power cables more than an inch thick. Results
are presented from a subset of this control system which has been used
to cast three 3.5 meter honeycomb mirrors in 1988 and 1989.
A six meter diameter furnace was constructed on a larger turntable in 1986 to cast three 3.5 meter blanks. Figure 1 shows a cross-section of the furnace now being expanded to produce 8 meter diameter blanks. The full size furnace has an internal diameter of 9.5 meters. There is a cylindrical wall 1.2 meters high, topped by a flat disc and conical lid section. The furnace has as major components a hearth, walls and lid. The circular bottom plate, or hearth, is pieced together from smaller pie-shaped segments of silicon carbide based refractory. The hearth provides the working surface on which to build the complex honeycomb mold. The hearth as well as the walls and lid contains embedded electric heater elements to heat the furnace. Figure 2 shows the heater elements in the furnace hearth.
It is imperative to have a reliable and error free control system, since the direct replacement cost of a failed casting is around two million dollars for glass, refractories and labor plus a year of schedule delay. The following sections give a summary of the oven control system and a series of examples of our efforts to make it reliable.
2. Features of the control system
Each microprocessor controlled power panel communicates with a main on-board VME computer by ASCII serial data through fiber optic cables. The VME computer contains the PID algorithm and tells the control panels which heaters to turn on or off. It then accepts the sense information if voltage is on to the heaters and if current is supplied both at the panel and at the heater. Figure 5 shows the schematic for the microcontroller that runs each power panel.
The control system architecture is shown in Figure 6. Each of the three computers is connected to two digital counter units (DCU) which each record 100 temperature readings. Each computer is also interfaced to three of the nine power control boxes. Telemetry is sent off the oven through fiber optics and slip rings for logging and analysis.
Once a minute, a power level for each heater element is calculated from a weighted average of its neighboring thermocouple temperatures, and the temperature called for at the current point in the schedule of temperature versus time. A proportional, integral, derivative servo algorithm is used calculate the desired power level. A one quadrant servo is used since losses through the insulation of the furnace provide a significant cooling loss. There are separate temperature versus time schedules for each of several zones of heater elements. These schedules can be interlocked so as to synchronize their progress according to real temperature data.
The current minute's power level for each heater element is broken into quarter second periods of full power or no power. The on periods are distributed throughout the minute so as to even the total load on each circuit of ten heater elements. Voltage and current sensors are used on each heater circuit to verify that the heater elements have indeed switched on or off at the moment commanded by the computer.
2.8 Control room interface The real-time control computers are networked to the control room computer, a Sun 3/280, which is the interface to the oven pilot. The control room computer is used to send parameters that govern the temperature control to the real-time computers. The control room computer also reads data back from the real-time computers for logging and monitoring by the oven pilot. Various graphical displays are used to interpret the large volume of data logged by the control room computer. In addition, the oven pilot is alerted to error conditions in the oven thermocouple, heater element, and electronic hardware as well as error conditions in the temperature servo of the oven itself. The oven analysis software runs under the IRAF environment (Interactive Reduction and Analysis Facility, Tody 19863). A typical display is shown in Figure 7.
The primary servo loop contains a Boston Gear P60 series three-phase full-converter speed controller. These controllers provide six-pulse conversion of three-phase 480 volt AC line power to regulated DC for the adjustable speed armature control of our shunt-wound DC motors. As a result of the adjustable armature voltage and a constant shunt field, the drive provides constant torque operation. An optional tach feedback with a TG3 generator is used to improve the speed regulation of the controller to ± 1% .
Our speed regulation requirements of an accuracy of 0.001 RPM and 0.01% long term stability was achieved by adding a custom digital control loop. A digital incremental encoder is attached to the input shaft of the gear box and provides about 10,000 pulses per oven revolution. A six second sample period is required to achieve the 0.001 RPM resolution.
A microprocessor is used to provide the servo parameters, read the control switches, display the speed, provide the control voltage to the controller and communicate with the operator through the oven control computer. The communication is by ASCII serial through fiber optic cables and receives the command speed on a 1 minute interval. The actual oven speed is sent to the control computer on each six second interval.
There are three modes of operation for speed control: manual, thumbwheel and command. In manual control a potentiometer is used to send a control voltage directly to the controller. In thumbwheel and digital command control, the micro sends the proper voltage according to the thumbwheel switches or the commanded value. The oven speed and the command speed are both displayed for each controller. Either micro will control either controller which can be operated together or separately. The usual mode of operation is to have both control systems in operation, but since they are physically connected, one controller will dominate while the other follows closely. If the dominant controller fails, the other one will take over with only a minor perturbation in the speed.
The third 3.5 meter mirror cast in June 1989 had a focal ratio of 1.5 and thus required the oven to spin at 9.227 rpm. The actual average speed for 24041 revolutions was 9.226 rpm. Without any correction for the contraction of the cooling glass, the measured radius of the parabola was 10402 mm compared to the target 10500 mm. This produces a surface error of 1.4 mm which is only slightly larger than the 250 micron rms ripples in the best fit parabola.
3. Reliability and fault tolerance considerations
The first step toward building a fault resistant system comes from designing in that reliability from the start. This involves sound engineering practice, conservatively specified components, careful failure analysis and a tight linkage between designers and technical staff.
The configuration of such a complex system must be carefully managed at both the procurement and assembly stages. The approximately twenty thousand wiring connections in the oven are stored in a dBaseIII database. This database is used to generate wire lists, labels and software initialization tables.
When we've got a workable design and the required materials we can get down to the actual fabrication, assembly, and installation tasks. Every effort is made to plan these tasks carefully, breaking them down into small units which can be readily accomplished. The key to success in this area is an excellent technical staff. As an example, several staff members recently produced 90 circuit cards from about 8000 individual parts, and made only two errors. This quality of workmanship is essential to the construction of a large control system that works.
Not withstanding a high level of capability, quality control procedures, though somewhat informal, are rather rigorous. Every unit is submitted to complete visual inspection and functional test. All wiring is checked at least once. A frequently utilized procedure is to have working technicians trade places to inspect or test each other's work. Although this is to some extent an accommodation to the small staff size, it seems to produce better results than either self inspection or a specially designated inspector. Last, but not least there is the ``astronomer test'' which involves poking, pulling, shaking and switching all parts of the assembled subsystems to see which ones will give up first. Systems which survive the ``astronomer test'' and the engineering quality control process have a very good record under operating conditions.
Whenever possible, components were selected for high reliability and high temperature rating. For instance, the CIM microcomputers were designed to run in an industrial environment. Mil-spec parts were generally not used because of the cost.
All of the computer systems are run on backup power with {\em Best} UPS systems. The temperature instrumentation also runs on rechargeable battery backup power. All of the sensitive equipment is isolated from the noisy line power with isolation transformers. Relays and other noise sources were snubbed and all major subsystems were protected with transorbs.
Before each oven run, the control system is tested and examined with an extensive checklist procedure. This procedure includes a computerized configuration check where each heater element is turned on individually to make sure that current and voltage appear at the right place. As the final test before the glass is placed in the mold, the oven is prefired to 1170 C ° to burn off organic contaminants and to set the ceramic glue bonds.
Each heater element is controlled by at least three nearby thermocouples so that a sensor failure will not shutdown that portion of the oven. Thermocouples and heaters from various subsystems and power sources are interleaved around the oven to minimize the effect of any component failure.
The 4 meter furnace was controlled with a single microprocessor and a hot spare. The calculations were repeated offline as a check in the control room workstation. The 8 meter furnace will have three redundant control computers on-board the rotating furnace.
To verify that each heater element turns on and off at the proper time, threshold current and voltage sensors monitor the condition of each heater circuit. This allows the computer to provide immediate warning if a fuse has blown or an electronic failure has turned on a heater accidentally.
Data and configuration status tables exchanged between computers are transferred in packets. Each packet contains a packet identifier, a time stamp and a checksum to ensure that valid data was transferred.
Temperature measurements on microvolt signals are frequently disturbed by noise sources such as lightning strikes, welding or arc lamps igniting. To prevent these momentary anomalies from disturbing the servo algorithm, the temperature data are temporally filtered. The median of five readings taken at five second intervals is used for the minute-by-minute servo input. Each temperature reading is also compared with those from nearby thermocouples to spatially filter out errant sensor readings.
Watchdog timers are used to make sure the code in each of the control micros is executing correctly.
Approximately 15% of the temperature sensors are used to monitor the condition of the electronics rather than actual oven temperatures. For example, thermocouples sense the gearbox temperature during rotation.
The casting process is also monitored by special air-cooled CCD video cameras mounted in the oven walls. This visual data provides additional data and peace-of-mind to the oven operator.
The temperature measurement system is isolated by fiber optics between the card that reads the thermocouple and the counter. The channel select is optically isolated from the card that reads the thermocouples. This way the system is tolerant of potential differences between chassis, as well as the power being turned off in one chassis. This enables us to perform maintenance of the system without taking the whole system down --- ie. we can remove a thermocouple reader card without affecting the rest of the thermocouples. The modularity also allows interleaving of various subsystems so that a failure cannot disable a large section of the oven.
Power distribution panels were also optically isolated from each other and from the control computers.
The thermocouple instrumentation modules were custom designed and built in-house. Commercial units could have easily given the required number of channels and resolution, but were found lacking in terms of system isolation and durability. Since the thermocouples are interspersed with the heaters, they are controlling, the shield often encounters 277VAC with currents of several amps due to leakage currents in the oven at high temperature. If a shield wire fails, then the full heater voltage is applied to the temperature system inputs. To prevent such a failure from destroying the entire control system, each of the individual thermocouple channels has diode clamps and fuses to divert stray common-mode voltages. A schematic of the thermocouple instrumentation cards (TIC) is shown in Figure 8. Figure 9 shows a photo of the TIC modules mounted in a panel.
Each power circuit is protected with a magnetic circuit breaker (30A) and a high speed fuse (60A). The circuit breaker provides normal overcurrent protection and provides a manual switch. The fuse prevents fast current transients, such as those caused by a short circuit, from fusing the SSR in an ``on'' state.
In addition to three on-oven control computers, both temperature measurement and power switching circuits are divided into independent subsystems. The various circuits in the oven can be separated into independent control zones which follow (or monitor) different temperature schedules. The control system is able to maintain temperature control without half of the heater circuits or without half of the temperature data.
When the on-oven control computers detect a control error, a message is sent to the operators console. The oven pilot acknowledges the error message by entering his/her initials and proceeds to take the appropriate corrective action.
The minute-by-minute temperature and power readings are stored on the disk in the control room for offline analysis. Data handling and display facilities in IRAF allow the operator to analyze the performance of the oven with contour maps, plots, time averages etc.. The data are archived to tape for post- casting analysis. Subtle problems in the control system are often noticed as small wiggles in the temperature history plots on the screen.
The worst case fault is a heater that will not turn off (Stuck-On-Heater). If this occurs the VME computer will signal the power panel and an alarm will sound and a fault LED, corresponding to the particular heater, will light. Each heater has LED indicators for fault, voltage, local and remote current.
In addition to providing redundancy, modular subsystems allow easy replacement for components that fail. Wherever possible, systems are designed to allow repairs while the oven is operating. For example the VME power supplies can be removed without disturbing the other connections to the computer.
To date, three successful 3.5 meter borosilicate honeycomb mirrors have been produced. Expansion of the oven and control system to the full 8 meter capacity is nearly complete. Casting of the 6.5 meter mirror is planned for early 1991. The cost of the final control system is estimated at 1.3 million dollars without including the cost of the diesel generators or refractory costs. Labor accounts for about two-thirds of this amount.