Flight controllers at the National Aeronautics and Space Administration (NASA) Goddard Space Flight Center (GSFC) lost contact with the Solar and Heliospheric Observatory (SOHO) in the early morning hours of June 25, 1998 Eastern Daylight Time. The SOHO mission is a joint European Space Agency (ESA)/NASA mission and is a component of the International Solar Terrestrial Program (ISTP). A SOHO Mission Interruption Joint ESA/NASA Investigation has been established to investigate this mishap. The Board co-chairs are Professor Massimo Trella, ESA Inspector General and Dr. Michael A. Greenfield, NASA Deputy Associate Administrator, Office of Safety and Mission Assurance. This Board has concluded its initial evaluation and interim analysis indicates that the loss of the spacecraft was not related to any on-board failures.

The Board is investigating three errors that appear to have led to the major anomaly experienced by the SOHO spacecraft. The first two errors were contained in preprogrammed command sequences executed from the Ground System, while the last error was a decision to send a command to the spacecraft in response to unexpected telemetry readings.

BACKGROUND: SPACECRAFT CONTROL

The incident was preceded by a routine calibration of the spacecraft's three roll gyroscopes (gyros). The three gyros, all aligned with the spacecraft roll axis, which is normally pointed toward the sun, measure incremental changes in roll attitude (orientation about the axis in space where the spacecraft rolls).

Gyro calibration is necessary in order to accurately measure their drift bias. If the spacecraft had absolutely no angular (rotational) motion, a gyro's output would typically indicate a nearly-constant non-zero angular rate, known as the drift bias. After the bias is accurately calibrated, the bias value is uplinked to the spacecraft computer, which subtracts the bias from the gyro measurements (to determine the actual motion of the spacecraft). Drift bias changes slowly over time and temperature, so it must be determined, or calibrated, well enough and often enough for the attitude control system to meet its pointing requirements.

The gyros are not required during most of the mission. They are used for thruster-based activities, such as momentum management, Emergency Sun Reacquisition (ESR), and Initial Sun Acquisition (ISA).

Momentum management, under the control of the spacecraft's Attitude Control Unit (ACU) computer, is performed approximately every 2 months to maintain the reaction wheel speeds within prescribed limits.

Reaction wheels provide control torques (twisting forces) to control spacecraft attitude (orientation in space). Control torques are needed in order to counteract internal and external disturbance torques experienced by the spacecraft, and to slew the spacecraft for special off-pointing and roll-offset maneuvers.

By far, the dominant external disturbance torque is due to solar radiation pressure. A small offset between the center of pressure of the sun-facing side of the spacecraft and the center of mass of the spacecraft causes the solar pressure to be unbalanced relative to the center of mass, resulting in an external torque on the spacecraft. The solar torque is in a fixed direction and would tend to rotate the spacecraft, but the attitude control system senses minute changes in attitude and commands its reaction wheels to accelerate. The torque exerted by the reaction wheel motors in order to accelerate the wheels is mirrored by an equal and opposite torque exerted on the spacecraft, due to Newton's second law of motion. The wheel torque cancels the solar torque, thus controlling attitude.

Momentum management is necessary because over time, the reaction wheels increase in speed in order to maintain spacecraft attitude in the presence of external disturbance torques. Over time, under the influence of a small but nearly-constant torque command, the wheels accelerate to speeds approaching their design limit, so momentum management is performed periodically to reset the reaction wheel speeds to a nominal value.

In the momentum management mode, the ACU computer controls the attitude of the spacecraft with thrusters while commanding the wheels to decelerate. The attitude disturbance that would be caused by the wheel deceleration is counteracted by firing thrusters.

The ESR is a "Safe Hold Mode" or a "safety net" configuration autonomously entered by the spacecraft in the event of anomalies. ESR is a hardwired, analog control mode that is part of the Fault Detection Electronics (FDE). Unlike the other control modes, it is not operated under the control of the ACU computer. Thrusters are used in ESR to control the spacecraft.

The Initial Sun Acquisition mode is used when the ACU computer fires the spacecraft's thrusters to point the spacecraft toward the Sun under the guidance of an onboard Sun-sensor.

To get to Mission Mode, in which science observations are made, SOHO progresses through a sequence of ACU control modes, starting with ISA. This was done early in the mission, following separation from the launch vehicle. Since that time, ISA is only commanded after any ESR events, in order to recover control of the spacecraft's attitude.

Gyro usage is kept to a minimum because these components are mechanical and subject to wear, and therefore life-limited.

Following gyro calibration, Gyro A is deactivated (despun) in order to conserve its life, while Gyros B and C remain fully active for use during the upcoming momentum management maneuver. Gyro A is configured for use by the ESR electronics and is therefore not needed for the maneuver.

Normally, the gyros perform the following functions:

Gyro A - connected to FDE for roll rate measurements for ESR using thrusters.

Gyro B - connected to FDE for roll anomaly detection

Gyro C - connected to ACU for roll attitude during computer-based control modes using thrusters

During momentum management, which is a computer-based mode, only Gyros B and C are needed: C is used in the control loop, and B is used for fault detection. In the event of an ESR, Gyro A was supposed to be spun up by the ACU. A software function in the spacecraft's central computer re-activates (re-spins) Gyro A in the event of an ESR. Due to an error in a preprogrammed command sequence, the onboard software function that activates the gyro needed by ESR was not enabled.

The fault detection gyro (B) can be used for "Roll Rate Anomaly Detection (RRAD)" or "Roll Attitude Anomaly Detection (RAAD)." During momentum management, the FDE is set to perform RAAD, which includes a gain of 20 being switched into the signal path for increased sensitivity, and a numerical integration of the (now amplified) gyro angular rate signal to obtain an angle signal. Following momentum management, a TSTOL procedure ("pre-programmed command sequence") was executed on the ground system to reconfigure from RAAD to RRAD, which does not include the gain of 20 or the numerical integration, but the procedure contained a logic error which left the gain of 20 in place.

Following the momentum management maneuver, Gyro B (which is used for fault detection), was mistakenly left in the high gain setting resulting in the indicated roll rate being 20 times greater than actual. The incorrect gain resulted in an on-board fault detection triggering an ESR. This ESR, the 5th since launch, occurred at 7:16 PM EDT, June 24, 1998.

During ESR-5, the control Gyro A was not active because of the first error referenced above; however, there is no evidence or belief that this resulted in any anomalous spacecraft behavior. As per design, the ESR event resulted in a reconfiguration of the gyros.

An ESR occurs because something went wrong. The ESR recovery strategy is to resume control of the spacecraft using a hardware configuration which is sufficiently far removed from the configuration in place at ESR entry so as to avoid repeated ESR events due to a single cause. The reconfiguration includes using the redundant computer, ACU B, and using a different gyro for "computer control," specifically, using Gyro A instead of Gyro C. So, following an ESR, the gyro have the following functions:

Gyro A - ESR roll control and ACU roll control

Gyro B - FDE, roll anomaly detection

Gyro C - not used

Gyro A now replaced Gyro C as the roll gyro for thruster-based control modes, while Gyro B remained configured as the fault detection gyro. The error in Gyro B's gain was discovered and corrected.

After transitioning to the Initial Sun Acquisition (ISA) mode as part of the normal ESR recovery sequence, the attitude control system began integrating the gyro drift rate bias associated with the still despun Gyro A.

Gyro A was not spinning and its digital output, which was connected to the ACU, was reading zero. However, the ACU compensates for the gyro drift bias; i.e., it subtracts the most recently uplinked drift bias estimate from the actual gyro reading so as to remove the bias from the signal. In this case, though, the gyro output was not "true rate plus bias" as it would have been if it was spinning, but instead, it was zero. So the drift-compensated rate, as calculated by the ACU, was a constant equal to the negative of the drift bias estimate. Since it was a constant and had no relation to actual spacecraft motion, no amount of roll thruster firing could drive it to zero.

After 15 minutes, this resulted in roll thruster firings intended to null the apparent (but non-existent) roll attitude error. In less than one minute, the roll rate was sufficiently high to trigger again the Gyro B based fault detection, resulting in ESR-6 at 10:35 PM EDT, June 24, 1998. Again, there is no evidence or belief that ESR-6 resulted in worsened spacecraft health and safety, even though the ESR gyro, Gyro A, was not active.

Although the spacecraft remained sun-pointing within nominal limits and was therefore in a power-positive and thermally-safe attitude, the state of the spacecraft was precarious at this point in time. It had an anomalous roll rate and was depending on a deactivated gyro for roll control in both ESR and ISA modes. The personnel on the ground were not aware of either of these facts at that time. Gyro C was out of the picture ever since the reconfiguration at ESR-5. Gyro B was active and on-line for fault detection, and it was correctly measuring the anomalous roll rate. A rapid decision was made that Gyro B was faulty and that it should be commanded off.

During the four previous ESR's that occurred prior to June 24, 1998, the roll rate was always observed to be very small or zero. This was probably due to 1) ESR roll control did a good job of driving the spacecraft roll rate toward zero, and 2) the quantization (digital resolution of an analog signal) was so coarse that very small rates would not be visible in telemetry. Therefore, Gyro A was giving the expected reading following an ESR, and Gyro B was giving an unexpected reading. Gyro B triggered both ESR-5 and ESR-6 and was showing readings that were different from previous ESR experience. This led to the decision to turn off the gyro.

During ESR-6 recovery, Ground Operations commanded the spacecraft to ISA mode. In ISA, the attitude control system resumed firing roll thrusters in an attempt to null the apparent attitude error associated with the despun Gyro A. Gyro B and the associated fault detection were now inactive. The increasing roll rate eventually resulted in pitch (up and down) and yaw (left and right) sun-pointing errors that exceeded a prescribed limit of five degrees, resulting in ESR-7 at 12:38 AM EDT, June 25, 1998. Due to the gyroscopic cross-coupling induced by the anomalous roll rate, the ESR controller was no longer stable, and the spacecraft went out of control

Gyroscopic coupling means (in this case) that when the spacecraft was spinning about its roll axis, this roll momentum causes the spacecraft to behave more like a spinning top than a simple rigid body. If you push a top so as to tip it over away from you, it tips to the left instead.

ESR could not null the roll rate because it had no gyro available since gyro A was not spinning. The ESR pitch and yaw control loops were not designed to control a spinning spacecraft. It fires pitch thrusters in response to pitch errors measured by the sun acquisition sensors and likewise for yaw errors and yaw thrusters. However, due to gyroscopic coupling, which was not a design assumption for ESR, the pitch thrusters result in spacecraft yaw motion and vice versus, as in the example above when a top is pushed away. Under these conditions, the ESR controller could not stabilize the spacecraft, and the sun-pointing errors grew rapidly.

The incorrect diagnosis of a Gyro B fault and the subsequent ground response to this diagnosis ultimately resulted in loss of attitude control, subsequent loss of telemetry, and presumed loss of power and thermal control. Loss of telemetry occurred at 12:43:56 AM EDT, June 25, 1998, and it is not known whether telemetry loss was a direct consequence of loss of power or loss of communication link due to spacecraft attitude.

Preliminary analysis of the SOHO recovery plans show that the plans are based upon reasonable assumptions; however, there are still many unknowns. The state of the spacecraft has been modeled and points to some hope of recovery once some power is restored.

The anomalous spacecraft spin, initially about the roll axis, will have transferred into the axis of maximum inertia, approximately along the spacecraft yaw axis. During this tumbling motion, the angular momentum vector remains inertially fixed approximately along the SOHO sun axis that existed at the time of the anomaly. This orientation results in the solar arrays being nearly edge-on to the sun and thus not generating power.

The orbit of the spacecraft and the seasonal change in the spacecraft-sun alignment should result in the increased solar illumination of the spacecraft solar arrays over the next few months. The power generated will be modulated due to the paddle-wheel rotation of the spacecraft about the yaw axis. This periodic power input may allow intermittent reacquisition of the spacecraft signal. The probability of success improves as the array illumination angle approaches maximum which is currently predicted to occur in late September 1998. The orbit of the spacecraft about the Lagrangian Point (L1) should be stable until late 1998.

In an attempt to recover SOHO as soon as possible, the Flight Operations Team is uplinking commands to the spacecraft via the NASA Jet Propulsion Laboratory (JPL) Deep Space Network (DSN) approximately 12 hours per day with no success to date. A recovery plan is under development by ESA and NASA to provide for orderly restart of the spacecraft and mitigate risks involved. The recovery of the Olympus spacecraft by ESA in 1991 under similar conditions leads to optimism that the SOHO spacecraft may be recoverable once contact is re-established.

In May, 1991, ESA's Olympus telecommunications satellite experienced a similar major anomaly which resulted in the loss of attitude, leading to intermittent power availability. As a direct consequence, there was insufficient power available to maintain adequate communications and internal heating. Temperatures plummeted as low as minus sixty degrees Celsius, and the batteries and fuel froze. From analysis of the data available prior to the loss, there was confidence that the power situation would improve over the coming months. A recovery plan was prepared, supported by laboratory tests, to assess the characteristics of thawing batteries and propellants. Telecommand access was regained four weeks later, and batteries and propellant tanks were thawed out progressively over the next four weeks. The attitude was then fully recovered and the payload switched back on three months after the original incident. Equipment damage was sustained as a result of the low temperatures, but nothing which prevented the successful resumption of the mission. The experience of Olympus is being applied, where possible, to SOHO and increases the hope of recovering also this mission.

-end-