The reformatter runs permanently on gavroche. It's fed with telemetry packets and builds FITS files out of them. Even though it has many fallbacks, the telemetry is sometimes so messed up that it cannot make sense of it and crashes. It also have difficulties to handle some splecial kind of data, like very high cadence sequences.
If the reformatter crashes, the planner must restart it. The status of the reformatter can be tested by checking all the batch jobs on gavroche. At the prompt of the reformatting window (open a new one if it's not already open), type
$ sh[ow] sys[tem] /b[atch]
where characters in brackets are optional. The output should be something like this:
OpenVMS V7.2-1 on node GAVROC 4-DEC-2001 22:33:19.87 Uptime 34 21:51:44 Pid Process Name State Pri I/O CPU Page flts Pages 00000432 QKL scan LEF 6 1998713 0 00:21:45.82 443720 92 B 00000455 RT time diff LEF 6 27316167 0 04:05:25.17 2238849 71 B 00017CAB RT reformatter LEF 6 2294987 0 00:23:27.12 1482 1141 B 000004F5 response plots LEF 6 1103810 0 01:30:38.45 379870 138 B 00000530 Movie archive LEF 6 677309 0 00:01:32.34 2668 69 B
Another way of doing it is:
$ shq REALTIME$BATCH
And the output should now be:
Batch queue REALTIME$BATCH, available, on GAVROC::
Entry Jobname Username Status
----- ------- -------- ------
24 REFORMAT_BATCH EIT Executing
If the reformatter is not running (it dosen't show up in the list of processes), restart it by first finding a raw telemetry record number just before the crash by displaying the last line of the reformatting log file:
$ tail /46 reformat_batch.log
Then search in the file for the last occurence of lines similar to these:
**** Image data found at record 3063944 ****
Exposure start time: 2001/12/04 20:48:10.595
Readout port: B, Readout mode: backside
Note that you may need to display more than 46 lines to find a valid record number. Note the record number and then restart the reformatter typing:
$ subm[it] /queue = realtime$batch [eit.reform.sci]reformat_batch /par = (record number, -1)
where record number is the valid record number you just found in the reformat_batch.log file. Tip: hit the F19 key. It will type for you all of the above except the record numbers in the parentheses at the end.
In case the reformatter systematically dies on one particular image (i.e. for some reason it dosen't want to reformat it), you may have to manually skip this image. To do this, you have to restart the reformatter with a record number that is not logged in the reformat_batch.log file. Check the ACQ window, and look for the record number (7 digits) just following the corrupted record. If the corrupted record dosen't show up anymore in the ACQ window, try to restart the reformatter with increasing record numbers, starting with the corrupted one, and adding 100 each time. You should eventually find a valid record number this way.
In case the reformatter skips an image, or if you restarted it with a incorrect record number (see above), you will have to stop it before you can restart it with a new record number. First, identify the entry number of the reformatter job. It's the number in the Entry field of the output of "shq REALTIME$BATCH". You can then stop the reformatter with:
$ deln /entry=entry number
In the example of the previous section, the entry number was 24. In this case you would have stoped the reformatter with
$ deln /entry=24
Do an shq again to check that the reformatter indeed stopped running. You can then restart it using the procedure describe in the previous section.
Last revised: Tuesday, December 25, 2001 2:26 PM - F. Auchère