Yohkoh/SXT raw & calibrated dataset prepared by the YLA

last modified 21-Apr-2015, by Aki Takeda and Loren Acton, Montana State University.

[0] Contents (links to the sections) and references

[1] Data classification.
[2] Directory structures.
[3] Data details and file name conventions   ( Level_0, Level_1, Level_2, Level_3).
[4] Sample images and movies.
[5] FITS files header definition.

The following papers will help to understand more about the project.

[1] Data classification

The YLA basic dataset consists of four levels of products from the Soft X-ray Telescope(SXT) on board Yohkoh satellite (1991-2001). Here are brief description (more details are given in section [3]):

The total number of our basic products (Level_1, 2, and 3) are summarized here. The Level 1 and Level_2 data products are in instrument units (DN/sec/pixel). Conversion to physical units requires knowledge or assumption about the solar differential emission measure and the use of software (such as sxt_teem.pro in SolarSoft). The Level 3 images are scaled so that the solar structures are best viewed and the scaling varies with the Sun's activity level. Therefore, the Level 3 images best serve for browsing, and not suitable for quantitative analysis.

Except Level 0 (raw) products, the YLA products are available in both Yohkoh-specific (XDA) and FITS formats. For quick-look (or web-browsing) purposes, Level_3 images are also available in JPEG format.

Yohkoh/SXT has two observing modes with respect to the field of view; full-sun v.s. partial-sun.

Due to the failure of the SXT entrance filter, the Aspect Sensor (co-axially mounted visible-light telescope) images became unavailable after 14-Nov-1992. White-light images are available in Level_1 and Level_3 data. For the latter, they are included in the "wl" directory. There are no Level_2 products for the white-light images.

[2] Directory structure at SDAC and typical file names

Here is the link to the data directory of the YLA basic data products at SDAC. Included in the following directories are YLA products as of July, 2013. 

sxt_fits/ ----- level_1/ ------ full_sun/ --- YYYY/MM/DD/sxtf_YYYYMMDD_HHMMSS_XXX.fts
            |              |--- part_sun/ --- YYYY/MM/DD/sxtp_YYYYMMDD_HHMMSS_XXX.fts
            |-- level_2/ -------------------- YYYY/MM/DD/sx2c_YYYYMMDD_HHMMSS_XXX.fts
            |                                      or    sx3c_YYYYMMDD_HHMMSS_XXX.fts
            |-- level_3/ ----- wl/ ---------- YYYY/MM/DD/sa08_YYYYMMDD_HHMMSS_XXX.fts  
                           |                       or    sa12_YYYYMMDD_HHMMSS_XXX.fts
                           |-- xray/ -------- YYYY/MM/DD/sx2m_YYYYMMDD_HHMMSS_XXX.fts
                                                   or    sx3m_YYYYMMDD_HHMMSS_XXX.fts

sxt_jpeg/  ---- level_3/ ----- wl/ ---------- YYYY/MM/DD/YYYYMMDD_HHMMSS_sa08_XXX.jpg
                           |                       or    YYYYMMDD_HHMMSS_sa12_XXX.jpg
                           |-- xray/ -------- YYYY/MM/DD/YYYYMMDD_HHMMSS_sx2m_XXX.jpg
                                                   or    YYYYMMDD_HHMMSS_sxem_XXX.jpg

sxt_xda/  ----- level_0/ -------------------- YYYY_WW/sfrYYMMDD.HHMM (1 file per orbit)
            |                                    or   sprYYMMDD.HHMM (1 file per orbit)
            |-- level_1/ ------ full_sun/ --- YYYY/sfrYYMMDD_HHMM (1 file per day)
            |              |--- part_sun/ --- YYYY/sprYYMMDD.HHMM (1 file per day)
            |-- level_2/ -------------------- YYYY/sscYYYYMMDD_HHMM_BBBBBBB.N (1 file per day)
            |                              or  sstYYYYMMDD_HHMM_BBBBBBB.N     (1 file per day)
            |                              or  sscYYYYMMDD_HHMM_BBBBBBB.N_unc (1 file per day)
            |                              or  sstYYYYMMDD_HHMM_BBBBBBB.N_unc (1 file per day)
            |-- level_3/ ----- wl/ ---------- YYYY/MM/DD/sa08_YYYYMMDD_HHMMSS_XXX.xda  
                           |                       or    sx12_YYYYMMDD_HHMMSS_XXX.xda
                           |-- xray/ -------- YYYY/MM/DD/sx2m_YYYYMMDD_HHMMSS_XXX.xda
                                                   or    sx3m_YYYYMMDD_HHMMSS_XXX.xda

[3] Data details and file name conventions

Level_0 data

They are raw data files (telemetry data after reformat, i.e., separated from data from other instruments and time-ordered in UT, universal time), additionally refined by YLA by eliminating corrupted or duplicated records. They include all calibration images, such as dark frames and diffuser images. In partial frame images(PFIs), a large field of view (FOV) with 128-pixel height or larger, are taken as a series of 64-pixel height image strips, and restored as separate frames. The level_0 data are in the Yohkoh specific format (called XDA), whose byte layout is defined in the Solar-A File Format Control Document. The XDA files are readable with a SolarSoft procedure. The level_0 data files are created for each satellite orbit, and organized in the week directories. 
Examples of the file name: sfr920720.0517, spr920720.0517.
This file is included in the week directory, 1992_30/, because the date of 
observation belongs to the 30th week of 1992.

      sfr : prefix for SXT Full Frame (full-sun) Images.  
      spr : prefix for SXT Partial Frame (partial-sun) Images
      other prefixes included in the same directory ;
          hda : HXT data
          bda : BCS data
          wda : WBS data
          ada, cba : for calibration purpose. 
      920720.0517 : date and approximate time (yymmdd.hhmm) at which 
                    the satellite started observation in the orbit.

Level_1 data

The Level_1 data include full frame images (FFI, full-sun images) and partial frame images (PFI, partial-sun images) in floating point format. They include every usable solar image, even if some of the frame may be missing due to the telemetry dropout. All calibration images (dark, diffuser, etc.) and detected problematic frames are eliminated. Basic calibration such as dark and leak(due to filter failure) corrections is done. In PFIs, observing regions with 128-pixel height or larger are assembled to show the whole region of interest in one frame, while in level_0 data, PFIs are stored as a series of 64-pix height image strips. To obtain better photometry in the optical filter, white-light images were often recorded in two separate files in raw data; a file with full intensity compressed to 8-bit, and a file contains only low 8 bit of the obtained intensity. In Level_1 data, the 'low-8bit images' are restored to their compressed counterpart (as long as available) to recover full intensity depth of 12 bit.

Level_2 data

SXT composite images combine 2(long/short) or 3(long/medium/short) exposures taken very close in time to increase the overall dynamic range of the image. The pixels with detector saturation are replaced with those of shorter exposure image(s). Level_2 images are thus most suitable for quantitative analysis. Each composite image has a related uncertainty image which provides the statistical error for each pixel. The Level 2 data only use complete or nearly-complete images. The Level 2 images are not co-aligned, but may be readily co-aligned with existing SSW mapping tools.

Level_3 data

The SXT Level_3 data, or "movie" images are prepared from Level_2 composite full-frame-images for convenience in viewing in movie mode to facilitate the study of coronal evolution and dynamic phenomena. This processing included co-alignment, rebinning 256x256 images to 512x512, logarithmic and byte scaling, adjustment of effective exposure to minimize movie flicker and enhance coronal intensity around solar minimum, and deletion of images unduly contaminated by radiation-belt particles or stray light. As a result, Level_3 SXT images sacrifice photometric accuracy for improved time-lapse viewing. Therefore, they are not intended for quantitative analysis.

The co-aligned full-sun white-light(WL) images are also categorized as Level_3 images. They are obtained with the SXT Aspect Sensor, the optical telescope coaxial with the SXT. It was in operation until 13-Nov-1992 when SXT had the entrance filter failure. The data have 12-bit or 8-bit intensity depth and archived in floating point. The images are co-aligned and have been cleaned to eliminate obvious hot pixels. The WL images are not photometrically useful, because the Aspect Sensor exponentially lost sensitivity after launch and because radiation damage to the CCD from excessive x-ray exposure left dark artifacts on the WL images.

[4] Sample images and movies

Here are sample images of the YLA basic dataset. All the FITS and JPEG files are created per image, while XDA files are created per satellite orbit (for Level_0 products), per day (for Level_1 and Level_2) and per image (for Level_3).

data leveloriginal
data format		file name
Level_0XDA sfr920720.0517
XDA sfr920720.1954
XDA spr920720.0517 (selected 60 images)
XDA spr920720.1324 (selected 60 images)
Level_1XDA sfr920720 (selected 100 images)
XDA spr920720 (selected 150 images)
XDA spr930428_large
FITS sxtf_19920720_141055_131.fts
FITS sxtp_19920720_141047_021.fts
Level_2XDA ssc19930720_0014_1110110.4
XDA ssc19930720_0014_1110110.4_unc
XDA sst19930720_0014_1110110.4
XDA sst19930720_0014_1110110.4_unc
FITS sx2c_19930720_214131_231.fts
FITS sx2cu_19930720_214131_231.fts
FITS sx3c_19930720_035753_121.fts
FITS sx3cu_19930720_035753_121.fts
Level_3XDA
FITS
JPEG		 sx2m_19970720_201339_231.xda
sx2m_19970720_201339_231.fts
19970720_201339_sx2m_231.jpg
XDA
FITS
JPEG		 sx3m_19970720_215723_121.xda
sx3m_19970720_215723_121.fts
19970720_215723_sx3m_121.jpg
XDA
FITS
JPEG		 sa08_19920120_025432_112.xda
sa08_19920120_025432_112.fts
19920120_025432_sa08_112.jpg
XDA
FITS
JPEG		 sa12_19920120_025224_112.xda
sa12_19920120_025224_112.fts
19920120_025224_sa12_112.jpg

[5] FITS files header definition

FITS headers of the YLA products consist of the following groups of tags (a,b,c & d):

(a) tags for the mandatory keywords of the FITS standard file :
e.g., SIMPLE, BITPIX,,,
c.f., Section 5.4.1 of http://archive.stsci.edu/fits/fits_standard/fits_standard.html .
(b) flattened version of the nested IDL index structre designed for the Yohkoh original (XDA) data format :
e.g., DP_MODE, DP_RATE, CCD_TEMP,,,.
c.f., the ".inc" files in a SolarSoft directory ($SSW/yohkoh/gen/idl/ref_access) or here.
(c) additional tags required to be handled with SolarSoft on a common platform with the other data sorces :
e.g., CRPIXn, XCEN, YCEN,,,
c.f., http://www.lmsal.com/solarsoft/ssw_standards.html.
(d) a comment tag describing the calibration history process by YLA :
i.e., HISTORY,
c.f., sample description of the HISTORY tag .

The tags in group (b) are often incomplete, because the original XDA index structures generally have vector values while the FITS tags only allow scalar values. To avoid information from lost by truncation, the vector components of some important tags are specifically labeled (e.g., SHAPE_Cn, CORNE_Cn) in the FITS header as the needs arises. Most of the group-(b) tags are expressed in binary in accordance with the original XDA index. Their decimal values are derived through a series of IDL/SolarSoft functions (e.g., gt_date.pro, gt_temp_ccd.pro), in the same way as they worked with XDA indexes.

Here is an example of the typical FITS headers of the YLA products (Level_2 data).