SWAS Data Pipeline

1.0 Overview of SWAS Data

While in orbit, SWAS makes continuous spectroscopic measurements. The AOS is read out every 10 milliseconds, regardless of the actual operation mode and pointing of the telescope.  These 10 millisecond scans are co-averaged on-board, generating one full 1,450-channel scan every 2 seconds. All the 2-second co-adds are stored on-board, time-tagged, and downlinked to the ground stations (downlinks require approximately 1 hour per day) twice per day.  The downlinked data are played back to the NASA data facilities twice-daily, and are subsequently passed on to the SWAS Science Operation Center (SOC) at SAO in Cambridge, Mass. In addition to the spectroscopic output which make up most of the SWAS science data, there are also the ``housekeeping'' data which include a large number of instrument and spacecraft status parameters.

The data stream is pre-processed using a modified NASA software program (``tcw'') upon arrival at SOC. This process decompresses, decodes, re-formats and partitions the science data packets into a ``segmental'' form, where data file boundaries are established based on the observational segments, as implemented by the SWAS Planning and Scheduling System (each segment usually observes a single target, and lasts up to 45 minutes).  At this stage, SWAS science data become individual files in FITS (binary table and image extensions) format, and are referred to as ``level-0.5'' (or LH) data in our discussions. Essentially, the LH data are identical to the SWAS instrument output in terms of the science content, with the only exception that the1,450 AOS channels are broken into two halves (corresponding to the 490 and 550GHz bands, respectively). In other words, the pre-processing mostly takes out effects related to the packaging the transmission of the data from the spacecraft to the SOC.

The SWAS Pipeline consists of a series of operations, each of which can be considered as a distinctive step in the data processing. Results of each processing step are recorded into intermediate data files. There is also a data reduction log file automatically generated in the process. The products of the pipeline are written into a new output data file in the final step of the processing. In its entire operation the Pipeline accesses the LH data file on a ``read-only'' basis, i.e. no information in the raw data is altered as a result of the pipeline processing. This ensures the consistency of SWAS data when re-processing becomes necessary.

The output of the pipeline are co-added spectra with a new set of header parameters generated during the processing. Currently, there are two parallel ``flavors'' of the output data : an IRAF-flavored FITS data file format, to suit those users who are familiar with tools within the IRAF environment; and a CLASS-flavored FITS format, which is intended for the general user community who are presumably more accustomed to the popular radio astronomy data analysis tools package CLASS from Observatory de Grenoble, Spain. These data products are referred to as ``level-1'' data.
 

2.0 Summary of Processing Steps

There are three basic SWAS science observing modes: nodding, chopping and mapping. The SWAS Pipeline reads the header parameters in the LH data file that identifies the observing mode, and takes actions accordingly.  The following descriptions summarize each of the main Pipeline reduction steps. Special reductions for data of other modes (e.g. planetary cal and matrix tests) are discussed elsewhere.
 

2.1 Step one (checking):

This task performs data validation and integrity checks on the raw (LH) data file. The validations are based on the ``header'' parameters that come with the spectral data and specify the instrument setup and conditions of the data taking. Potential problems such as internal inconsistencies of the instrumental parameters and observing time gaps are reported and,  flagged. Input data files that are not of the three basic observing modes are noted and excluded from the following normal processing steps. In addition, necessary conversions of parameters into more useful units (such as times, frequencies, temperatures and celestial coordinates) are performed. However, there is no direct access or manipulation of the spectral (pixel) data at this stage.
 

2.2 Step two (ON/OFF pairing and subtraction):

This step mainly performs the standard (ON-OFF)/(OFF-ZERO) calculations of the spectral data. This involves pairing and evaluating ON, OFF and ZERO or CAL scans. The scans are pairied on a one-to-one basis for Nod and Chop modes, usually with the ON and OFF scans taken within 30 seconds in time. For mapping data where unequal numbers of ON and OFF scans are sometimes taken, a procedure similar to the so-called ``on-the-fly'' mapping processing is adopted, where each ON is paired with an average of a small number of OFF scans. Because of the Doppler effects of the orbiting spacecraft, stringent limits are put on the time intervals over which the pairing can be accepted.

Similar operations are performed on the spectral data for internal CAL sequences, where the ONs correspond to scans of the ``hot load'', while OFFs are those of the blank sky. Results of the CAL data are used in latter steps for passband and flux calibrations.

SWAS ZERO scans correspond to data taken while the blanking switch is engaged. However, since it has been discovered that engaging the blanking switch causes a time-dependent hysteresis effect on the AOS continuum levels, SWAS has been using ZERO scans taken at irregular time intervals.  Tests have shown that the ZERO scans are statistically stable over a long (months) time scale.

In addition to the spectral AOS data, there are also the individual data points from the two SWAS continuum detectors, which are processed in the same way in this step.

Frequency calibration is performed via an on-board COMB generator.  The pipeline selects unsaturated COMB lines, performs a line center fitting for each, and calculates a dispersion relation based on the results. As a rule of thumb, both CAL and COMB scans are taken during the same observing segment as the science data to which the calibrations are applied.
 

2.3 Step three (flagging):

This task allows an adjustable control over the criteria by which individual ON-OFF scans are allowed to enter the final coadds. It is kept flexible so that flagging criteria can be added or changed depending on the actual requirements and types of observation. Basically, scans that exhibit unusual levels of noise, data spikes, and baseline variations are flagged. The statistics of these flagging operations are kept in log files.

This step differs from the flagging in step one (above) in that the criteria imposed here are based on ``derived'' quantities from after the ON-OFF pairing and subtraction step.
 

2.4 Step four (frequency calibration and co-adding):

This step performs frequency shifts and coadds based on the nature of the data and requirements. Nominal co-adds consist of data within a 3-minute (or less) on-target time interval. For each 2-second scan that enters the co-add, a velocity shift is evaluated and added to compensate for the Doppler effect of the spacecraft's motion around the Earth. This effectively normalizes the velocity scale for all scans taken at different positions of the orbit to the line-of-sight velocity of the center of the Earth. Also, this shift and co-add operation is performed on each spectral scan in two opposite directions, because the Doppler effect is in reversed directions depending on whether one is interested in the upper or lower sidebands of the SWAS receivers. It is at this step that the SWAS data are split into four different frequency ranges.

The absolute frequency calibration is based on the fits to the COMB scans taken during the same segment as the science data. If more than one set of the COMB scans are taken, then their results are averaged.  Finally, the VLSR velocity scales for each line are added based on the time and target coordinates of the observations.

For each coadd, several header parameters such as observation time are also summed or averaged as appropriate. For mapping mode data, the coadds are grouped according to their actual pointing offsets.
 

2.5 Step five (Flux Calibration and System Temperature Evaluation):

In standard calibration sequences, SWAS takes CAL data with its hot load at a known temperature, and compares that with signal from the empty sky. This calibration step extracts this information, evaluates the gain profile of the SWAS passband and the flux conversion factor, and apply these to the data for flux calibration.  The continuum detector data are also used to provide the ``Y-factor'' measurement of the receiver and, thus, the corresponding system temperature. Note that there are only two double-sidband system temperatures derived for the four SWAS bands, i.e the upper and lower sidebands of the same frequency range (490 or 550 GHz) share the same system temperature. The cosmic background with a blackbody temperature of a constant 2.97K in the SWAS continuum bands is assumed in these calculations.

The Pipeline generates both IRAF and CLASS flavored data output at this point which are identical except for some minor differences in parameter settings. However, in the CLASS flavored data files, a total co-add spectrum per band, per pointing position (in case of mapping data), per segment is also provided in addition to the nominal (3-minute) averaged spectra.  Note that each output spectrum has its own set of header parameters that specify, among other things, the integration time, antenna temperature and frequency scales, target coordinates, time of observation and data reduction, etc.

At this stage (known as Level-1), the data products are still organized according to observing segments, which is also chronologically ordered. One of the last operations of the SWAS Pipeline is to sort the data files (one per segment) according to the target being observed. For example, most of Orion data files are copied to a data directory ``omc-1'' (but see notes about different LO settings below).
 

3.0 Additional Notes

The ``integration time'' recorded in the SWAS data product is only the sum of ON scan times. The actual observing time, which include the OFF scans, is of course usually twice as long.

The amount of data flagged and excluded from co-adding in the Pipeline processing has been monitored throughout the SWAS mission. There are a variety of reasons for flagging the data, but the ultimate goal is to obtain the highest possible signal-to-noise ratios from the existing data without introducing any artifacts.  The total amount of excluded data has been shown to be only a small fraction of the available observing time. The overall stability of the SWAS instrument has been found to be very good, as evidenced by spectra of the same target taken over 6-month intervals that are essentially indistinguishable.

By a convention established in the SWAS mission, each unique ``target name'' has a single (0,0) sky coordinate position. Mapping positions of the same target are expressed as angular offsets from that central position.

In some cases where a target has been observed with several different local oscillator (LO) frequency settings, the data are separated using different target names. For example, target names ``sgra'' and ``sgra-150'' are of the same target, but of different LO settings (i.e. sgra-150 had the LO shifted by -150 km/s relative to the LO setting used for sgra).  One could, if necessary, combine these data in further data reductions, for example, within the CLASS package.

The co-added spectra are each of 725 channels, with pixel-to-pixel velocity scales of approximately 0.6 km/sec. However, because of the shift-and-coadd operation in the calibration process as well as the ``edge-effect'' of the AOS bands, the approximately 25 pixels at each end of a co-added spectrum suffer from much higher uncertainties and cannot be taken at face value for scientific analysis purposes. Also, since the shift-and-coadd operation was performed on integer number of pixel basis, (i.e. no sub-pixel interpolation is used), the effective velocity resolution of the co-added SWAS spectra is about 1.5 pixels, or approximately 0.9 km/sec.

In the header information of the SWAS Pipeline data product, one of the items that are unique to SWAS (as opposed to ground-based radio telescopes) is the ``roll angle'' of the spacecraft. This parameter can be important in some cases, especially because the SWAS beam is not perfectly circular. Since this is not a standard FITS spectral header item and thus could be missed by the analysis tools a user might have, we have decided to use the parameter ``azimuth angle'' (which is meaningless in the context of SWAS) to register the value of roll angle.  Similarly, in the ASCII string of FITS header parameter ``telescope'' we record information regarding the observing segment. For example, ``s506006 NOD'' means that the data were obtained on Day of Mission number 506, of observing segment 6, and was taken in nodding mode.
 

Data Segments with Illegal System Temperatures

For ground support events, SWAS engages a blanking switch. Because no data are recorded from the AOS when the blanking switch is engaged, ground supports appears as time gaps (a jump in the time index) in the science data.  The use of the blanking switch has been found to induce a hysteresis effect in the AOS after it is disengaged, with the response of the AOS returning to normal levels in about 100 seconds. If a ground support event occurs entirely within a science segment, the pipeline software detects the event and automatically disregards the next 50 scans (100 seconds of data) after the event for processing.

Under certain (rare) circumstances, the engagement of the blanking switch will occur such that the calibration scans for a science segment are subsequently disregarded by the pipeline after the blanking switch is disengaged.  A bug in the pipeline software allows these segments to be processed to level-one in an uncalibrated state and assigns system temperature values of "9999" to channels one and two.

Segments in the 1st and 2nd public SWAS data releases that are affected by the illegal system temperature problem are listed below.  These segments have been deleted from the post-pipeline data products at the SWAS Science Operations Center and are not used for scientific analysis.  In the Class-format archive, the affected science segments are identified in the "all.dat" file by "mission day_segment number" for the targets listed below.  For the third public data release (August 2001), both the Class-format archives and the FITS-format archives have had the affected segments removed.

segment ID           target

0241_032            ON2S
0052_036            S140
0075_030            S140
0265_064           HELIX
0099_062           OMC-1
0117_012           OMC-1
0320_021           OMC-1
0320_056           OMC-1
0320_061           OMC-1
0396_004           TMC-1
0310_016          SATURN
0239_042         CRL2591
0306_027         JUPITER
0112_053         NGC1333
0113_048         NGC1333
0059_043         NGC7538
0067_046         NGC7538
0070_046         NGC7538
0384_010        IC443B-60
0385_013        IC443B-60
0386_011        IC443B-60
 

Reprocessing of Data Taken in Chop_Nod Observing Mode

Early in the SWAS mission, it was discovered that observations made using the chopping secondary (chop mode) have a standing wave in the spectral baseline. This standing wave is likely due to the presence of a reflection in the chopped off-beam somewhere in the telescope.  For this reason, the vast majority of SWAS observations are made by nodding the spacecraft itself to the off position (nod mode).

A hybrid observing mode called "chop_nod" (dual beam switching or balanced beam switching) was introduced on mission day 348. For this mode to be effective, the pipeline software was modified to apply the Doppler correction to the AOS data after the on-minus-off pairs are averaged.  This observing mode, processed in this way, removes the standing wave from the spectral baseline. Data taken in the chop_nod mode that were previously distributed in the second SWAS public data release have been reprocessed with the modified pipeline software and have been updated for the third SWAS public data release (August 2001).  Observing segments affected by this reprocessing, listed as "mission day_segment number" and target are listed below.

segment ID         target

0348_041         VY_CMA
0369_002         JUPITER
0369_006         JUPITER
0369_010         JUPITER
0369_014         JUPITER
0369_018         JUPITER
0369_022         JUPITER
0369_026         JUPITER