Visibilities and associated data: The .SCN-file
Contributed by Johan Hamaker, November 1994
.SCN files hold all visibility data in NEWSTAR . The basic unit of data is the scan, an array of visibilities for one mosiac subfield, hour angle and frequency channel, with polarisation (XX, XY, ,YX, YY) and interferometer sequence number as its two dimensions. Series of scans that are contiguous in hour angle together form a sector. The organisation of the sectors in a .SCN file will be discussed below.
It is a fundamental tenet of NEWSTAR that visibility data are always stored as received from Westerbork. Corrections are stored separately as such, and only applied to the visibilities when the data are read in by a program. Thus, what changes during the history of a .SCN file is the correction tables, not the visibility data.
An important mechanism for eliminating faulty data points is the use of flags. Since there are several different reasons for rejecting data, 8 different flag types are recognised. A flag byte is associated with each data point. Programs normally reject a data point if any one of these flags is set, but the user has the option ignore certain flag types.
The self-calibration methods in NEWSTAR rely on comparing the observed visibility data with the visibilities of a source model to find the instrumental errors. To economise on the lengthy calculations necessary to generate the model visibilities, the latter may be saved in the .SCN file for reuse.
The scan and sector structures in a .SCN-file with the indexing hierarchy in which they are embedded.
FIGURE
See text for further explanation.
Visibilities in a synthesis observation are functions of a series of coordinate parameters. One may conceptually arrange them in a hypercube whose coordinate axes correspond to these parameters; alternatively, this hypercube may be visualised as a hierarchy in which each level corresponds to one of the parameters.
The organisational model used in the .SCN file is a hyprid, in which the lowest levels form a compact array, which is embedded in a hierarchical index structure for the remaining levels. The order of the parameters in this arrangement is in principle quite arbitratry. NEWSTAR 's choice was motivated by considerations of efficiency in both storage space and sorting for the most important processes. The order is, from the bottom upwards:
It is customary to loosely refer to the levels of this hierarchy as the dimensions of a six-dimensional data (hyper)cube. One should bear in mind, however, that the same 'dimension' may be of different magnitude in different 'sub-cubes' of this hypercube or, in other words, that the 'cube' is not regularly filled.
An overview of the .SCN-file hierarchy is shown in figure. The levels correspond to those discussed above. We shall discuss them from the bottom up.
It is worth noting that the scan and sector headers combined contain all the information necessary to process the data. A sector may be divorced from the .SCN file without impairing its ability to be processed. The index blocks in the levels on top contain no 'scientific' information. Their only purpose is to organise the sectors.
The scan is the fundamental agggregate of data in a .SCN file. It is a two-dimensional array of visibilities for a single integration interval, with polarisation and interferometer as its dimensions.
Associated with this block of data is a scan header. It contains descriptive data such as frequency and position parameters. In addition, it holds tables of those gain and phase corrections that may change from scan to scan, as well as data and correction statistics that are a measure of the data's quality. The contents are listed in the definion file.
A sequence of scans that is contiguous in hour angle is grouped together in a sector. The sector's contents are described in a sector header. Like the scan header, it contains descriptive data, corrections parameters that may be assumed to be constant for the time period covered by the sector and data/correction statistics for all the sector's scans combined. The contents are listed in the definion file.
The index structure in which the sectors are organised follows the levels listed above. The <sequence number> is an extra index that allows one to have more than one sector for which the first four indices are identical. A complete .SCN-sector designation reads:
Sectors in the uv plane.
FIGURE
Left: For a standard observation the scans for a single observation and channel are contiguous and can be lumped together in a single sector.
Right: In a mosaic observation the cuts for a given field are interleaved with cuts for other fields, so each cut is stored in a sector of its own.
The sector numbers shown are just by way of example.
One particularly important application of this possibility is in the organisation of the non-contiguous scans in a mosaic observation. In such an observation, the scans are not contiguous and therefore each 'hour-angle cut' forms a sector of its own (figure).
The allocation of the sector indices for a .SCN file.
FIGURE
- Every run of the NSCAN LOAD function creates a new group.
- Every (tape of optical-disk) label specified as input for such a run is stored in a new observation.
- The field and channel numbers are copied from the input.
- For a standard observation, the sequence number is 0. For a mosaic observation, the successive cuts for the same field and channel are given successive sequence numbers.
The method by which NSCAN allocates sector numbers is schematically shown in figure. The only control the user has is over the allocation of groups: Every NSCAN.load operation creates a new group, and the user defines which WSRT labels will be stored in that group.
The sector header contains parametric data tha are constant throughout the sector. The most important groups are the following:
The scan header contains the parametric data that apply specifically to one scan:
Displays in tabular form:
Graphic displays:
Additional possibilities for extracting. manipulating and displaying a variety of items from the .SCN file are available in NGCALC.
Almost every value (observation parameters, corrections, etc) in the SCN-file headers may be edited manually through NSCAN SHOW EDIT. The effect of changing a value may range from trivial to catastrophic. If you run into a situation that you think can only be remedied by manual editing, you do better to contact the NEWSTAR group first.
You may want to reorganise the indexing of your sectors. This can be done to some extent through the NSCAN REGROUP function.
The most important reason why one would want to delete sectors is to reclaim disk space. It would be fairly simple to provide a DELETE command that would make sectors invisible, but this would not free any space.
An equivalent result can be achieved, however: Use NCOPY to copy those sectors that you want to keep to a new .SCN file, then delete the old file.
.
Operations on corrections generally fall into either of two categories:
Dipole errors:
Corrections are selectively applied to the visibilities whenever the data is read into memory to be processed. By default, all available corrections are selected. The user has the option to make his own selection. To this end, start your program with the /ASK switch:
(We acknowledge that the appearance of many irrelevant prompts in this situation is unsatisfactory. We hope to correct this situation in the future.)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.