Introduction to NEWSTAR for new and prospective users Contributed by Johan Hamaker, november 1994
NEWSTAR is an acronym for the 'Netherlands East-West Synthesis Telescope Analysis and Reduction'. It stands for a small collection of programs whose purpose is to process 'raw' visibility data from the WSRT into sky maps, source lists and the like that are ready for astronomical interpretation.
Its functionality allows one to fully exploit the strong points of East-West systhesis arrays in general and the WSRT in particular: Wide-field mapping, and high dynamic range both in 'total-power' and polarisation observations. To achieve this, NEWSTAR contains a number of powerful and refined algorithms not known to exist in other systems. Their presence makes NEWSTAR quite overwhelming for the uninitiated. This document is intended to provide a simple introduction of a tutorial type.
The documentation generally refers to the WSRT as the source of observations. NEWSTAR also includes provisions for the processing of data from the Australian Telecope (ATNF) compact array (ATCA), but these are few and simple and we skip them here.
NEWSTAR 's primary maintenance is carried out by NFRA on Sun and HP systems. It is actively supported elsewhere for a DECstation and a DEC Alpha; it has been supported in the past for a Convex and is therefore likely to be readily portable to there. VAX support has been discontinued.
For general operations, such as moving or deleting files, manipulating text files, etc. you must rely on your host system. Inasmuch as NEWSTAR now only runs in Unix environments, a generic knowledge of that environment should be sufficient, but also necessary.
A perceptive user may well notice that NEWSTAR programs are rather careless in their use of terminology. This is a heritage that we are to some extent stuck with: Automatic 'batch' procedures in several on-going large WSRT projects (WENSS and WHISP) rely on the present structure and parameter names of the user interface. We try to clarify ambiguities in the documentation and on-line Help, but the user must be wary that names do not always mean what they seem to suggest.
The basic functionality and data files of NEWSTAR in a Selfcal application.
FIGURE
Rectangles represent the data files, ovals the programs. The numbers refer to the description in the text.
See text for further explanation.
The most common aim of processing visibility data is to obtain a representation of the sources observed, with artefacts from the observing process adequately removed. Such artefacts arise both from the finite sampling of the data (primary and synthesised antenna beams) and from instrumental errors.
If errors were absent, the problem would be 'simply' to deconvolve the known antenna pattern (synthesised beam) from the observed data. This is an ambiguous process: There is an infinite number of possible solutions. Acceptable ones are favoured by the a priori knowledge incorporated in radio-astronomy algorithms; in addition, human judgment is needed to control the reduction process.
Instrumental errors complicate the problem. To first order they may be determined from calibrator sources observed before and/or after the source. This typically yields a dynamic range in the order of one in a few hundred.
Much higher dynamic ranges can only be obtained through self-calibration. This is a generic name for processes in which a source model and a model of the instrumental errors are constructed simultaneously. Self-calibration relies on the fact that the signature of true sources in an observation is quite different from that imposed by instrumental errors. Algorithms have been invented that discriminate between the two quite well. The separation is not perfect, however, and even more than for source modeling alone, human judgment must keep the process on course.
Figure shows the fundamental functions and the primary data files with their most important contents. The connecting arrows show the data flow in the basic reduction process. We enumerate the steps here in order for a typical reduction process. In practice, the complexity of the observed field and/or the quality of the observation may give rise to a wide variety of situations; the same steps or only part of them may have to be combined in a different sequence. If you need really high dynamic range, take your time to become an expert...
The basic reduction steps shown combined in figure are:
With a general picture of the reduction process in mind, we can take a look first at the data files and then at the programs involved.
.SCN files hold all visibility data in NEWSTAR . The basic unit of data is the scan, an array of visibilities for one mosiac subfield, hour angle and frequency channel, with polarisation (XX, XY, ,YX, YY) and interferometer sequence number as its two dimensions. Series of scans that are contiguous in hour angle together form a sector. The organisation of the sectors in a .SCN file will be discussed below.
It is a fundamental tenet of NEWSTAR that visibility data are always stored as received from Westerbork. Corrections are stored separately as such, and only applied to the visibilities when the data are read in by a program. Thus, what changes during the history of a .SCN file is the correction tables, not the visibility data.
An important mechanism for eliminating faulty data points is the use of flags. Since there are several different reasons for rejecting data, 8 different flag types are recognised. A flag byte is associated with each data point. Programs normally reject a data point if any one of these flags is set, but the user has the option ignore certain flag types.
The self-calibration methods in NEWSTAR rely on comparing the observed visibility data with the visibilities of a source model to find the instrumental errors. To economise on the lengthy calculations necessary to generate the model visibilities, the latter may be saved in the .SCN file for reuse.
FIGURE
NEWSTAR's Selfcal error model.
The sums of all phase and gain errors that occur in the path between the source and the correlator differ per telescope and polarisation, i.e. per IF channel. Each IF channel contributes one and the same error to all interferometers in which it is connected. The correlator is assumed in first instance to be error-free; this is almost perfectly true for the WSRT in most observing modes.
See text for further explanation.
Errors, - and consequently corrections for them -, can be broken down in four categories, as shown schematically in figure:
In the somewhat inaccurate terminology of NEWSTAR no dictinction is made between telescope- and IF-based errors; the two are jointly known as telescope errors as opposed to interferometer errors. The distinction between the two types of error is fundamental to the Selfcal methods which are used to separate instrumental errors from the true source visibilities. A basic understanding of the Selfcal paradigm is assumed throughout NEWSTAR .
In addition to the Selfcal constraint, WSRT data are subject to the Redundancy constraint: Baselines of the same length (and the same orientation as they always have for the WSRT) must produce the same visibilities.
.WMP files are the containers for two-dimensional images: sky maps and antenna patterns. Each immage is an array of data points with horizontal and vertical positions l, m as coordinates. It represents a projection of the celestial sphere on a plane tangent in the field centre; the direction of the projection is parallel to the polar axis.
The organisation of the images in a .WMP file will be discussed shortly.
Unlike the .SCN and .WMP files that may contain a collection of sectors or images, the .MDL file harbours only a single source model. The concept of such a model is familiar from CLEAN as a list of point sources with a position and flux for each.
The NEWSTAR source mode is more refined: It may also contain for each source an (elliptical-Gaussian) extent, its 4 Stokes parameters, its spectral index and its intrinsic rotation measure. Traditional CLEAN components are a degenerate case of this more general concept.
We have seen that NEWSTAR data files are collections of data units: Let us first consider the .SCN file, where these units are the sectors. Each such unit is self-describing, but we need additional information to understand its relation to other units, viz. what are its channel number and field number (for a mosaic observation). Moreover it is possible to store more than one observation in a scan file and we need a way of knowing which is which.
The organisation in NEWSTAR data files is by means hierarchical indexing: Each unit is addressed through a composite index, consisting of a string of integers 'indices' separated by dots, e.g.
3.1.0.5
For the .SCN file the indices are:
<group>.<observation>.<field>.<channel>.<sequence\ nr where
For the .WMP file the indices are:
<group>.<field>.<channel>.<polarisation>.<image type>.<sequence nr> where <group>, <field> and <channel> are the same as for the .SCN file; in addition we have
In closing, we note that the indexing structure is purely administrative. The data units are entirely self-contained in terms of their data contents and associated headers. A sector, for example, can be copied from one .SCN file to another without any loss of the information necessary for its interpretation.
Figure includes only the most basic functions of the most important programs. Each of the programs includes a considerable number of other functions: NEWSTAR is a small collection of large multifunctional programs rather than a large collection of small programs (like AIPS or GIPSY).
We will briefly mention the most interesting functions in each of the programs, but ignore the utility functions which are part of most programs. These will, e.g.:
NSCAN is the program that handles .SCN files. Apart from utilities, its only operation is:
It is capable to read WSRT data in all existing formats including the old 'Leiden format'.
NCALIB is the program that determines corrections from the visibilities in a .SCN file, optionally using a source model as a reference. Its primary functions include:
NMAP is the program that handles .WMP files. Its primary functions include:
NFLAG is the only program that changes flag settings. Its FLAG function is composed of a large number of subfunctions that set different flags on the basis of criteria that may be
NFLAG is still evolving. It is hoped that both NFRA and the NEWSTAR users will contribute new ideas and algorithms, to further the art of eliminating faulty data with a minimum of efforts as well as minimal accidental loss of healthy data.
NMODEL is the program that handles source-model (.MDL) files. Its most vital function is FIND, i.e. finding source components of small extent and fitting elliptical Gaussians to them. This operation differs from CLEAN in two ways:
NMODEL also harbours the UPDATE function, which fine-tunes the components of a model by comparing the model visibilities with the observed ones.
NCLEAN is NEWSTAR 's CLEAN program. It's purpose is to create source models for thos source components that cannot be adequately represented by elliptical Gaussian's. It implements only the Högbom, Clark and Cotton-Schwab CLEAN algorithms.
Figure shows only the most vital programs. There are several others:
NEWSTAR programs communicate with the user through a uniform parameter interface. (Historically this subsystem is referred to as DWARF, but this name now serves only to cause confusion.) We briefly mention the features one normally uses in an interactive session.
On-line help in a running program is available in two modes:
Each program run produces a .LOG file in your current directory. These files are named
<prog><timestamp>A.LOG,
where <prog> is the first three characters of the program name and <timestamp> is a string of 12 digits (yymmddhhmmss). The newest log for a program is also known under the name
<program>.LOG
The log file contains a record of all the parameter values that were used in the program run, a copy of all messages and other output that appeared on the terminal and in some cases a lot more information.
In many cases the best way to digest a log file is not to print it. Instead, take a quick look at it on your terminal to decide what information is most relevant; then use the Unix utility grep to extract it.
The possibilities to exchange data between NEWSTAR and 'the rest of the world' are limited to the following:
NEWSTAR 's history dates back to the pioneering work on Redundancy by J.E. Noordam in the early 1980s. W.N. Brouw systematised and expanded his work in subsequent years, in a suite of programs known as the 'R-series'. In 1990-1991, this series served as the prototype for an entirely new collection of programs that eventually became NEWSTAR . During this entire development, A.G.de Bruyn played an indispensable role as an active and stimulating user. Several users contributed to the documentation; they are acknowledged in the individual Document chapters. Incorporated in NEWSTAR are:
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.