The Program NFLAG
Contributed by Johan Hamaker, October 1995.
Revision of original docyment by Jan Noordam, August 1993.
This is a preliminary version of a new document replacing the old version of August 1993 by J.E. Noordam. The information in it is thought to be accurate but known to be incomplete. It should, however, in its present from help the user to get to grips with NFLAG's overwhelming user interface and functionality.
The basic function of NFLAG is to mark visibility points in a .SCN file as faulty by 'flagging' them, i.e. by setting bits in the 'flag mask' associated with each point. In general, visibilities thus flagged will be considered no longer to exist; in this sense flagging is equivalent to deletion, but unlike deletion it is reversible, i.e. flag bits can be cleared at a later time. It is also possible to instruct processing programs to ignore flag bits and treat flagged visibilities as valid.
A secondary function in NFLAG, not further described here, is the SHOW option through which one may explore the contents of a .SCN file. For a description, see the descriptions of NSCAN and its parameter interface.
The simplest way to get rid of bad visibilities is by deleting them. Typically this is done by replacing each bad value by a special value that processing programs recognise as an instruction to ignore the data point. Since its original value is overwritten, a deleted data point is irreversibly lost.
The first step toward refinement is to flag visibilities as bad without overwriting them; processing programs ignore those data for which the flag is set. Since the original value is retained, is can be 'undeleted' by clearing the flag; flagging is therefore also known as 'reversible deletion'. This method is a well-known one, because it is implemented AIPS.
In NEWSTAR , it is carried a step further by differentiating between a number of different reasons for which a visibility may be considered bad. Rather than a single flag, NEWSTAR associates a complete flag byte with each visibility point: Thus one has not a single bit to play with, but eight different ones. The advantage of this approach is that different flagging operations can be treated independently of one another. This makes it possible in many cases to backtrack form a flagging operation that produced undesirable results and thus introduces an important margin of safety: For instance, a selection made in a laborious manual flagging session cannot be accidentally undone by a subsequent automatic clipping operation going wrong.
Five of the eight bits in the flag byte are used by NFLAG, the user may use the remaining three in whichever way he wishes. By default, programs bypass all visibilities for which any one of the eight flags is set; it is also possible, however, to instruct a program to ignore certain flags. The three user-definable flag-types can be used for experiments with different selections of uv-data. (This is analogous to the use of different flag tables in AIPS.)
Below is a list of the flags defined by NEWSTAR . The numbers shown in parentheses are the bit number (LSB=0) and hexadecimal values for these flags:
The NEWSTAR visibility flagging scheme gives the user unprecedented flexibility for reversible data-editing, although its very power makes it a little more difficult to understand and use than a simple scheme such as AIPS's.
The only program that actiually sets and clears flags is NFLAG. It contains a number of options to select bad data on a varied range of criteria and set flags accordingly. To verify the effect, it also offers the option to count flags over various cross-sections or projections of the data hypercube and tabulate the results.
Another option is to select flags on a display of some two-dimensinal cross section through the hypercube, using the program NGIDS. NGIDS then produces a file with flagging commands which may subsequently be read and executedy NFLAG.
On a micro-level, it is possible to inspect the individual flag bytes in individual scans through the SHOW option.
In addition to the flag bytes associated with the individual visibilities, the scan header also contains a flag byte. Through the flags in this byte one may invalidate the scan as a whole. This is obviously a more expedient way to handle flagging, since a single computer instruction will flag or test the entire scan.
In the present implementation of NFLAG, visibility flagging and scan-header flagging operate independently. That is, when the header is flagged the corresponding flags on the individual visibilities are left unchanged, and if the header is later unflagged those flags come back into force. On one hand this introduces some additional independence between different flagging operations as discussed above. On the other hand it forces upon the user a technical distinction between two types of flagging that are functionally equivalent: The user must contend with such flagging operations as HASCANS, CLHEAD, TOHEAD, TODATA, statistics specifiers such as SCANS. (Unfortunately, the keywords used are not an example of clarity and consistency...)
We assume here that the concept of the visibility hypercube is understood. We recall that the group/observation, mosaic-subfield, frequency-channel and (for a mosaic observation) the successive hour-angle 'cuts' are selected through the compound sector index; the scan's hour-angle, and within a scan the interferometer and polarisation are selected by direct specification.
At the very start, NFLAG asks you to select a primary data hypercube to operate on, through the parameters
This primary cube may later be redefined partly or completely through the MODE branch of NFLAG which is accessible from several places in the program.
For certain operations, one may narrow down the range of visibilities to be operated on by defining a secondary hypercube; the data volume operated upon will then be the cross section of the primary and secondary cubes. Secondary-cube definition is limited to the latter three of the parameters listed above; note, in particular, that it is not presently possible to define a secondary frequency-channel range because that definition is part of the SCN_SETSparameter.
A word of caution is in order about the effect of data-cube definitions in operations that affect the scan-header flags. Since such actions operate on the entire scan, the ranges of interferometers and polarisations (parameters SELECT_IFRS and SELECT_XYX) that you defined for your hypercubes have no effect.
FIGURE
.]Bird's eye view of the flagging branch of NFLAG, showing the four main branches and their interconnections. Regular text represents manpulation of program variables. Interactions with the user are shown in boldface; an arrow shows parameter input into a program variable.
Full-drawn lines represent the main-branch connections, dashed lines the detours from one main branch into another one; the user selects these routes through the nparameter inputs shown.
Dotted lines represent the return paths that are selected through a QUIT input in each of the branches. All returns are through label 101 in the top left; in the case where the return is from a detour, control is returned to the branch where the detour originated.
FIGURE
.]Schematic of the operations branch of NFLAG.
This branch contains options to set visibility or scan-header flags on the basis of a variety of criteria. See text for a description.
The flagging branch is where the actual work of NFLAG is done. All commands that change flag setting on visibility points or scan headers are to be found here. The flagging options are shown in the upper left box in figure A detailed overview of the branch is given in figure.
FIGURE
.]Schematic of the INSPECT branch of NFLAG.
By default NFLAG executes a COUNT operation at the end of each
flagging operation. The COUNT option is provided mainly for convenience, but must be executed if you want to display flagging statistics before doing anything else.
The purpose of the inspection branch is to provide overviews of the flags present. Generally speaking, the bulk of the visibilities is much too large for making useful graphic or tabular representations of the individual flag settings. For most purposes, however, the information one needs can be represented in the form of tables of flag counts along various cross-sections of the visibility hypercube. Such tables is what the inspection commands produce.
A schematic overview of the branch is given in figure.
From the INSPECT branch, it is possible also to make a detour into the MODE or STATISTICS branch.
FIGURE
.]Schematic of the STATIST branch of NFLAG.
This branch accumulates statistical characteristics from the visibilities and/or the noise parameters in the scan headers and provides displays of the results in various formats.
The purpose of the statistics branch is to provide statistical overviews of the visibility data. Generally speaking, the bulk of the visibilities is much too large for making useful graphic or tabular representations of the individual values (except with NGIDS. For flagging purposes, however, much of the information one needs can be represented in the form of tables of data statistics along various cross-sections of the visibility hypercube. Such tables is what the statistics commands produce.
A schematic overview of the branch is given in figure.
From the STATISTICS branch, it is possible also to make a detour into the INSPECT branch.
FIGURE
.]The use of a flag file to transfer flagging commands into NFLAG.
Flag files contain flagging commands that NFLAG can read and execute. Both a compact binary form (the .FLF file) and a bulky text form are available. The latter form can be edited manually.
Flag files may be generated by NGIDS and through commands in the
FLIST branch of NFLAG.
For detailed point-by-point flagging of visibilities the flagging modes of NFLAG are much too cumbersome. For this purpose one may instead use a combination of
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.