Overview
During the course of the survey, portions of the sky are scanned multiple
times, yielding mulitple detections of the same astronomical object. The
goals of target selection are to:
- Produce a complete and unique list of objects
comprising the Primary Survey within the survey area. By unique,
we mean that for a given astronomical object in the sky, there is only
one corresponding object in the Primary Survey.
- Select spectroscopic targets from the Primary Survey.
- Track the progress of the imaging survey across the sky ---
i.e., declaring that we have detected the astronomical objects in a
given chunk of sky.
We must accomplish these goals subject to the following constraints:
- For a given piece of sky, targets come from only a single scan
(i.e., no averaging of scans).
- The complete object list comprising the Primary Survey shall be allowed
to grow incrementally
(i.e., target selection can be interleaved with the acquisition and
calibration of new imaging data).
This document describes the steps necessary to accomplish these goals.
First, some definitions:
Astronomical Object
- In this document, we will be referring to a real object in the sky (star,
galaxy, asteroid, etc) as an astronomical object (hereafter AO).
- Primary Survey
- The Primary Survey is the catalog of non-duplicate detections of AOs
(i.e., one entry per AO) in the survey area, complete to a limiting
magnitude, from which spectroscopic targets are selected.
- Object
- An object is a single detection, in all five filters, of an AO
in the Frames Pipeline. A given AO may be detected numerous times over
the course of the survey, thus giving rise to several objects
for a given AO. Causes for such multiple detections include:
- The Frames Pipeline is fed frames such that adjacent frames overlap
by 128 rows. All AOs within those overlap rows should
be detected twice, thus yielding two separate objects each. While
these objects should be exactly the same within the errors, comparing
these multiple detections provides useful quality analysis internal
to a Frames Pipeline Run.
- Adjacent drift scans overlap by some variable amount, thus
leading to multiple detections of AOs, and thus multiple objects.
- A given piece of sky may be scanned multiple times for a variety of
reasons: quality tests, unacceptable data from the first scan, or
for scientific reasons as in the Southern Survey.
- A given Drift Scan may be processed multiple times by the Frames
Pipeline, leading to multiple objects for each AO (more specifically,
multiple objects for a single observation of an AO). This will
happen certainly during the test year, and perhaps later as well if
we end up reprocessing all or some of the data one or more times.
While a given AO may be detected and parameterized as an object
numerous times in the course of the survey, we must choose one such object
for each AO to serve as the primary object of that
AO; i.e., the Primary Survey catalog entry for that AO.
Good objects which are not part of the Primary Survey are referred to
here as secondary objects; most of these will be duplicate
detections of primary objects.
- Field
- While imaging data for a given column of CCDs in the camera is acquired
as a continous drift scan, it is processed as individual fields. The
strip of data is divided into fields of 1361 rows by 2048 columns. The
Frames Pipeline processes each field in turn, appending the first 128
rows from the next field to each field, thus leading to duplicate
detections in these overlap regions. While these duplicate detections
may be of some interest for quality analysis internal to a run, they are
of no scientific interest, and so we need a means to declare which objects
are the detections for this Frames Pipeline Run which "count" (i.e.,
which are eligible for inclusion in the Primary Survey), and which are
the duplicate detections. We do so as follows. For each field, the
first 128 rows (X coordinates 0-128) and last 128 rows (X coordinates
1361-1489) overlap the previous and next fields, respectively. We split
each of these two sets of overlap rows in half. The portion of sky which
intersects rows 64 through 1425 (specifically, 64 <= X < 1425) is
considered to map onto this field; the portions of sky which intersect
the outside set of 64 rows on either end, 0 <= X < 64 and
1425 <= X < 1489, duplicate a portion of
sky which maps onto the previous and next fields, respectively. This
gives us a one-to-one mapping from sky to field for a given Frames
Pipeline Run. The unique objects to use from a given Frames Pipeline
Run are then those which are detected within the central set of rows
which map onto the sky for that field. There are problems with this
schema at the border, but these will be discussed below.
- Object List
- Objects are created by the Frames Pipeline. Each Frames Pipeline
Run (hereafter FPR) targets a contiguous set of fields in a single
column of the Imaging Camera for a single Drift Scan,
detecting and parameterizing all objects in each field.
The Frames Pipeline has been designed to work on a single field at a time.
Astrometric and photometric calibrations are quantized in units of fields.
This leads us to the concept of the Object List (hereafter OL).
An OL is the list of objects detected in a single field in a single run
of the Frames Pipeline. Note that there is not a one-to-one mapping of
OL to field; there may be multiple OLs for a given field, one for each
time the Frames Pipeline ran on a given camera column for a given Drift
Scan. An OL serves as the fundamental quanta for assigning quality to
the output of the Frames Pipeline. Each OL is assigned a single quality
parameter.
- Stripe
- A stripe is a predefined piece of sky, with a minimum and maximum survey
latitude, and a minimum and maximum survey longitude. Adjacent stripes
meet at a constant survey latitude border. There is a one-to-one
mapping from sky to stripe.
- Scanline
- The survey observing strategy leads to a natural carving of the sky into
overlapping scanlines. A scanline is defined as that portion of the sky
covered by a single column of the imaging camera while scanning along
either the north or south strip of a single stripe. Each scanline
overlaps at least one other scanline on either side. For example,
scanline 6 in the north strip of stripe 3 overlaps scanline 6 in the
south strip of stripe 3 and scanline 1 in the south strip of stripe 4.
Towards the poles of the survey area, three or more scanlines may overlap
in a given area of sky.
We may define a one-to-one mapping from sky to scanline for a given
stripe. Each drift scan along a given stripe tracks the same predefined
great circle. The imaging pipelines use a great circle coordinate system
defined such that the equator of the coordinate system lies on top of the
tracked great circle. In this coordinate system, scanlines very closely
follow lines of constant latitude. We can then, for each scanline,
define a range of great circle latitude on the sky assigned to
that scanline, which abuts the range of latitude assigned to the
adjacent scanlines without overlapping them. The CCDs in the associated
camera column will always cover this range in latitude with additional
overhang. This requires assumptions on the Image Camera geometry, as
well as on the accuracy with which
each drift scan tracks its strip, however a substantial error in these
assumptions may be tolerated due to the overlap between scanlines.
For each stripe, we can thus uniquely assign a given piece of sky within a
stripe to a scanline. Scanlines between stripes may overlap, but by
limiting our attention only to those portions of the sky assigned to a
scanline which intersect the portion of the sky assigned to the stripe,
we circumvent such overlap.
The assignment of primary/secondary status would be simplified if we could
first uniquely assign an AO to a given field/scanline/stripe. However,
this is not possible. In the case of AOs which are close to the border
between two fields/scanlines/stripes, different detections/objects for the
same AO may fall in different fields/scanlines/stripes due to random errors in
the astrometric calibrations or measured instrumental positions, blending of
objects in one Frames Pipeline Run but not the other, etc. Even for a given
object, a new astrometric calibration may move it from one scanline/stripe to
another. We need a mechanism to assure a complete and unique sample of primary
objects near such borders. This is discussed here, but since the mechanism
depends on matching duplicate detections of the same AO, we must first discuss
our matching algorithm.
We must be able to match up multiple detections/objects of the same
AO before we can perform much quality analysis, select out primary
objects, or perform target selection. Such matches will not always be
one-to-one between two given Frames Pipeline Runs; e.g., two or more objects
may be differentiated in one Frames Pipeline Run but blended together as one
object on another. Various algorithms may be used for matching. For example,
two objects could be considered to match (i.e., are
considered two detections of the same AO) if the measured center
of one object falls within the effective area of the other (as determined by
its atlas image). This should be a rigorous, unambiguous, but CPU-intensive
criterion. A simpler criterion is that two objects match if their centers lie
within 1 arcsec of each other. The choice of a matching algorithm is a detail
(albeit an important one) which does not affect the stucture for tracking
survey progress and target selection presented here.
We adopt the following matching algorithm. Matching is always done in quanta
of Frames Pipeline Runs (FPRs), in the sense that when looking for matches to
a given target object, we compare that target object to all objects in an FPR,
and loop over FPRs. If the
coordinate separation of the target object and another object is less than a
specified value, then the two objects match. If the target object matches
more than one object in an FPR, only the closest match is retained. Note that
this process is not associative; the target object can only match one object
in a given FPR, however an object in the FPR may end up matching more than one
target object from a single FPR containing many target objects. Thus the order
in which matching is performed matters, but presumably not in any way we care
about. A given target object may match objects in more than one FPR.
A link between the two objects in each matched pair is created in the database.
We now return to the problem presented by borders between fields, scanlines,
and stripes. At various stages, we need to decide whether an object belongs
to a field, scanline, or stripe. All objects that do so have a status
bit-flag set, which we'll generically refer to as XXXX (in reality there is a
different bit-flag depending on whether we're resolving fields,
scanlines, or stripes). This decision is based on whether the object
lies in the unique piece of sky mapped onto that field, scanline, or stripe.
Since the mapped pieces of sky are unique within their context, by choosing
objects found only on those pieces of sky we assure ourself of a unique set of
objects. This fails at the borders between pieces of sky (i.e., between
adjacent fields in an FPR, between adjacent scanlines in a stripe, between
adjacent stripes). Two detections of the same AO may fall in different
pieces-of-sky/fields/scanlines/stripes due to imperfect position measurements,
thus leading to AOs with multiple or no associated objects considered eligible
for the Primary Survey. We handle such situations as follows. When adjusting
an object's status based on it position in a predefined piece of sky (i.e.,
within the set of primary rows for a field, within the range of great
circle latitude for a scanline, within the range of survey latitude for a
stripe), we check whether there is an abutting piece of sky that has already
been resolved at that level. If so, then we examine all objects in our target
piece of sky (and thus with the bit-flag XXXX set) that lie within the
matching radius (the minimum separation used to find matching objects) of the
border. If any of these objects have a matching object which lies within the
abutting piece of sky (and thus has its bit-flag XXXX already set), then we
unset the bit-flag XXXX on the object in our target piece of sky; that AO has
already been assigned to the previously resolved piece of sky. Similarly,
we then examine all objects that lie just outside our target
piece of sky (and thus with the bit-flag XXXX unset) but within the
matching radius of the
border. If any of these objects have a matching object which lies just outside
the abutting piece of sky (and thus has its bit-flag XXXX unset), but which
would have been assigned to that piece of sky had their position placed them
within it, then we
set the bit-flag XXXX on the object in our target piece of sky; that AO was
not assigned to the previously resolved piece of sky and so we must assign it
to our target piece of sky or it will be missed. If all AOs yield exactly
one object each time they were observed, the measured positions were always
well within the matching radius of the true position, and our matching were
perfect, then this process would yield a truly complete and unique set of
objects for the Primary Survey. In effect, none of those assumptions will be
perfectly met, but in as much as they are we will be left with a complete and
unique sample.
We are now set to discuss the operations in target selection. Each object has
a status bit-flag attribute which tracks the status of the object within
the survey. Most operations touch one or more of these bits. All bits are
unset (0) at the start of these operations. Below, the different bit-flags are
capitalized (e.g., GOOD, OK_RUN, etc). When a bit-flag is
set, its value is set to 1; when unset, its value is set to 0. When we refer
to "all XXXX objects", where XXXX is a bit-flag (e.g., "all GOOD objects"), we
refer to all objects which have bit-flag XXXX set to 1. When we say
"an object is set XXXX", where XXXX is a bit-flag, we mean that object's
bit-flag XXXX is set.
There are 6 operations which bring the objects from the Final Photometric
Calibration through target selection, and which perform most of what is
required to define the imaging survey. Each step is outlined here.
A single execution of the Set Object Status task operates on all
objects in a single FPR. This task performs all operations that can be
performed on a single FPR independent of all other FPRs. It performs the
following:
- The final quality of an object (good/bad) is set, based on the flags
set by the Frames Pipeline for each of the individual filter detections.
A "good" object is one whose quality is sufficiently high that it is
acceptable for the Primary Survey (this does not guarantee that it will make
it into the Primary Survey). A bad object is considered unacceptable for
the Primary Survey. If the object is considered good, then its status bit-flag
GOOD is set.
- If a GOOD object lies within the piece of sky mapped to that field
(i.e., if its X coordinates lies within the range 64 <= X < 1425; see the
definition of field), then it's status bit OK_RUN is set.
- GOOD objects, of class STAR, GALAXY, or UNK only, lying in the overlap region
between two adjacent fields are matched to GOOD objects of class STAR, GALAXY,
or UNK in the adjacent field. Each object in a matching pair has its DUPLICATE flag
set. A border resolution
is performed at the border between fields, such that for matched pairs between
adjacent fields near the border, exactly one object in each pair has its
OK_RUN flag set.
In summary, the following bits in the status flag are set for each
object in the FPR:
- SET
- 1 for each object, indicating it has been processed through this
operation.
- GOOD
- 1 if the object quality is acceptable for the Primary Survey, 0 if it
is not.
- OK_RUN
- 1 if this is the unique detection of this AO within this FPR (GOOD objects
only); such objects are eligible for inclusion in the Primary Survey.
- DUPLICATE
- 1 if this OK_RUN object (type STAR, GALAXY, or UNK only) has a duplicate OK_RUN
detection (type STAR, GALAXY, or UNK only) in an adjacent field.
The set of OK_RUN objects comprises a complete and unique set of
objects within that FPR; these objects alone are eligible for inclusion in
the Primary Survey.
This task is entirely automatic. It may be undone with no side-effects, so
long as the task Resolve has not been run on any
portion of the FPR.
A single execution of the Merge Objects task operates on all OK_RUN
objects in a single FPR. Each OK_RUN object, of class STAR, GALAXY, or UNK
only, is matched against all OK_RUN objects, of class STAR, GALAXY, or UNK only,
in all other FPRs which themselves have been merged, and database links are
made between any matches found.
This task is entirely automatic. It may be undone with no side-effects, so
long as the task Resolve has not been run on any
portion of the FPR.
A quality must be assigned to every OL by a human. Only OLs whose quality
attributes are other than "undetermined" may be processed through
Resolve. This task
operates on an arbitrary set of OLs. It may be performed multiple times. No
record of quality assignment is kept, other than the setting of the quality
flags for the individual OLs.
If an OL is assigned a quality more than once, the
old quality is written over, with no history kept of any previous QA.
There are two natural times to perform Quality Assignment:
- The Frames Pipeline Postprocessor stuffs into the
Operational Database the results of
a single FPR, making the appropriate bookkeeping links, and assigns an
initial data and reduction quality to each OL in the run. Initial
quality analysis and assignment must be done at this stage, as we have
easy access to
all outputs of the Frames Pipeline (including things like corrected
frames which are not easily accessed from the Operational Databases).
- Just before running Resolve, we have a set of
FPR which cover all scanlines in a stripe over a range of great cirlce
longitude. These have all been processed through
Set Object Status and
Merge, and so we have access to a complete set of
matching statistics in the overlap regions between the scanlines. These
matching statistics are vital to Quality Analysis, and thus some Quality
Assignment must be done at this point.
This task may be undone (i.e., the quality attributes of an OL set to
"undetermined") so long as the OLs have not been processed through the
task Resolve.
The Resolve task selects primary and secondary objects in a given
piece of sky. The piece of sky is defined as a range in great circle
longitude on a given stripe (known as a chunk of sky). All FPRs
associated with that stripe, and which have been processed through the
Set Object Status and Merge
tasks, are broken up into segments, where a segment consists of all
objects in an FPR
whose coordinates lie within a range in great circle longitude, and whose OLs
all have the same data quality (good, bad, or undetermined). There must exist
a set of good segments (data quality is good) which spans the chunk's range of
longitude for each scanline in the range.
Each scanline is processed separately. The user may prioritize the list
of good segments for each scanline; by default, they are prioritized by run
number, such that the earliest run has the highest priority.
The highest priority segment is used to select primary objects in as much of
the scanline within the chunk as it can, with lower priority segments filling
in the gaps. Undetermined segments are not used. For each good and bad
segment, the following steps occur:
- Each good segment is broken up into smaller segments; primary segments in
longitude ranges that have not yet been resolved
(primary objects selected), and secondary
segments in longitude ranges where they have been resolved.
- All objects that lie within the longitude range of a segment
are considered to belong to that segment. If a primary
segment abuts an adjacent previously-resolved primary segment in
the same scanline, then a border resolution is
performed at the border between the segments so as to guarantee all
objects belong to exactly one segment. All OK_RUN
objects belonging to the segment have their RESOLVED bit set
to indicate that it has been resolved. If the object lies in the first
OL in the segment, then its FIRST_FIELD bit is also set, to keep
track of which segment objects belong to for OLs that are shared by two
segments. If the segment is a primary segment, then each objects'
PSEGMENT flag is also set.
- For secondary segments, all RESOLVED objects, of class STAR,
GALAXY, UNK, or SKY only, belonging to
the segment have their SECONDARY bit set, indicating that this
object is a secondary object (i.e., it is a good detection, but is
not part of the Primary Survey).
- For primary segments, if a segment RESOLVED object lies within the
piece of
sky mapped to its associated scanline (i.e., with the range of great
circle latitude predefined for that scanline), then its OK_SCANLINE bit
is set (a border resolution is performed at the
latitude border if a previously resolved primary segment in an adjacent
scanline overlaps it in longitude). For each such OK_SCANLINE object
in the segment, if it lies within the stripe's survey latitude
boundaries, then
the OK_STRIPE bit is set (again resolving at the stripe borders if
necessary). For all RESOLVED objects, of class STAR, GALAXY, UNK, or SKY
only, belonging to the segment, if their OK_STRIPE bit-flag is
set and they lie within the survey longitude borders for that stripe,
then their PRIMARY bit-flag is set, else their SECONDARY bit-flag
is set. The set of all PRIMARY objects in all primary segments
comprises the Primary Survey.
In summary, the following bits in the status flag are set for each
resolved object in the chunk of sky:
- RESOLVED
- 1 for each object, indicating it has been processed through this
operation. Object must be OK_RUN. None of the bit-flags listed below may
be set if this one is not set.
- PSEGMENT
- 1 if this RESOLVED object belongs to a primary segment, else 0.
- FIRST_FIELD
- 1 if this RESOLVED object belongs to the first OL in the segment.
This is used to
determine which segment an object belongs to for OLs which are shared by two
segments.
- OK_SCANLINE
- 1 if this RESOLVED object falls within the great circle latitude limits
for this scanline, indicating this is the unique detection of this object in
this stripe. This can be set only for objects in primary segments (PSEGMENT
set).
- OK_STRIPE
- 1 if this RESOLVED object falls within the survey latitude limits for this
stripe, indicating this is the unique detection of this object in the sky.
This can be set only for objects in primary segments (PSEGMENT set) which
have OK_SCANLINE set.
- SECONDARY
- 1 if this object is a good detection but does not belong to the Primary
Survey. All objects of class STAR, GALAXY, UNK, or SKY with RESOLVED set but
either OK_STRIPE not set or which lies outside the stripe's longitude borders
will have this flag set.
- PRIMARY
- 1 if this object is part of the Primary Survey. All objects with
RESOLVED and OK_STRIPE both set, of class STAR, GALAXY, UNK, or SKY only, and which
lie within the stripe's longitude borders will have this flag set.
This task is automatic, except for the optional prioritizing of good segments
to use as primary segments.
This task may be undone only if the resultant chunk has not been processed
through Automatic Target Selection.
Automatic target selection operates on a chunk of sky. All
primary objects belonging to that chunk are passed through the
target selection algorithms; those objects selected as targets have their
TARGET bit set.
This task is an entirely automatic task, requiring no human decisions.
This task may be undone only if none of the objects in the chunk have been
tiled.
Manual target selection operates on a chunk of sky previously
processed through Automatic Target Selection).
All primary objects contained within this chunk are presented to the Survey
collaborators via the Science Archive Database for manual target selection.
Those objects selected as targets have their TARGET bit set.
This task is an entirely human task.
This task may be undone only if none of the objects in the chunk have been
tiled.