Usage

Setting Up

sdssdp1> export OO_FD_BOOT=/opdb/d1/data/opdb/photo

sdssdp4> setenv OO_FD_BOOT /opdb/d1/data/opdb/photo

sdssdp4> setup ts
sdssdp4> ts
ts>

ts> dbInit -nMaxPages 5000
ts> sessionBegin
ts> photoInit

ts> sessionBegin -readonly -mrow
ts> photoInit -readonly

ts> sessionEnd
ts> quit
sdssdp4>

Doing

Set Object Status

ts> set stats [setObjectStatus 94 1 0 -authors adelman]

ts> set stats [setObjectStatus 94 1 -authors adelman -comment "Blame Yanny"]

ts> set stats [setObjectStatus 94 1 -authors adelman -matchRadius 3]

ts> set stats2 [mergeQABin $stats 20]

ts> dbMatchPlot $stats

ts> dbMatchPlot $stats2 -file plots.ps

ts> dbMatchPrint $stats pos

The same statistics can also be generated for a Frames Pipeline Run which has already been through "setObjectStatus" by using the selfMergeQA task. For example, to generated the statistics for run 94, camCol 1:

ts> set run [pipeGet frames 94 1]
ts> set stats [selfMergeQA $run -noBlends -nSigma 5]
ts> objDel $run

ts> taGenericChainDestroy $stats

Merge

ts> runMerge 94 1 0 -authors adelman

ts> runMerge 94 1 -authors adelman -matchRadius 1.5

Quality Analysis

ts> set run1 [pipeGet frames 94 1 -rerun 0]
ts> set run2 [pipeGet frames 125 2 -rerun 1]
ts> set stats [runMergeQA $run1 $run2]
ts> objDel $run1
ts> objDel $run2

ts> set stats [runMergeQA $run1 $run2 -noBlends -firstField $field1 \
                          -lastField $field2]

runMergeQA returns a chain of TSMATCHSTATS statistics, one entry for each field in the first run, based on the pairs of objects matched between the two runs. By default, matching statistics are compiled for objects in a default magnitude range in a default filter: these values may be changed using the arguments "-min", "-max", and "-filter". For example, to examine matched pairs where the object in run 1 in the pair is between 17th and 19th magnitude in g':

ts> set stats [runMergeQA $run1 $run2 -min 17 -max 19 -filter g]

The statistics are for OK_RUN objects of class STAR, GALAXY, or UNK only. They can be further limited by specifying sets of flags that an object must have and must not have, with a separate set of flags for each run. For example, to limit the selection to objects in run1 which have the BLEND and CHILD flags set, but don't have the EDGE flag set, matching objects in run2 which have the EDGE flag:

ts> set stats [runMergeQA $run1 $run2 -flags1On {BLEND CHILD} \
               -flags1Off EDGE -flags2On EDGE]

ts> set stats [runMergeQA $run1 $run2 -flags1On {BLEND CHILD} \
               -flags1Off EDGE -flags2On EDGE -flagsFilter g]

Chains containing each matched pair may be returned by specifying the "-obj"
option.  One chain will contain all star-star pairs, one all galaxy-galaxy
pairs, and one all other pairs (unknown-unknown pairs and mismatches).  If
the "-full" option is all specified, then the pair chains will contain the
full calibrated objects, and an two additional chains containing field
information for each run will also be returned
(the "-full" option slows things down and increases memory usage
by about a factor of ten, since the full objects need to be read).  For
example, to repeat the above query but also return the full objects for every
matched pair:
ts> set results [runMergeQA $run1 $run2 -flags1On {BLEND CHILD} \
                 -flags1Off EDGE -flags2On EDGE -flagsFilter g -obj -full]
ts> set stats [lindex $results 0]
ts> set stars [lindex $results 1]
ts> set galaxies [lindex $results 2]
ts> set others [lindex $results 3]
ts> set fields1 [lindex $results 4]
ts> set fields2 [lindex $results 5]


If a second run isn't specified, then only the non-matching statistics
are calculated.  If, in addition, the "-full" option is specified, then the
calibrated objects meeting the selection criterion are returned in three
TA_CALIB_OBJ chains, one each for objects of class STAR, GALAXY, and UNK.
For example, to fetch all CHILD objects from a run which are not EDGE:
ts> set results [runMergeQA $run1 -flags1On CHILD -flags2Off EDGE -full]
ts> set stats [lindex $results 0]
ts> set stars [lindex $results 1]
ts> set galaxies [lindex $results 2]
ts> set unknowns [lindex $results 3]
ts> set fields [lindex $results 4]

The statistics are on a per field basis.  They may be binned over fields to
increase the number of pairs used for each set of statistics using the
mergeQABin proc.  This proc returns a
TSMATCHSTATS chain.  For example, to bin over 20 fields:
ts> set stats2 [mergeQABin $stats 20]

Plots based on the returned statistics chain may be displayed using the
dbMatchPlot proc.
ts> dbMatchPlot $stats -calib -log

The "-calib" arguments tells the proc that calibrated quantities are being
plotted, rather than instrumental quantities.  The "-log" argument causes
counts to be plotted as log counts.
To plot to a postscript file, rather than the screen, use the "-file" option:
ts> dbMatchPlot $stats -calib -log -file plots.ps

A TCL proc is provided to create plots for all overlapping columns between two
runs.  For example, to do so for runs 1140 and 1231:
ts> runMatchPlot 1140 1231 30 mag -file all.ps -calib

The statistics can be printed to the screen using the
dbMatchPrint proc.  For example, to print
the differences in positions:
ts> dbMatchPrint $stats pos

For a full description of the statistics chain and routines to examine it,
see here.

runMergeQA can also return chains containing each matched pair, with
a separate chain for stars and galaxies.  Use the "-obj" flag for this feature:
ts> set list [runMergeQA $run1 $run2 -obj]
ts> set stats [lindex $list 0]
ts> set stars [lindex $list 1]
ts> set galaxies [lindex $list 2]

For each matched pair, the great circle longitude and latitude (mu and nu, in
degrees),
the magnitude in each filter (PSF magnitude for stars, petrosian magnitude
for galaxies), and the model colors are given for the object in the first run,
along with the differences in those quantities between the two runs (in the
sense of run 1
quantity minus run 2 quantity, in arcsecs for dmu and dnu, mag for dmag and).
dcolor).  The photo flags for each object in the pair are also given.
For example, to plot the difference in mu (in arcsecs) versus mu (in degrees)
for stars:
ts> set pg [pgstateNew]
ts> pgstateOpen $pg
ts> ePlot $pg $stars.mu $stars.dmu

If the "-full" flag is specified, then both entire calibrated objects are also
included: 
ts> set list [runMergeQA $run1 $run2 -obj -full]
ts> set stats [lindex $list 0]
ts> set stars [lindex $list 1]
ts> set galaxies [lindex $list 2]

For example, to plot the difference in row velocity versus r' petrosian
magnitude for galaxies:
ts> ePlot $pg $galaxies.mag<2> \
          $galaxies.obj1->rowv.val-$galaxies.obj2->rowv.val

The "-full" flag slows things down, as it must read a much bigger chunk of the
OPDB to fetch the full objects.
When done with the stats chain, it may be destroyed with
"taGenericChainDestroy":
ts> taGenericChainDestroy $stats

The matched-pairs chains must be destroyed with "matchedPairChainDel":
ts> matchedPairChainDel $stars
ts> matchedPairChainDel $galaxies

An alternate way to examine match-ups between Frames Pipeline Runs is to use
the GUI.  Select one Frames Pipeline Run with the left mouse button.  Then
select another with the right mouse button.  The matching statistics will be
displayed using the GUI's tool.

Quality Assignment

Once quality analysis is finished on a segment of a Frames Pipeline Run, a
quality (undetermined, bad, acceptable, good) may be assigned.  Quality is
assigned on a field by field basis.  The proc to set quality,
fieldQualitySet, assigns
the same quality to a contiguous range of fields in a single Frames Pipeline
Run.  For example, to set fields 23-248, inclusive, in run 94, camCol 1,
rerun 0 to a quality of "good":
ts> fieldQualitySet 94 1 good 23 248 -rerun 0

The "-rerun" argument is required only if there is more than one Frames
Pipeline Run stuffed for that run and camCol.  If a starting and ending field
aren't specified, then all fields in the Frames Pipeline Run are set to that
quality.  For example, to set all fields to "bad" for run 125, camCol 3:
ts> fieldQualitySet 125 3 bad

To set fields 244 to the end of the run to acceptable for run 125, camCol 3:
ts> fieldQualitySet 125 4 acceptable 244


Resolve

"Resolve" specifies which objects are primaries and which secondaries in a
given chunk of sky (for a complete description, see
here).
OPDB maintains two versions of the sky, the version used to drill plates, and
the current (latest-and-greatest) version.  The drilling version is assigned
the number 0 and the current version the number 1.  When you resolve you must
specify which version of the sky you're resolving against.

Resolving is a two stage process.  The first stage selects the chunks to be
resolved using a GUI.  The actual resolving is not performed at this stage.
Rather, scripts files are created which may be submitted outside of the GUI
to do the actual resolving, which comprises the second stage.
The GUI takes one argument, namely the version of the sky to work on.
It can also be run in "readonly" mode, which it should be for this purpose.
Thus, to launch the GUI on the drilling version of the sky:
sdssdp4> ts -useTk
ts> tsGui 0 -readonly

The GUI should now display all segments, color coded as indicated
by the legend on the left.  The southern stripes are displayed first, then
the northern stripes.  "Primary", "secondary", and "bad" segments have already
been resolved.  "Unresolved ok" and "unresolved bad" segments are unresolved
pieces of Frames Pipeline Runs which have had a quality assigned
("unresolved bad" corresponding to quality "bad", and "unresolved ok"
corresponding to quality "good" or "acceptable"), and which have been
processed through the setObjectStatus and
merge tasks.  "Undetermined" segments are unresolved
pieces of Frames Pipeline Runs which have been processed
through setObjectStatus and
merge, but have not had a quality assigned.  "Unprocessed"
segments are unresolved pieces of Frames Pipeline Runs which have not been
processed through either or both setObjectStatus
and merge.  Only segments which have been processed
through both setObjectStatus
and merge, which have had a quality assigned (bad,
acceptable, or good), and which have not previously been resolved are eligible
to be resolved (that is, only "unresolved ok" and "unresolved bad" segments).

Many tasks not directly related to resolving are available from the GUI, and
are briefly summarized in this paragraph.  Clicking with the middle mouse
button on a segment will list identifying information on it in the lower left
panel.
Clicking with the left mouse button on a segment "selects" it (indicated by
it's border changing colors).  The buttons on the left perform the tasks listed
by clicking with the left mouse button (additional tasks are available via
the "Actions" menu).  Clicking the "set object status"
button will process the selected segment through the
setObjectStatus task (using the default
matching radius).    Clicking the "merge" button will
process the selected segment through the
merge task (using the default matching radius).  Clicking
on a second segment with the right mouse button will generate matching
statistics between it and the selected segment, and display those statistics
with the GUI's QA plotting tool (as well as give the user the option to set
the quality over ranges of fields for the selected segment).  The plot may
be manipulated using the "QA" menu.  Many of the tasks required that the
"readonly" mode not be used, however these tasks will generally be performed
without the GUI.

To resolve, we must first display those regions of the sky which are ready to
resolve.  A resolvable piece of sky is a continuous range in survey longitude
of a stripe which is completely covered by "unresolved ok" segments.
Press the "plot chunks-to-be" button with the left mouse button.  A purple
rectangle will be plotted delimiting each piece of a stripe which can be
resolved.  Click the border on
one of the purple boxes.  The border will turn gold, indicating
you've selected it.  Then press the "resolve" button with the left mouse
button.  You will be prompted for a list of runs to use for the primary
segments.  You may specify a list of runs to use.  For example, if you wish
to use runs 259 and 273 as the primary segments, then entery "259 273".  If
nothing is entered, then the lowest numbered runs will be used as the primary
segments.  That is, for a given scanline, the good segment with the lowest run
number will be used as the primary segment over its full extent, with the
rest of the scanline filled by the good segment with the next lowest run number
and so on.  No actual resolving will be performed.  Rather, the file
"$opdb_root/targetChunk/stripe$stripe_mu$startMu/resolve/resolve.tcl" will be
created.  Additional chunks-to-be may be selected and similar files created
with one GUI session.  To finish the resolving, exit the GUI and execute
each script.  For example, say you are resolving the chunk for stripe 10
starting at mu 45827 (arcsecs).  After "resolving" the chunk in the GUI, exit
the GUI and:
sdssdp4> cd $opdb_root/targetChunk/stripe10_mu45827/resolve
sdssdp4> ts -noTk -file resolve.tcl

The chunk will now be resolved.  This takes quite a while; 
progress will be indicated in the xterm in which each field will be printed
as it is resolved.


You can get some feedback on whether things work by using
fieldPlot.  This is
primarily a debugging tool to visualize whether the object status flags were
properly set.  It plots up GOOD objects in a set of contiguous fields (spans
of three fields are nice).  For example:
ts> set skyVersion 0
ts> set run [pipeGet frames 745 5]
ts> fieldPlot $run $skyVersion -first 142 -last 145 -res

The meaning of the various symbols and lines is given in the task
documentation.

After a chunk of sky has been resolved, there may be additional scans which go
through the same chunk of sky.  We would like to be able to resolve this
additional data, setting all objects in the portion overlapping the previously
resolved chunk to SECONDARY (since PRIMARIES have already been selected in
that piece of sky).  When this situation arises, the following steps may be
performed.  First, fetch a list of all segments (both resolved and unresolved),
using the proc segmentsGet:
ts> set segments [segmentsGet $skyVersion]

Next, call the proc
repeatSegmentsGet, which will
parse the list of segments and return a list of all segments which have not
been resolved, have an acceptable survey quality, and are completely overlapped
by previously resolved primary segments:
ts> set repeats [repeatSegmentsGet $segments $skyVersion]

A TCL list of segments is returned, where each element in the list is itself
a 5-element TCL list, consisting of the run number, camera column, rerun
number, starting great circle longitude (in integer arcsecs), and ending
great circle longitude (in integer arcsecs) of the segment.  If desired, one
can remove segments from the list that you don't wish to resolve (using
the TCL command "lreplace").
Finally, resolve that set of repeat segments using the
repeatResolve proc:
ts> repeatResolve $repeats $skyVersion yanny


Automatic Target Selection

Automatic target selection in the OPDB is performed almost exactly like its
done on flatfiles.  Set up your plan (tsPlan.par) and tunable parameter files.
When running out of the OPDB, there are a few differences in keywords in the
plan file.  For a description of the differences, see the data model
definition of the file
here.
You must also specify which version of the sky you're selecting targets
from (0=drilling, 1=current).
Once you've set up the input files, then just issue the command
dbTargetSelect.  For example,
to select target in the current version of the sky:
ts> set skyVersion 1
ts> dbTargetSelect tsPlan.par $skyVersion {adelman yanny}

The first argument is the name of the plan file, and the second argument the
list of folks who are running target selection.  If this chunk has already
been run through target selection, you may force it to be rerun through
target selection again using the "-force" argument (all effects of the
previous target selection will be undone):
ts> dbTargetSelect tsPlan.par $skyVersion {adelman yanny} -force

The "target" product provides a routine, "taDiagnostics", to generate various
QA files after target selection runs off of flatfiles.  A simple TCL proc,
tsDiagnostics, is provided to
provide the same functionality out of the OPDB.  It only works if flatfiles
were exported when target selection ran.  It simply generates a "tsChunk" file
and then calls "taDiagnostics" on the flatfiles.  Just run the routine on
the same plan file used to run target selection:
ts> tsDiagnostics tsPlan.par

You can optionally specify a directory for the output files.  For example,
to create all output files in the directory "output":
ts> tsDiagnostics tsPlan.par output


Manual Target Selection

Manual target selection occurs after automatic target selection has been
performed on a chunk, and the chunk is exported to the Science Database.
A fixed period of time is allotted during which people may manually select
additional spectroscopic targets for that chunk.  After the time period has
elapsed, these targets are collected in a single
tsManual
file.  To enter these manually selected targets into the OPDB (necessary to
include them in tile and plate creation), issue the command
manualSelect with the
"tsManual" file:
ts> manualSelect tsManual.par


Tile Preprocessor

The "tile" pipeline requires as input a
tiInputTargets
file, which lists all untiled tilable spectroscopic targets.  This file may be
generated using the command untiledToFits:
ts> untiledToFits

The file, named "tiInputTargets.par", will be created in the current directory.
Optionally, the output directory for the file may be specified.  For example,
to create the "tiInputTargets.par" file in the directory "output":
ts> untiledToFits output

The "tile" postprocessor is described in the "photoSch" product.

Complete Tile Preprocessor

The "completeTile" pipeline requires as input a list of all tiles not yet
processed through "completeTile"
(tiTileCenters.par), and a list of all spectropic targets, whether already assigned to
a tile or not
(ctObjs.fit).
These files may be generated using the 
completeTilePre command:
ts> completeTilePre

The files, named "tiTileCenters.par" and "ctObjs.fit", are created in the
current directory.  Optionally, the output directory for the file may be
specified.  For example, to create the files in the directory
"/data/dp1.9/data/completeTile":
ts> completeTilePre /data/dp1.9/data/completeTile

The "completeTile" postprocessor is described in the "photoSch" product.

Undoing


This section describes how to undo, or back out of the OPDB, the various tasks
involved in target selection.

Undo Set Object Status
setObjectStatus may be undone with no
side-effects, so long as the task resolve has not been
run on any portion of the Frames Pipeline Run.  To undo the operation, use
the proc unsetObjectStatus.  For
example, to undo "setObjectStatus" on run 94, camCol 1:
ts> set run [pipeGet frames 94 1]
ts> unsetObjectStatus $run
ts> objDel $run


Undo Merge Objects
runMerge may be undone with no
side-effects, so long as the task resolve has not been
run on any portion of the Frames Pipeline Run.  To undo the operation, use
the proc runUnmerge.  For
example, to undo "runMerge" on run 94, camCol 1:
ts> set run [pipeGet frames 94 1]
ts> runUnmerge $run
ts> objDel $run


Undo Set Quality Assignment
fieldQualitySet may be undone with no
side-effects, so long as the task resolve has not been
run on any of the object lists corresponsing to the fields whose qualities you
are unsetting.  To undo the operation, use the same
fieldQualitySet proc, simply
setting the qualities to "undetermined".  For
example, to undo "fieldQualitySet" on run 94, camCol 1, fields 120-248:
ts> fieldQualitySet 94 1 undetermined 120 248


Undo Resolve
resolve may be undone only if the chunk has not been
processed through Automatic Target Selection.  Chunks are
identified by their stripe number and the starting great circle longitude (mu),
expressed in an integer number of arcsecs.  To undo the "resolve" operation,
use the proc chunkUnresolve.  For
example, to unresolve (and thus destroy) the chunk starting at mu=65378 arcsecs
for stripe 15:
ts> set chunk [queryGetOne arChunk $photo_segments \
               "stripe_ == 15 && startMu_ == 65378"]
ts> chunkUnresolve $chunk


Undo Target Selection
Target selection, both automatic and
manual, on a chunk may be undone only
if none of the targets selected have been assigned to a tile.  To undo
target selection, use the proc
dbTargetUnselectFromChunk.
Both automatically and manually selected targets will be de-selected.
For example, to undo target selection for the chunk starting at mu=65378
arcsecs for stripe 15 in the current version of the sky:
ts> set chunk [queryGetOne arChunk $photo_segments \
               "stripe_ == 15 && startMu_ == 65378 && skyVersion_ == 1"]
ts> dbTargetUnselectFromChunk $chunk
ts> objDel $chunk


Undo Tiling
To remove a single tile from the OPDB:
ts> set tileId 259
ts> untile $tileId

The tile must not have any plates attached to it.  To remove all tiles from a
given tiling run, specify the "tiTileCenters" file generated by that tiling run
to the command allUntile:
ts> allUntile tiTileCenters-0259-0287.par