sdssdp1> export OO_FD_BOOT=/opdb/d1/data/opdb/photoFor "csh" users:
sdssdp4> setenv OO_FD_BOOT /opdb/d1/data/opdb/photoYou should check with Survey Operations for the correct value to set the variable to. To access the OPDB, one must setup and run the "ts" product.
sdssdp4> setup ts sdssdp4> ts ts>One must then start a database session, and initialize access to the OPDB. Set the maximum number of page buffers to a comfortable number.
ts> dbInit -nMaxPages 5000 ts> sessionBegin ts> photoInitThe above pair of commands grants update-access to the OPDB. For read-only access (a nice safety precaution if you're only browsing):
ts> sessionBegin -readonly -mrow ts> photoInit -readonlyAfter performing all operations, the database session must be closed before quitting the "ts" executable.
ts> sessionEnd ts> quit sdssdp4>
ts> set stats [setObjectStatus 94 1 0 -authors adelman]If there is only one Frames Pipeline Run for this camCol of this run, then it is not necessary to specify the rerun number. An optional comment may also be entered:
ts> set stats [setObjectStatus 94 1 -authors adelman -comment "Blame Yanny"]A default value is used for the radius when matching objects in adjacent fields (see here for the default value, or type "help setObjectStatus" in the "ts" shell). A different value (in pixels) may be specified using the "-matchRadius" flag. Thus, to match with a radius of 3 pixels:
ts> set stats [setObjectStatus 94 1 -authors adelman -matchRadius 3]The statistics are on a per field basis. They may be binned over fields to increase the number of pairs used for each set of statistics using the mergeQABin proc. This proc returns a TSMATCHSTATS chain. For example, to bin over 20 fields:
ts> set stats2 [mergeQABin $stats 20]Plots based on the returned statistics chain (binned or unbinned) may be displayed using the dbMatchPlot proc.
ts> dbMatchPlot $statsTo plot to a postscript file, rather than the screen, use the "-file" option:
ts> dbMatchPlot $stats2 -file plots.psThe statistics can be printed to the screen using the dbMatchPrint proc. For example, to print the differences in positions:
ts> dbMatchPrint $stats pos
The same statistics can also be generated for a Frames Pipeline Run which has already been through "setObjectStatus" by using the selfMergeQA task. For example, to generated the statistics for run 94, camCol 1:
ts> set run [pipeGet frames 94 1] ts> set stats [selfMergeQA $run -noBlends -nSigma 5] ts> objDel $runFor a full description of the statistics chain and routines to examine it, see here. When done with the stats chain, it may be destroyed with "taGenericChainDestroy":
ts> taGenericChainDestroy $stats
ts> runMerge 94 1 0 -authors adelmanThe rerun argument is optional if there is only one Frames Pipeline Run for the specified run and camCol. The "-authors" argument is required. A default value is used for the matching radius (see here for the default value, or type "help runMerge" in the "ts" shell). Use the "-matchRadius" argument to specify a different value (in arcsecs). Thus, to merge with a matching radius of 1.5 arcsecs:
ts> runMerge 94 1 -authors adelman -matchRadius 1.5The current astrometric calibration is used by default. A different astrometric calibration may be specified with the "-astroCalib" argument. DCR corrections are applied using cosmic colors, since a final photometric calibration can not be guaranteed to exist for the run at this time.
ts> set run1 [pipeGet frames 94 1 -rerun 0] ts> set run2 [pipeGet frames 125 2 -rerun 1] ts> set stats [runMergeQA $run1 $run2] ts> objDel $run1 ts> objDel $run2In "pipeGet", the rerun is optional if only one Frames Pipeline Run for that run and camCol have been stuffed. If the range of fields in the first run that intersects the second run is known ahead of time, it may be specified to speed the task:
ts> set stats [runMergeQA $run1 $run2 -noBlends -firstField $field1 \ -lastField $field2]By default, the current astrometric and photometric calibrations are used; the arguments "-astro1", "-photo1", "-astro2", and "-photo2" may be used to specify different calibrations, useful for comparing various calibrations.
runMergeQA returns a chain of TSMATCHSTATS statistics, one entry for each field in the first run, based on the pairs of objects matched between the two runs. By default, matching statistics are compiled for objects in a default magnitude range in a default filter: these values may be changed using the arguments "-min", "-max", and "-filter". For example, to examine matched pairs where the object in run 1 in the pair is between 17th and 19th magnitude in g':
ts> set stats [runMergeQA $run1 $run2 -min 17 -max 19 -filter g]The cut is based one PSF magnitudes for objects of class STAR and UNK, and petrodsian magnitudes for objects of class GALAXY; the same magitude may be used for objects regardless of class using the "-psf", "-petro", or "-model" flags.
The statistics are for OK_RUN objects of class STAR, GALAXY, or UNK only. They can be further limited by specifying sets of flags that an object must have and must not have, with a separate set of flags for each run. For example, to limit the selection to objects in run1 which have the BLEND and CHILD flags set, but don't have the EDGE flag set, matching objects in run2 which have the EDGE flag:
ts> set stats [runMergeQA $run1 $run2 -flags1On {BLEND CHILD} \ -flags1Off EDGE -flags2On EDGE]The flags are from the "objc_flags". The flags may be taken from a specific filter's flag using the "-flagsFilter" option. For example, to repeat the above query but using the g' flags:
ts> set stats [runMergeQA $run1 $run2 -flags1On {BLEND CHILD} \ -flags1Off EDGE -flags2On EDGE -flagsFilter g]
Chains containing each matched pair may be returned by specifying the "-obj" option. One chain will contain all star-star pairs, one all galaxy-galaxy pairs, and one all other pairs (unknown-unknown pairs and mismatches). If the "-full" option is all specified, then the pair chains will contain the full calibrated objects, and an two additional chains containing field information for each run will also be returned (the "-full" option slows things down and increases memory usage by about a factor of ten, since the full objects need to be read). For example, to repeat the above query but also return the full objects for every matched pair:ts> set results [runMergeQA $run1 $run2 -flags1On {BLEND CHILD} \ -flags1Off EDGE -flags2On EDGE -flagsFilter g -obj -full] ts> set stats [lindex $results 0] ts> set stars [lindex $results 1] ts> set galaxies [lindex $results 2] ts> set others [lindex $results 3] ts> set fields1 [lindex $results 4] ts> set fields2 [lindex $results 5]If a second run isn't specified, then only the non-matching statistics are calculated. If, in addition, the "-full" option is specified, then the calibrated objects meeting the selection criterion are returned in three TA_CALIB_OBJ chains, one each for objects of class STAR, GALAXY, and UNK. For example, to fetch all CHILD objects from a run which are not EDGE:
ts> set results [runMergeQA $run1 -flags1On CHILD -flags2Off EDGE -full] ts> set stats [lindex $results 0] ts> set stars [lindex $results 1] ts> set galaxies [lindex $results 2] ts> set unknowns [lindex $results 3] ts> set fields [lindex $results 4]The statistics are on a per field basis. They may be binned over fields to increase the number of pairs used for each set of statistics using the mergeQABin proc. This proc returns a TSMATCHSTATS chain. For example, to bin over 20 fields:
ts> set stats2 [mergeQABin $stats 20]Plots based on the returned statistics chain may be displayed using the dbMatchPlot proc.ts> dbMatchPlot $stats -calib -logThe "-calib" arguments tells the proc that calibrated quantities are being plotted, rather than instrumental quantities. The "-log" argument causes counts to be plotted as log counts. To plot to a postscript file, rather than the screen, use the "-file" option:ts> dbMatchPlot $stats -calib -log -file plots.psA TCL proc is provided to create plots for all overlapping columns between two runs. For example, to do so for runs 1140 and 1231:ts> runMatchPlot 1140 1231 30 mag -file all.ps -calibThe statistics can be printed to the screen using the dbMatchPrint proc. For example, to print the differences in positions:ts> dbMatchPrint $stats posFor a full description of the statistics chain and routines to examine it, see here.runMergeQA can also return chains containing each matched pair, with a separate chain for stars and galaxies. Use the "-obj" flag for this feature:
ts> set list [runMergeQA $run1 $run2 -obj] ts> set stats [lindex $list 0] ts> set stars [lindex $list 1] ts> set galaxies [lindex $list 2]For each matched pair, the great circle longitude and latitude (mu and nu, in degrees), the magnitude in each filter (PSF magnitude for stars, petrosian magnitude for galaxies), and the model colors are given for the object in the first run, along with the differences in those quantities between the two runs (in the sense of run 1 quantity minus run 2 quantity, in arcsecs for dmu and dnu, mag for dmag and). dcolor). The photo flags for each object in the pair are also given. For example, to plot the difference in mu (in arcsecs) versus mu (in degrees) for stars:ts> set pg [pgstateNew] ts> pgstateOpen $pg ts> ePlot $pg $stars.mu $stars.dmuIf the "-full" flag is specified, then both entire calibrated objects are also included:ts> set list [runMergeQA $run1 $run2 -obj -full] ts> set stats [lindex $list 0] ts> set stars [lindex $list 1] ts> set galaxies [lindex $list 2]For example, to plot the difference in row velocity versus r' petrosian magnitude for galaxies:ts> ePlot $pg $galaxies.mag<2> \ $galaxies.obj1->rowv.val-$galaxies.obj2->rowv.valThe "-full" flag slows things down, as it must read a much bigger chunk of the OPDB to fetch the full objects. When done with the stats chain, it may be destroyed with "taGenericChainDestroy":ts> taGenericChainDestroy $statsThe matched-pairs chains must be destroyed with "matchedPairChainDel":ts> matchedPairChainDel $stars ts> matchedPairChainDel $galaxiesAn alternate way to examine match-ups between Frames Pipeline Runs is to use the GUI. Select one Frames Pipeline Run with the left mouse button. Then select another with the right mouse button. The matching statistics will be displayed using the GUI's tool.Quality Assignment
Once quality analysis is finished on a segment of a Frames Pipeline Run, a quality (undetermined, bad, acceptable, good) may be assigned. Quality is assigned on a field by field basis. The proc to set quality, fieldQualitySet, assigns the same quality to a contiguous range of fields in a single Frames Pipeline Run. For example, to set fields 23-248, inclusive, in run 94, camCol 1, rerun 0 to a quality of "good":ts> fieldQualitySet 94 1 good 23 248 -rerun 0The "-rerun" argument is required only if there is more than one Frames Pipeline Run stuffed for that run and camCol. If a starting and ending field aren't specified, then all fields in the Frames Pipeline Run are set to that quality. For example, to set all fields to "bad" for run 125, camCol 3:ts> fieldQualitySet 125 3 badTo set fields 244 to the end of the run to acceptable for run 125, camCol 3:ts> fieldQualitySet 125 4 acceptable 244Resolve
"Resolve" specifies which objects are primaries and which secondaries in a given chunk of sky (for a complete description, see here). OPDB maintains two versions of the sky, the version used to drill plates, and the current (latest-and-greatest) version. The drilling version is assigned the number 0 and the current version the number 1. When you resolve you must specify which version of the sky you're resolving against.Resolving is a two stage process. The first stage selects the chunks to be resolved using a GUI. The actual resolving is not performed at this stage. Rather, scripts files are created which may be submitted outside of the GUI to do the actual resolving, which comprises the second stage. The GUI takes one argument, namely the version of the sky to work on. It can also be run in "readonly" mode, which it should be for this purpose. Thus, to launch the GUI on the drilling version of the sky:
sdssdp4> ts -useTk ts> tsGui 0 -readonlyThe GUI should now display all segments, color coded as indicated by the legend on the left. The southern stripes are displayed first, then the northern stripes. "Primary", "secondary", and "bad" segments have already been resolved. "Unresolved ok" and "unresolved bad" segments are unresolved pieces of Frames Pipeline Runs which have had a quality assigned ("unresolved bad" corresponding to quality "bad", and "unresolved ok" corresponding to quality "good" or "acceptable"), and which have been processed through the setObjectStatus and merge tasks. "Undetermined" segments are unresolved pieces of Frames Pipeline Runs which have been processed through setObjectStatus and merge, but have not had a quality assigned. "Unprocessed" segments are unresolved pieces of Frames Pipeline Runs which have not been processed through either or both setObjectStatus and merge. Only segments which have been processed through both setObjectStatus and merge, which have had a quality assigned (bad, acceptable, or good), and which have not previously been resolved are eligible to be resolved (that is, only "unresolved ok" and "unresolved bad" segments).Many tasks not directly related to resolving are available from the GUI, and are briefly summarized in this paragraph. Clicking with the middle mouse button on a segment will list identifying information on it in the lower left panel. Clicking with the left mouse button on a segment "selects" it (indicated by it's border changing colors). The buttons on the left perform the tasks listed by clicking with the left mouse button (additional tasks are available via the "Actions" menu). Clicking the "set object status" button will process the selected segment through the setObjectStatus task (using the default matching radius). Clicking the "merge" button will process the selected segment through the merge task (using the default matching radius). Clicking on a second segment with the right mouse button will generate matching statistics between it and the selected segment, and display those statistics with the GUI's QA plotting tool (as well as give the user the option to set the quality over ranges of fields for the selected segment). The plot may be manipulated using the "QA" menu. Many of the tasks required that the "readonly" mode not be used, however these tasks will generally be performed without the GUI.
To resolve, we must first display those regions of the sky which are ready to resolve. A resolvable piece of sky is a continuous range in survey longitude of a stripe which is completely covered by "unresolved ok" segments. Press the "plot chunks-to-be" button with the left mouse button. A purple rectangle will be plotted delimiting each piece of a stripe which can be resolved. Click the border on one of the purple boxes. The border will turn gold, indicating you've selected it. Then press the "resolve" button with the left mouse button. You will be prompted for a list of runs to use for the primary segments. You may specify a list of runs to use. For example, if you wish to use runs 259 and 273 as the primary segments, then entery "259 273". If nothing is entered, then the lowest numbered runs will be used as the primary segments. That is, for a given scanline, the good segment with the lowest run number will be used as the primary segment over its full extent, with the rest of the scanline filled by the good segment with the next lowest run number and so on. No actual resolving will be performed. Rather, the file "$opdb_root/targetChunk/stripe$stripe_mu$startMu/resolve/resolve.tcl" will be created. Additional chunks-to-be may be selected and similar files created with one GUI session. To finish the resolving, exit the GUI and execute each script. For example, say you are resolving the chunk for stripe 10 starting at mu 45827 (arcsecs). After "resolving" the chunk in the GUI, exit the GUI and:
sdssdp4> cd $opdb_root/targetChunk/stripe10_mu45827/resolve sdssdp4> ts -noTk -file resolve.tclThe chunk will now be resolved. This takes quite a while; progress will be indicated in the xterm in which each field will be printed as it is resolved.You can get some feedback on whether things work by using fieldPlot. This is primarily a debugging tool to visualize whether the object status flags were properly set. It plots up GOOD objects in a set of contiguous fields (spans of three fields are nice). For example:
ts> set skyVersion 0 ts> set run [pipeGet frames 745 5] ts> fieldPlot $run $skyVersion -first 142 -last 145 -resThe meaning of the various symbols and lines is given in the task documentation.After a chunk of sky has been resolved, there may be additional scans which go through the same chunk of sky. We would like to be able to resolve this additional data, setting all objects in the portion overlapping the previously resolved chunk to SECONDARY (since PRIMARIES have already been selected in that piece of sky). When this situation arises, the following steps may be performed. First, fetch a list of all segments (both resolved and unresolved), using the proc segmentsGet:
ts> set segments [segmentsGet $skyVersion]Next, call the proc repeatSegmentsGet, which will parse the list of segments and return a list of all segments which have not been resolved, have an acceptable survey quality, and are completely overlapped by previously resolved primary segments:ts> set repeats [repeatSegmentsGet $segments $skyVersion]A TCL list of segments is returned, where each element in the list is itself a 5-element TCL list, consisting of the run number, camera column, rerun number, starting great circle longitude (in integer arcsecs), and ending great circle longitude (in integer arcsecs) of the segment. If desired, one can remove segments from the list that you don't wish to resolve (using the TCL command "lreplace"). Finally, resolve that set of repeat segments using the repeatResolve proc:ts> repeatResolve $repeats $skyVersion yannyAutomatic Main Survey Target Selection
Automatic target selection in the OPDB is performed almost exactly like its done on flatfiles. Set up your plan (tsPlan.par) and tunable parameter files. When running out of the OPDB, there are a few differences in keywords in the plan file. For a description of the differences, see the data model definition of the file here. You must also specify which version of the sky you're selecting targets from (0=drilling, 1=current). Once you've set up the input files, then just issue the command dbTargetSelect. For example, to select target in the current version of the sky:ts> set skyVersion 1 ts> dbTargetSelect tsPlan.par $skyVersion {adelman yanny}The first argument is the name of the plan file, and the second argument the list of folks who are running target selection. If this chunk has already been run through target selection, you may force it to be rerun through target selection again using the "-force" argument (all effects of the previous target selection will be undone):ts> dbTargetSelect tsPlan.par $skyVersion {adelman yanny} -forceThe "target" product provides a routine, "taDiagnostics", to generate various QA files after target selection runs off of flatfiles. A simple TCL proc, tsDiagnostics, is provided to provide the same functionality out of the OPDB. It only works if flatfiles were exported when target selection ran. It simply generates a "tsChunk" file and then calls "taDiagnostics" on the flatfiles. Just run the routine on the same plan file used to run target selection:ts> tsDiagnostics tsPlan.parYou can optionally specify a directory for the output files. For example, to create all output files in the directory "output":ts> tsDiagnostics tsPlan.par outputManual Target Selection
Manual target selection occurs after automatic target selection has been performed on a chunk, and the chunk is exported to the Science Database. A fixed period of time is allotted during which people may manually select additional spectroscopic targets for that chunk. After the time period has elapsed, these targets are collected in a single tsManual file. To enter these manually selected targets into the OPDB (necessary to include them in tile and plate creation), issue the command manualSelect with the "tsManual" file:ts> manualSelect tsManual.parSouthern Survey Target Selection
On the southern equatorial stripe, we will select targets multiple times. The first time targets are selected using the main surey target selection criteria, which is performed the same as in the rest of the survey in the drilling (0) version of the sky. When one wishes to repeat target selection on the southern equatorial stripe, using other than the main target selection criteria, you do it in the current (1) version of the sky. Thus, if necessary, first resolve a portion of the southern stripe in the current version of the sky. You then copy that resolved chunk to a new and unique version of the sky. For example, the first time the target selection is performed with the southern survey criteria you would copy the chunk to version 2 of the sky. The next time version 3, etc. Thus, lets assume we have the southern equatorial stripe (stripe 76) resolved as a single chunk in the current version of the sky, and this is the first time we've run the southern survey target selection:ts> set current [queryGetOne arChunk $photo_segments \ "stripe_ == 76 && skyVersion_ == 1"] ts> set skyVersion 2 ts> set southernChunk [chunkCopyToSouthernSurvey $current $skyVersion]Now run target selection as you normally do, but specify the the southern survey sky version:ts> dbTargetSelect tsPlan.par $skyVersion {adelman yanny}Southern survey target selection differs from main survey target selection in the following ways:The sky version number must be unique each time you run target selection for the southern survey, even if you are running it on non-overlapping chunks that you consider to be parts of the same sky.
- For each object selected as a target, all matching objects in the OPDB are searched to see if they were previously selected as targets. If any matching objects were previously selected as targets, then the target selection flags and priority from this run are appended to a list of such information for that matching target, but a new target object is not created. If none of the matching objects had previously been targetted, then a new spectroscopic target is created.
- The targetting flags and priority are set on the spectroscopic target itself, not on the photo object. This differs from running the main survey target selection on either the drilling or current version of the sky, where the flags and priority are set in the photo object. Thus, selecting targets for the southern survey will not overwrite any of the main survey target selection information.
- If the current version of the sky is re-resolved, you will loose the object-by-object resolve information for any southern survey chunks which were copied from the current version chunk. You will still know what the chunk and segment boundaries were for each southern survey chunk, but the "status" flag for each object will not be set for the southern survey chunks.
Tile Preprocessor
The "tile" pipeline requires as input a tiInputTargets file, which lists all untiled tilable spectroscopic targets. This file may be generated using the command untiledToFits:ts> untiledToFitsThe file, named "tiInputTargets.par", will be created in the current directory. Optionally, the output directory for the file may be specified. For example, to create the "tiInputTargets.par" file in the directory "output":ts> untiledToFits outputThe "tile" postprocessor is described in the "photoSch" product.Complete Tile Preprocessor
The "completeTile" pipeline requires as input a list of all tiles not yet processed through "completeTile" (tiTileCenters.par), and a list of all spectropic targets, whether already assigned to a tile or not (ctObjs.fit). These files may be generated using the completeTilePre command:ts> completeTilePreThe files, named "tiTileCenters.par" and "ctObjs.fit", are created in the current directory. Optionally, the output directory for the file may be specified. For example, to create the files in the directory "/data/dp1.9/data/completeTile":ts> completeTilePre /data/dp1.9/data/completeTileThe "completeTile" postprocessor is described in the "photoSch" product.Undoing
This section describes how to undo, or back out of the OPDB, the various tasks involved in target selection.Undo Set Object Status
setObjectStatus may be undone with no side-effects, so long as the task resolve has not been run on any portion of the Frames Pipeline Run. To undo the operation, use the proc unsetObjectStatus. For example, to undo "setObjectStatus" on run 94, camCol 1:ts> set run [pipeGet frames 94 1] ts> unsetObjectStatus $run ts> objDel $runUndo Merge Objects
runMerge may be undone with no side-effects, so long as the task resolve has not been run on any portion of the Frames Pipeline Run. To undo the operation, use the proc runUnmerge. For example, to undo "runMerge" on run 94, camCol 1:ts> set run [pipeGet frames 94 1] ts> runUnmerge $run ts> objDel $runUndo Set Quality Assignment
fieldQualitySet may be undone with no side-effects, so long as the task resolve has not been run on any of the object lists corresponsing to the fields whose qualities you are unsetting. To undo the operation, use the same fieldQualitySet proc, simply setting the qualities to "undetermined". For example, to undo "fieldQualitySet" on run 94, camCol 1, fields 120-248:ts> fieldQualitySet 94 1 undetermined 120 248Undo Resolve
resolve may be undone only if the chunk has not been processed through Automatic Target Selection. Chunks are identified by their stripe number and the starting great circle longitude (mu), expressed in an integer number of arcsecs. To undo the "resolve" operation, use the proc chunkUnresolve. For example, to unresolve (and thus destroy) the chunk starting at mu=65378 arcsecs for stripe 15:ts> set chunk [queryGetOne arChunk $photo_segments \ "stripe_ == 15 && startMu_ == 65378"] ts> chunkUnresolve $chunkUndo Main Survey Target Selection
Target selection, both automatic and manual, on a chunk may be undone only if none of the targets selected have been assigned to a tile. To undo target selection, use the proc dbTargetUnselectFromChunk. Both automatically and manually selected targets will be de-selected. For example, to undo target selection for the chunk starting at mu=65378 arcsecs for stripe 15 in the current version of the sky:ts> set chunk [queryGetOne arChunk $photo_segments \ "stripe_ == 15 && startMu_ == 65378 && skyVersion_ == 1"] ts> dbTargetUnselectFromChunk $chunk ts> objDel $chunkUndo Southern Survey Target Selection
This operation is not yet supported. Contact Jeff Munn if it needs doing.Undo Tiling
To remove a single tile from the OPDB:ts> set tileId 259 ts> untile $tileIdThe tile must not have any plates attached to it. To remove all tiles from a given tiling run, specify the "tiTileCenters" file generated by that tiling run to the command allUntile:ts> allUntile tiTileCenters-0259-0287.par