Flatfile Based OPDB: Best Sky Version, Imaging Only

Initialize The Flatfile Based OPDB from the Objectivity Based OPDB

In transferring from an Objectivity based OPDB to a flatfile based OPDB, you may need to transfer information from the OBjectivity based OPDB to the flatfile based OPDB. This should need be done only once. The only information that needs to be transferred is 1) the list of all run/rerun combos in the database, and 2) the per-field quality information for each run/rerun. This new information is stored in two types of files:

/sdss/data/bestRoot/runs.par: This is an ASCII parameter file listing every run/rerun in the database, along with the stripe number and strip.
/sdss/data/$run/$rerun/q-$run-$rerun.par: There is one such file per run/rerun, which stores the quality information (quality, culled, bad/acceptable/good/holeMask) for each field in the run.

Since we're creating the best version of the sky, we will typically want to restrict this to reruns equal to or greater than some specific value. Two create these flatfiles from the Objectivity based OPDB, run the following comands (in this example, we want only rerun 40 or greater):

sdssdp25> setup ts
sdssdp25> ts
ts> sessionBegin -mrow -readonly
ts> photoInit -readonly -notargets
ts> set skyVersion 1
ts> set minRerun 40
ts> runsDbMakeFromOPDB $skyVersion $minRerun
ts> sessionEnd
ts> quit

Automatic Quality Set Pipeline

In the Objectivity based OPDB, the field quality was set using the "dp"
product's command "opdbQualitySet".   To set quality using the flatfile based
OPDB, use the same command, but add the flag " -flatFiles".

Manual Quality Setting

In the Objectivity based OPDB, the field quality set by the automatic pipeline
could be overrode using the "dp" product's command "fieldQualityOverride".
To override the automatically set quality using the flatfile based OPDB,
use the same command, but add the flag " -flatFiles".

Stuffing

Stuffing a run/rerun now means simply adding it to the
"/sdss/data/bestRoot/runs.par" file.  Use the "ts" command "runStuff":
sdssdp25> setup ts
sdssdp25> ts
ts> set skyVersion 1
ts> set newRun 2131
ts> set newRerun 40
ts> runStuff $skyVersion $newRun $newRerun
ts> quit


Resolving

Resolving a chunk now strictly creates the "tsChunk" file.  The actual
object-by-object resolution occurs during target selection/export.

Running the GUI

To run "tsGui" using the flatfile based OPDB:
sdssdp25> setup ts
sdssdp25> ts -useTk
ts> set skyVersion 1
ts> tsGui $skyVersion -flatFiles

The " -readonly" option has no effect when using flatfiles;  you are always
in read only mode, as the only files you can modify are creating a new
"tsChunk"' file.  You cannot plot run-to-run QA or set field quality; those
options are still available, but they won't work, and will probably crash
the GUI, so don't try them.

Most other things should work.  Specifically, it plots segments correctly
and resolves potential chunks correctly.  When you've selected a chunk-to-be
and clicked "resolve", it creates a "tsChunk" file for that chunk in the
directory you started the GUI in;  it does nothing beyond creating
the "tsChunk" file.

Manually Resolving

Just as with the Objectivity based OPDB, you can resolve manually in the
flatfile based OPDB.
sdssdp25> setup ts
sdssdp25> ts
ts> set skyVersion 1
ts> set segs [fsegmentsGet $skyVersion]
ts> set chunks [resolvableChunksGet $segs]
ts> foreach chunk $chunks {
>>      set stripe [lindex $chunk 0]
>>      set startMu [lindex $chunk 1]
>>      set endMu [lindex $chunk 2]
>>      fresolve $segs $stripe $startMu $endMu $skyVersion
>>  }

This will create a "tsChunk" file for each resolvable chunk in the current
directory.  Note that the old commands "segmentsGet" and "resolve" now have
a prefix of "f" for the flatfile based versions of these commands.

Calibrate, Target Selection, and Exporting

Similarly to the Objectivity based OPDB, you calibrate, run target
selection, and export in a single pass, now reading the raw pipeline flatfile
outputs rather than the Objectivity database.  To do so, set up a
target selection plan file that looks like:
chunkFile   tsChunk-42-750191.par
inputDir   
outputDir   /sdss/data/bestRoot/stripe42_mu750191_1/%d
parametersDir   .
quasarsParameters   tsQuasarsParams.par
galaxiesParameters   tsGalaxiesParams.par
starsParameters   tsStarsParams.par
serendipityParameters   tsSerendipityParams.par
rosatParameters   tsRosatParams.par
standardsParameters   tsStandardsParams.par
matchParameters   tsMatchParams.par
diagParameters   tsDiagParams.par
firstCatalog   /opdb/d1/data/first/first.03apr11.stripe42.fit
rosatCatalog   /opdb/d1/data/rosat/rass-fsc-bsc-19jul00id.fit
usnoDir            /opdb/d1/data/usnob
diagDir            /data/dp12.a/data/munn/bestRoot/stripe42_mu750191_1
calibrate   1
exportDir   /sdss/data/bestRoot

The plan file is essentially that when running target selection off of
"tsObj" files on disk, except for two new keywords.   "calibrate" needs
to be set to 1, indicating that we are reading raw pipeline outputs.
"exportDir" needs to be set to the root directory for exported chunks;  if
not specified, then no border resolution will be performed between chunks.
Setting "inputDir" to be blank will cause it to look at the default locations
at FNAL for the pipeline outputs.   See the Data Model for additional
optional keywords (for instance, if working outside the FNAL file structure).

Target selection can be run in parallel (that is, different chunks on different
processors), except for adjacent chunks.  This is because of border resolution
between chunks.  For similar reasons, you can't parallel process within chunks
(such as different scanlines or different fields at the same time).