Update

Given a plate/mjd/fiber query list. This update command will update the base hdf5 accordingly. Basically, update will
  • Compare the existing plate/mjd/fibers in the base hdf5 file with this query,
  • Search the new plate/mjd/fiber, which are not existed in the base hdf5 file, from the input hdf5 files
  • Add the founded plate/mjd/fiber into the base file
  • Remove the existing plate/mjd/fiber from the base file, which are not found in the query list.
  • [optional] Repacking the final output to make the file contiguous on storage

Usage:

update -h

usage:

update [-h] base input pmf

positional arguments:

base        Pre-subseted HDF5 file

input       HDF5 input files list

pmf         Plate/mjd/fiber list

optional arguments:

-h, --help  show this help message and exit
--repack, REPACK  repack after changing the file, yes or no

Example:

Prepare input:

cat input_sample.txt

 /global/cscratch1/sd/jialin/h5boss/3665-55247.hdf5

 ...

cat pmf_update.txt

 plates mjds fibers

 6697 56419 796

 4697 55660 190

 4191 55444 636

 ...

Download: input_sample.txt, pmf_update.txt, base.h5

Execute command:

update base.h5 input_sample.txt pmf_update.txt  --repack=yes

Output:

Will repack the file for better storage layout afterwards.
Query: Plates/Mjds/Fibers: 11
Input: 2393 hdf5 files
Output: base.h5
Running Updating:
Query time: 0.00 seconds
plates/mjds/fibers to be added: 10
Running selection:
-Source file open: 1.93
-Fiber query time: 0.00
-Fiber copy time: 0.20
-Catalog copy time: 0.83
-Group create time: 0.00
-File close time: 0.04
Selection Time: 5.15 seconds
plates/mjds/fibers to be removed: 6
Running removing:
Removed 6 plates/mjds/fibers,Skipped 0
Remvoing Time: 0.02 seconds
Running repacking:
Repacking Time: 2.33 seconds
Updating complete: 8.40 seconds