Started working on galaxy PGC027949 at Sat Feb  8 08:04:00 2020
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 146.31578100000002 69.3902517 --width 1069 --height 1069 --pixscale 0.262 --threads 8 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC027949 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run 90prime-mosaic --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC027949/PGC027949-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC027949/PGC027949-runbrick-%%(stage)s.p --unwise-coadds 

runbrick.py starting at 2020-02-08T08:04:00.856542
legacypipe git version: dr9e-pre1-7-g54c974a3
Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '146.31578100000002', '69.3902517', '--width', '1069', '--height', '1069', '--pixscale', '0.262', '--threads', '8', '--outdir', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC027949', '--survey-dir', '/global/cscratch1/sd/desimpp/dr9e', '--run', '90prime-mosaic', '--largegalaxy-preburner', '--write-stage', 'tims', '--write-stage', 'srcs', '--skip-calibs', '--checkpoint', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC027949/PGC027949-runbrick-checkpoint.p', '--pickle', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC027949/PGC027949-runbrick-%%(stage)s.p', '--unwise-coadds']
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 146.31578100000002 69.3902517 --width 1069 --height 1069 --pixscale 0.262 --threads 8 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC027949 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run 90prime-mosaic --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC027949/PGC027949-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC027949/PGC027949-runbrick-%%(stage)s.p --unwise-coadds

WARNING: no WISE sky background maps in /global/cscratch1/sd/desimpp/dr9e/calib/wise/modelsky
Parsed RA,Dec 146.31578100000002 69.3902517
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/home/legacyhalos/.astropy/cache/download/py3/lock' [astropy.utils.data]
Starting process 41407 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071168 MB
Starting process 41408 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071168 MB
Starting process 41409 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071168 MB
Starting process 41410 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071168 MB
Starting process 41411 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071168 MB
Starting process 41412 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB
Starting process 41413 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071176 MB
Starting process 41414 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07118 MB
Runstage writecat
Runstage wise_forced
Runstage coadds
Runstage fitblobs
Runstage srcs
Runstage largegalaxies
Runstage halos
Runstage outliers
Runstage refs
Runstage tims
Running stage tims
Running stage tims at 2020-02-08T08:04:02.151050
Keeping 15 CCDs:
   ksb_190112_101522_ooi_g_ls9.fits.fz ksb_190112_101522_ooi_g_ls9-CCD1 g exptime 124.63 propid NOTBASS    seeing 1.54 MJD 58495.427 object 
   ksb_190128_105804_ooi_r_ls9.fits.fz ksb_190128_105804_ooi_r_ls9-CCD1 r exptime 163.914 propid NOTBASS    seeing 1.39 MJD 58511.457 object 
   ksb_180411_072612_ooi_g_ls9.fits.fz ksb_180411_072612_ooi_g_ls9-CCD1 g exptime 215.859 propid BASS       seeing 1.54 MJD 58219.310 object 45082
   ksb_190131_114353_ooi_r_ls9.fits.fz ksb_190131_114353_ooi_r_ls9-CCD1 r exptime 139.363 propid NOTBASS    seeing 1.54 MJD 58514.489 object 
   ksb_190128_062918_ooi_g_ls9.fits.fz ksb_190128_062918_ooi_g_ls9-CCD2 g exptime 118.299 propid NOTBASS    seeing 1.36 MJD 58511.270 object 
   ksb_190209_093432_ooi_r_ls9.fits.fz ksb_190209_093432_ooi_r_ls9-CCD2 r exptime 140.004 propid NOTBASS    seeing 1.48 MJD 58523.399 object 
   ksb_180411_072612_ooi_g_ls9.fits.fz ksb_180411_072612_ooi_g_ls9-CCD3 g exptime 215.859 propid BASS       seeing 1.56 MJD 58219.310 object 45082
   ksb_190131_114353_ooi_r_ls9.fits.fz ksb_190131_114353_ooi_r_ls9-CCD3 r exptime 139.363 propid NOTBASS    seeing 1.46 MJD 58514.489 object 
   k4m_170520_050940_ooi_zd_ls9.fits.fz k4m_170520_050940_ooi_zd_ls9-CCD1 z exptime 249.105 propid 2016A-0453 seeing 1.20 MJD 57893.215 object MzLS_713103_z
   k4m_170208_080820_ooi_zd_ls9.fits.fz k4m_170208_080820_ooi_zd_ls9-CCD2 z exptime 124.104 propid 2016A-0453 seeing 1.14 MJD 57792.339 object MzLS_711764_z
   k4m_170208_080820_ooi_zd_ls9.fits.fz k4m_170208_080820_ooi_zd_ls9-CCD4 z exptime 124.104 propid 2016A-0453 seeing 1.17 MJD 57792.339 object MzLS_711764_z
   k4m_160125_093024_ooi_zd_ls9.fits.fz k4m_160125_093024_ooi_zd_ls9-CCD2 z exptime 127.103 propid 2016B-0453 seeing 1.00 MJD 57412.396 object MzLS_712215_z
   k4m_171028_104349_ooi_zd_ls9.fits.fz k4m_171028_104349_ooi_zd_ls9-CCD1 z exptime 158.103 propid 2016A-0453 seeing 1.02 MJD 58054.447 object MzLS_712661_z
   k4m_170520_050940_ooi_zd_ls9.fits.fz k4m_170520_050940_ooi_zd_ls9-CCD3 z exptime 249.105 propid 2016A-0453 seeing 1.09 MJD 57893.215 object MzLS_713103_z
   k4m_160125_092224_ooi_zd_ls9.fits.fz k4m_160125_092224_ooi_zd_ls9-CCD4 z exptime 123.105 propid 2016B-0453 seeing 1.04 MJD 57412.391 object MzLS_712214_z
Faked up EXPNUM 85140227
Faked up EXPNUM 85140227
Faked up EXPNUM 85140227
90prime get_fwhm: self.fwhm = 3.39
WARNING: EXPNUM None!=160125093024 in /global/cscratch1/sd/desimpp/dr9e/images/mosaic/CP/V4.3/CP20160124/k4m_160125_093024_ooi_zd_ls9.fits.fz header
WARNING: EXPNUM None!=160125092224 in /global/cscratch1/sd/desimpp/dr9e/images/mosaic/CP/V4.3/CP20160124/k4m_160125_092224_ooi_zd_ls9.fits.fz header
Faked up EXPNUM 82190101
Faked up EXPNUM 82190101
Faked up EXPNUM 82190101
90prime get_fwhm: self.fwhm = 3.39
Faked up EXPNUM 85110142
Faked up EXPNUM 85110142
Faked up EXPNUM 85110142
90prime get_fwhm: self.fwhm = 2.99
Faked up EXPNUM 85230158
Faked up EXPNUM 85230158
Faked up EXPNUM 85230158
90prime get_fwhm: self.fwhm = 3.27
Faked up EXPNUM 84950057
Faked up EXPNUM 84950057
Faked up EXPNUM 84950057
90prime get_fwhm: self.fwhm = 3.39
mosaic get_fwhm: self.fwhm = 4.19
Swapping in SKY_JOHN values for 1 splinesky cells; 109.86305 -> 109.832535
Traceback (most recent call last):
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3605, in <module>
    sys.exit(main())
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3575, in main
    run_brick(opt.brick, survey, **kwargs)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3161, in run_brick
    initial_args=initargs, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  [Previous line repeated 6 more times]
  File "/usr/local/lib/python/astrometry/util/stages.py", line 124, in runstage
    R = stagefunc(stage, **Px)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3150, in mystagefunc
    R = stagefunc(stage, mp=mp, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 40, in __call__
    rtn = super(CallGlobalTime, self).__call__(stage, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 32, in __call__
    return func(*self.args, **kwa)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 271, in stage_tims
    tims = list(mp.map(read_one_tim, args))
  File "/usr/local/lib/python/astrometry/util/multiproc.py", line 89, in map
    return self.pool.map(f, args, cs)
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 268, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 657, in get
    raise self._value
AssertionError
Something went wrong; please check the logfile.
ERROR: galaxy PGC027949; please check the logfile.
Finished galaxy PGC027949 in 0.084 minutes.
Started working on galaxy PGC027949 at Sat Feb  8 08:39:56 2020
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 146.31578100000002 69.3902517 --width 1069 --height 1069 --pixscale 0.262 --threads 8 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC027949 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run 90prime-mosaic --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC027949/PGC027949-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC027949/PGC027949-runbrick-%%(stage)s.p --unwise-coadds 

runbrick.py starting at 2020-02-08T08:39:56.619727
legacypipe git version: dr9e-pre1-7-g54c974a3
Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '146.31578100000002', '69.3902517', '--width', '1069', '--height', '1069', '--pixscale', '0.262', '--threads', '8', '--outdir', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC027949', '--survey-dir', '/global/cscratch1/sd/desimpp/dr9e', '--run', '90prime-mosaic', '--largegalaxy-preburner', '--write-stage', 'tims', '--write-stage', 'srcs', '--skip-calibs', '--checkpoint', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC027949/PGC027949-runbrick-checkpoint.p', '--pickle', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC027949/PGC027949-runbrick-%%(stage)s.p', '--unwise-coadds']
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 146.31578100000002 69.3902517 --width 1069 --height 1069 --pixscale 0.262 --threads 8 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC027949 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run 90prime-mosaic --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC027949/PGC027949-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC027949/PGC027949-runbrick-%%(stage)s.p --unwise-coadds

WARNING: no WISE sky background maps in /global/cscratch1/sd/desimpp/dr9e/calib/wise/modelsky
Parsed RA,Dec 146.31578100000002 69.3902517
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/home/legacyhalos/.astropy/cache/download/py3/lock' [astropy.utils.data]
Starting process 1145 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07116 MB
Starting process 1146 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071164 MB
Starting process 1147 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071164 MB
Starting process 1148 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071164 MB
Starting process 1149 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071164 MB
Starting process 1150 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071168 MB
Starting process 1151 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB
Starting process 1152 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB
Runstage writecat
Runstage wise_forced
Runstage coadds
Runstage fitblobs
Runstage srcs
Runstage largegalaxies
Runstage halos
Runstage outliers
Runstage refs
Runstage tims
Running stage tims
Running stage tims at 2020-02-08T08:39:57.880368
Keeping 15 CCDs:
   ksb_190112_101522_ooi_g_ls9.fits.fz ksb_190112_101522_ooi_g_ls9-CCD1 g exptime 124.63 propid NOTBASS    seeing 1.54 MJD 58495.427 object 
   ksb_190128_105804_ooi_r_ls9.fits.fz ksb_190128_105804_ooi_r_ls9-CCD1 r exptime 163.914 propid NOTBASS    seeing 1.39 MJD 58511.457 object 
   ksb_180411_072612_ooi_g_ls9.fits.fz ksb_180411_072612_ooi_g_ls9-CCD1 g exptime 215.859 propid BASS       seeing 1.54 MJD 58219.310 object 45082
   ksb_190131_114353_ooi_r_ls9.fits.fz ksb_190131_114353_ooi_r_ls9-CCD1 r exptime 139.363 propid NOTBASS    seeing 1.54 MJD 58514.489 object 
   ksb_190128_062918_ooi_g_ls9.fits.fz ksb_190128_062918_ooi_g_ls9-CCD2 g exptime 118.299 propid NOTBASS    seeing 1.36 MJD 58511.270 object 
   ksb_190209_093432_ooi_r_ls9.fits.fz ksb_190209_093432_ooi_r_ls9-CCD2 r exptime 140.004 propid NOTBASS    seeing 1.48 MJD 58523.399 object 
   ksb_180411_072612_ooi_g_ls9.fits.fz ksb_180411_072612_ooi_g_ls9-CCD3 g exptime 215.859 propid BASS       seeing 1.56 MJD 58219.310 object 45082
   ksb_190131_114353_ooi_r_ls9.fits.fz ksb_190131_114353_ooi_r_ls9-CCD3 r exptime 139.363 propid NOTBASS    seeing 1.46 MJD 58514.489 object 
   k4m_170520_050940_ooi_zd_ls9.fits.fz k4m_170520_050940_ooi_zd_ls9-CCD1 z exptime 249.105 propid 2016A-0453 seeing 1.20 MJD 57893.215 object MzLS_713103_z
   k4m_170208_080820_ooi_zd_ls9.fits.fz k4m_170208_080820_ooi_zd_ls9-CCD2 z exptime 124.104 propid 2016A-0453 seeing 1.14 MJD 57792.339 object MzLS_711764_z
   k4m_170208_080820_ooi_zd_ls9.fits.fz k4m_170208_080820_ooi_zd_ls9-CCD4 z exptime 124.104 propid 2016A-0453 seeing 1.17 MJD 57792.339 object MzLS_711764_z
   k4m_160125_093024_ooi_zd_ls9.fits.fz k4m_160125_093024_ooi_zd_ls9-CCD2 z exptime 127.103 propid 2016B-0453 seeing 1.00 MJD 57412.396 object MzLS_712215_z
   k4m_171028_104349_ooi_zd_ls9.fits.fz k4m_171028_104349_ooi_zd_ls9-CCD1 z exptime 158.103 propid 2016A-0453 seeing 1.02 MJD 58054.447 object MzLS_712661_z
   k4m_170520_050940_ooi_zd_ls9.fits.fz k4m_170520_050940_ooi_zd_ls9-CCD3 z exptime 249.105 propid 2016A-0453 seeing 1.09 MJD 57893.215 object MzLS_713103_z
   k4m_160125_092224_ooi_zd_ls9.fits.fz k4m_160125_092224_ooi_zd_ls9-CCD4 z exptime 123.105 propid 2016B-0453 seeing 1.04 MJD 57412.391 object MzLS_712214_z
Faked up EXPNUM 85230158
Faked up EXPNUM 85230158
Faked up EXPNUM 85230158
90prime get_fwhm: self.fwhm = 3.27
Faked up EXPNUM 85110243
Faked up EXPNUM 85110243
Faked up EXPNUM 85110243
90prime get_fwhm: self.fwhm = 3.07
WARNING: EXPNUM None!=160125092224 in /global/cscratch1/sd/desimpp/dr9e/images/mosaic/CP/V4.3/CP20160124/k4m_160125_092224_ooi_zd_ls9.fits.fz header
Faked up EXPNUM 82190101
Faked up EXPNUM 82190101
Faked up EXPNUM 82190101
90prime get_fwhm: self.fwhm = 3.39
Faked up EXPNUM 85110142
Faked up EXPNUM 85110142
Faked up EXPNUM 85110142
90prime get_fwhm: self.fwhm = 2.99
Faked up EXPNUM 82190101
Faked up EXPNUM 82190101
Faked up EXPNUM 82190101
90prime get_fwhm: self.fwhm = 3.45
mosaic get_fwhm: self.fwhm = 4.63
Faked up EXPNUM 84950057
Faked up EXPNUM 84950057
Faked up EXPNUM 84950057
90prime get_fwhm: self.fwhm = 3.39
mosaic get_fwhm: self.fwhm = 4.51
Traceback (most recent call last):
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3605, in <module>
    sys.exit(main())
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3575, in main
    run_brick(opt.brick, survey, **kwargs)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3161, in run_brick
    initial_args=initargs, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  [Previous line repeated 6 more times]
  File "/usr/local/lib/python/astrometry/util/stages.py", line 124, in runstage
    R = stagefunc(stage, **Px)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3150, in mystagefunc
    R = stagefunc(stage, mp=mp, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 40, in __call__
    rtn = super(CallGlobalTime, self).__call__(stage, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 32, in __call__
    return func(*self.args, **kwa)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 271, in stage_tims
    tims = list(mp.map(read_one_tim, args))
  File "/usr/local/lib/python/astrometry/util/multiproc.py", line 89, in map
    return self.pool.map(f, args, cs)
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 268, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 657, in get
    raise self._value
AssertionError
Something went wrong; please check the logfile.
ERROR: galaxy PGC027949; please check the logfile.
Finished galaxy PGC027949 in 0.076 minutes.
Started working on galaxy PGC027949 at Sat Feb  8 12:44:31 2020
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 146.31578100000002 69.3902517 --width 1069 --height 1069 --pixscale 0.262 --threads 8 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC027949 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run 90prime-mosaic --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC027949/PGC027949-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC027949/PGC027949-runbrick-%%(stage)s.p --unwise-coadds 

runbrick.py starting at 2020-02-08T12:44:32.645052
legacypipe git version: dr9e-pre1-7-g54c974a3
Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '146.31578100000002', '69.3902517', '--width', '1069', '--height', '1069', '--pixscale', '0.262', '--threads', '8', '--outdir', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC027949', '--survey-dir', '/global/cscratch1/sd/desimpp/dr9e', '--run', '90prime-mosaic', '--largegalaxy-preburner', '--write-stage', 'tims', '--write-stage', 'srcs', '--skip-calibs', '--checkpoint', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC027949/PGC027949-runbrick-checkpoint.p', '--pickle', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC027949/PGC027949-runbrick-%%(stage)s.p', '--unwise-coadds']
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 146.31578100000002 69.3902517 --width 1069 --height 1069 --pixscale 0.262 --threads 8 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC027949 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run 90prime-mosaic --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC027949/PGC027949-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC027949/PGC027949-runbrick-%%(stage)s.p --unwise-coadds

WARNING: no WISE sky background maps in /global/cscratch1/sd/desimpp/dr9e/calib/wise/modelsky
Parsed RA,Dec 146.31578100000002 69.3902517
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/home/legacyhalos/.astropy/cache/download/py3/lock' [astropy.utils.data]
Starting process 58170 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 731 MB, VmSize: 731 MB, VmRSS: 70 MB, VmData: 143 MB, maxrss: 0.071176 MB
Starting process 58172 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 731 MB, VmSize: 731 MB, VmRSS: 70 MB, VmData: 143 MB, maxrss: 0.071176 MB
Starting process 58174 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 731 MB, VmSize: 731 MB, VmRSS: 70 MB, VmData: 143 MB, maxrss: 0.071176 MB
Starting process 58178 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 731 MB, VmSize: 731 MB, VmRSS: 70 MB, VmData: 143 MB, maxrss: 0.07118 MB
Starting process 58179 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 731 MB, VmSize: 731 MB, VmRSS: 70 MB, VmData: 143 MB, maxrss: 0.071184 MB
Starting process 58181 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 731 MB, VmSize: 731 MB, VmRSS: 70 MB, VmData: 143 MB, maxrss: 0.071184 MB
Starting process 58180 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 731 MB, VmSize: 731 MB, VmRSS: 70 MB, VmData: 143 MB, maxrss: 0.071184 MB
Starting process 58182 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 731 MB, VmSize: 731 MB, VmRSS: 70 MB, VmData: 143 MB, maxrss: 0.071188 MB
Runstage writecat
Runstage wise_forced
Runstage coadds
Runstage fitblobs
Runstage srcs
Runstage largegalaxies
Runstage halos
Runstage outliers
Runstage refs
Runstage tims
Running stage tims
Running stage tims at 2020-02-08T12:44:34.884756
Keeping 15 CCDs:
   ksb_190112_101522_ooi_g_ls9.fits.fz ksb_190112_101522_ooi_g_ls9-CCD1 g exptime 124.63 propid NOTBASS    seeing 1.54 MJD 58495.427 object 
   ksb_190128_105804_ooi_r_ls9.fits.fz ksb_190128_105804_ooi_r_ls9-CCD1 r exptime 163.914 propid NOTBASS    seeing 1.39 MJD 58511.457 object 
   ksb_180411_072612_ooi_g_ls9.fits.fz ksb_180411_072612_ooi_g_ls9-CCD1 g exptime 215.859 propid BASS       seeing 1.54 MJD 58219.310 object 45082
   ksb_190131_114353_ooi_r_ls9.fits.fz ksb_190131_114353_ooi_r_ls9-CCD1 r exptime 139.363 propid NOTBASS    seeing 1.54 MJD 58514.489 object 
   ksb_190128_062918_ooi_g_ls9.fits.fz ksb_190128_062918_ooi_g_ls9-CCD2 g exptime 118.299 propid NOTBASS    seeing 1.36 MJD 58511.270 object 
   ksb_190209_093432_ooi_r_ls9.fits.fz ksb_190209_093432_ooi_r_ls9-CCD2 r exptime 140.004 propid NOTBASS    seeing 1.48 MJD 58523.399 object 
   ksb_180411_072612_ooi_g_ls9.fits.fz ksb_180411_072612_ooi_g_ls9-CCD3 g exptime 215.859 propid BASS       seeing 1.56 MJD 58219.310 object 45082
   ksb_190131_114353_ooi_r_ls9.fits.fz ksb_190131_114353_ooi_r_ls9-CCD3 r exptime 139.363 propid NOTBASS    seeing 1.46 MJD 58514.489 object 
   k4m_170520_050940_ooi_zd_ls9.fits.fz k4m_170520_050940_ooi_zd_ls9-CCD1 z exptime 249.105 propid 2016A-0453 seeing 1.20 MJD 57893.215 object MzLS_713103_z
   k4m_170208_080820_ooi_zd_ls9.fits.fz k4m_170208_080820_ooi_zd_ls9-CCD2 z exptime 124.104 propid 2016A-0453 seeing 1.14 MJD 57792.339 object MzLS_711764_z
   k4m_170208_080820_ooi_zd_ls9.fits.fz k4m_170208_080820_ooi_zd_ls9-CCD4 z exptime 124.104 propid 2016A-0453 seeing 1.17 MJD 57792.339 object MzLS_711764_z
   k4m_160125_093024_ooi_zd_ls9.fits.fz k4m_160125_093024_ooi_zd_ls9-CCD2 z exptime 127.103 propid 2016B-0453 seeing 1.00 MJD 57412.396 object MzLS_712215_z
   k4m_171028_104349_ooi_zd_ls9.fits.fz k4m_171028_104349_ooi_zd_ls9-CCD1 z exptime 158.103 propid 2016A-0453 seeing 1.02 MJD 58054.447 object MzLS_712661_z
   k4m_170520_050940_ooi_zd_ls9.fits.fz k4m_170520_050940_ooi_zd_ls9-CCD3 z exptime 249.105 propid 2016A-0453 seeing 1.09 MJD 57893.215 object MzLS_713103_z
   k4m_160125_092224_ooi_zd_ls9.fits.fz k4m_160125_092224_ooi_zd_ls9-CCD4 z exptime 123.105 propid 2016B-0453 seeing 1.04 MJD 57412.391 object MzLS_712214_z
Faked up EXPNUM 85140227
Faked up EXPNUM 85140227
Faked up EXPNUM 85140227
90prime get_fwhm: self.fwhm = 3.39
WARNING: EXPNUM None!=160125092224 in /global/cscratch1/sd/desimpp/dr9e/images/mosaic/CP/V4.3/CP20160124/k4m_160125_092224_ooi_zd_ls9.fits.fz header
Faked up EXPNUM 82190101
Faked up EXPNUM 82190101
Faked up EXPNUM 82190101
90prime get_fwhm: self.fwhm = 3.39
Faked up EXPNUM 85110243
Faked up EXPNUM 85110243
Faked up EXPNUM 85110243
90prime get_fwhm: self.fwhm = 3.07
mosaic get_fwhm: self.fwhm = 4.63
Faked up EXPNUM 84950057
Faked up EXPNUM 84950057
Faked up EXPNUM 84950057
90prime get_fwhm: self.fwhm = 3.39
WARNING: EXPNUM None!=160125093024 in /global/cscratch1/sd/desimpp/dr9e/images/mosaic/CP/V4.3/CP20160124/k4m_160125_093024_ooi_zd_ls9.fits.fz header
mosaic get_fwhm: self.fwhm = 4.19
Swapping in SKY_JOHN values for 1 splinesky cells; 109.86305 -> 109.832535
Traceback (most recent call last):
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3605, in <module>
    sys.exit(main())
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3575, in main
    run_brick(opt.brick, survey, **kwargs)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3161, in run_brick
    initial_args=initargs, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  [Previous line repeated 6 more times]
  File "/usr/local/lib/python/astrometry/util/stages.py", line 124, in runstage
    R = stagefunc(stage, **Px)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3150, in mystagefunc
    R = stagefunc(stage, mp=mp, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 40, in __call__
    rtn = super(CallGlobalTime, self).__call__(stage, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 32, in __call__
    return func(*self.args, **kwa)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 271, in stage_tims
    tims = list(mp.map(read_one_tim, args))
  File "/usr/local/lib/python/astrometry/util/multiproc.py", line 89, in map
    return self.pool.map(f, args, cs)
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 268, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 657, in get
    raise self._value
AssertionError
Something went wrong; please check the logfile.
ERROR: galaxy PGC027949; please check the logfile.
Finished galaxy PGC027949 in 0.165 minutes.
Started working on galaxy PGC027949 at Sun Feb  9 11:11:07 2020
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 146.31578100000002 69.3902517 --width 1069 --height 1069 --pixscale 0.262 --threads 32 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC027949 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run 90prime-mosaic --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC027949/PGC027949-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC027949/PGC027949-runbrick-%%(stage)s.p --unwise-coadds 

runbrick.py starting at 2020-02-09T11:11:08.205736
legacypipe git version: dr9e-pre1-7-g54c974a3
Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '146.31578100000002', '69.3902517', '--width', '1069', '--height', '1069', '--pixscale', '0.262', '--threads', '32', '--outdir', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC027949', '--survey-dir', '/global/cscratch1/sd/desimpp/dr9e', '--run', '90prime-mosaic', '--largegalaxy-preburner', '--write-stage', 'tims', '--write-stage', 'srcs', '--skip-calibs', '--checkpoint', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC027949/PGC027949-runbrick-checkpoint.p', '--pickle', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC027949/PGC027949-runbrick-%%(stage)s.p', '--unwise-coadds']
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 146.31578100000002 69.3902517 --width 1069 --height 1069 --pixscale 0.262 --threads 32 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC027949 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run 90prime-mosaic --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC027949/PGC027949-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC027949/PGC027949-runbrick-%%(stage)s.p --unwise-coadds

WARNING: no WISE sky background maps in /global/cscratch1/sd/desimpp/dr9e/calib/wise/modelsky
Parsed RA,Dec 146.31578100000002 69.3902517
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/home/legacyhalos/.astropy/cache/download/py3/lock' [astropy.utils.data]
Starting process 59524 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071168 MB
Starting process 59525 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071168 MB
Starting process 59526 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071168 MB
Starting process 59527 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071168 MB
Starting process 59529 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071168 MB
Starting process 59530 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB
Starting process 59531 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071176 MB
Starting process 59532 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07118 MB
Starting process 59533 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07118 MB
Starting process 59534 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07118 MB
Starting process 59535 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071184 MB
Starting process 59536 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071184 MB
Starting process 59537 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071188 MB
Starting process 59538 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071188 MB
Starting process 59539 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071188 MB
Starting process 59540 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071188 MB
Starting process 59541 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 71 MB, VmData: 140 MB, maxrss: 0.071196 MB
Starting process 59542 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 71 MB, VmData: 140 MB, maxrss: 0.072208 MB
Starting process 59543 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 71 MB, VmData: 140 MB, maxrss: 0.072208 MB
Starting process 59544 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071916 MB
Starting process 59545 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071912 MB
Starting process 59546 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071192 MB
Starting process 59547 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071548 MB
Starting process 59548 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071548 MB
Starting process 59549 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071548 MB
Starting process 59550 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071548 MB
Starting process 59551 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071548 MB
Starting process 59552 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071548 MB
Starting process 59553 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071548 MB
Starting process 59554 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071548 MB
Starting process 59555 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071548 MB
Starting process 59556 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071548 MB
Runstage writecat
Runstage wise_forced
Runstage coadds
Runstage fitblobs
Runstage srcs
Runstage largegalaxies
Runstage halos
Runstage outliers
Runstage refs
Runstage tims
Running stage tims
Running stage tims at 2020-02-09T11:11:10.174768
Keeping 15 CCDs:
   ksb_190112_101522_ooi_g_ls9.fits.fz ksb_190112_101522_ooi_g_ls9-CCD1 g exptime 124.63 propid NOTBASS    seeing 1.54 MJD 58495.427 object 
   ksb_190128_105804_ooi_r_ls9.fits.fz ksb_190128_105804_ooi_r_ls9-CCD1 r exptime 163.914 propid NOTBASS    seeing 1.39 MJD 58511.457 object 
   ksb_180411_072612_ooi_g_ls9.fits.fz ksb_180411_072612_ooi_g_ls9-CCD1 g exptime 215.859 propid BASS       seeing 1.54 MJD 58219.310 object 45082
   ksb_190131_114353_ooi_r_ls9.fits.fz ksb_190131_114353_ooi_r_ls9-CCD1 r exptime 139.363 propid NOTBASS    seeing 1.54 MJD 58514.489 object 
   ksb_190128_062918_ooi_g_ls9.fits.fz ksb_190128_062918_ooi_g_ls9-CCD2 g exptime 118.299 propid NOTBASS    seeing 1.36 MJD 58511.270 object 
   ksb_190209_093432_ooi_r_ls9.fits.fz ksb_190209_093432_ooi_r_ls9-CCD2 r exptime 140.004 propid NOTBASS    seeing 1.48 MJD 58523.399 object 
   ksb_180411_072612_ooi_g_ls9.fits.fz ksb_180411_072612_ooi_g_ls9-CCD3 g exptime 215.859 propid BASS       seeing 1.56 MJD 58219.310 object 45082
   ksb_190131_114353_ooi_r_ls9.fits.fz ksb_190131_114353_ooi_r_ls9-CCD3 r exptime 139.363 propid NOTBASS    seeing 1.46 MJD 58514.489 object 
   k4m_170520_050940_ooi_zd_ls9.fits.fz k4m_170520_050940_ooi_zd_ls9-CCD1 z exptime 249.105 propid 2016A-0453 seeing 1.20 MJD 57893.215 object MzLS_713103_z
   k4m_170208_080820_ooi_zd_ls9.fits.fz k4m_170208_080820_ooi_zd_ls9-CCD2 z exptime 124.104 propid 2016A-0453 seeing 1.14 MJD 57792.339 object MzLS_711764_z
   k4m_170208_080820_ooi_zd_ls9.fits.fz k4m_170208_080820_ooi_zd_ls9-CCD4 z exptime 124.104 propid 2016A-0453 seeing 1.17 MJD 57792.339 object MzLS_711764_z
   k4m_160125_093024_ooi_zd_ls9.fits.fz k4m_160125_093024_ooi_zd_ls9-CCD2 z exptime 127.103 propid 2016B-0453 seeing 1.00 MJD 57412.396 object MzLS_712215_z
   k4m_171028_104349_ooi_zd_ls9.fits.fz k4m_171028_104349_ooi_zd_ls9-CCD1 z exptime 158.103 propid 2016A-0453 seeing 1.02 MJD 58054.447 object MzLS_712661_z
   k4m_170520_050940_ooi_zd_ls9.fits.fz k4m_170520_050940_ooi_zd_ls9-CCD3 z exptime 249.105 propid 2016A-0453 seeing 1.09 MJD 57893.215 object MzLS_713103_z
   k4m_160125_092224_ooi_zd_ls9.fits.fz k4m_160125_092224_ooi_zd_ls9-CCD4 z exptime 123.105 propid 2016B-0453 seeing 1.04 MJD 57412.391 object MzLS_712214_z
WARNING: EXPNUM None!=160125092224 in /global/cscratch1/sd/desimpp/dr9e/images/mosaic/CP/V4.3/CP20160124/k4m_160125_092224_ooi_zd_ls9.fits.fz header
WARNING: EXPNUM None!=160125093024 in /global/cscratch1/sd/desimpp/dr9e/images/mosaic/CP/V4.3/CP20160124/k4m_160125_093024_ooi_zd_ls9.fits.fz header
Faked up EXPNUM 82190101
Faked up EXPNUM 82190101
Faked up EXPNUM 82190101
90prime get_fwhm: self.fwhm = 3.45
Faked up EXPNUM 82190101
Faked up EXPNUM 82190101
Faked up EXPNUM 82190101
90prime get_fwhm: self.fwhm = 3.39
mosaic get_fwhm: self.fwhm = 4.19
Swapping in SKY_JOHN values for 1 splinesky cells; 109.86305 -> 109.832535
Faked up EXPNUM 85140227
Faked up EXPNUM 85140227
Faked up EXPNUM 85140227
90prime get_fwhm: self.fwhm = 3.21
Faked up EXPNUM 85140227
Faked up EXPNUM 85140227
Faked up EXPNUM 85140227
90prime get_fwhm: self.fwhm = 3.39
Faked up EXPNUM 85110243
Faked up EXPNUM 85110243
Faked up EXPNUM 85110243
90prime get_fwhm: self.fwhm = 3.07
mosaic get_fwhm: self.fwhm = 4.51
Faked up EXPNUM 84950057
Faked up EXPNUM 84950057
Faked up EXPNUM 84950057
90prime get_fwhm: self.fwhm = 3.39
Faked up EXPNUM 85230158
Faked up EXPNUM 85230158
Faked up EXPNUM 85230158
90prime get_fwhm: self.fwhm = 3.27
Faked up EXPNUM 85110142
Faked up EXPNUM 85110142
Faked up EXPNUM 85110142
90prime get_fwhm: self.fwhm = 2.99
Traceback (most recent call last):
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3605, in <module>
    sys.exit(main())
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3575, in main
    run_brick(opt.brick, survey, **kwargs)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3161, in run_brick
    initial_args=initargs, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  [Previous line repeated 6 more times]
  File "/usr/local/lib/python/astrometry/util/stages.py", line 124, in runstage
    R = stagefunc(stage, **Px)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3150, in mystagefunc
    R = stagefunc(stage, mp=mp, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 40, in __call__
    rtn = super(CallGlobalTime, self).__call__(stage, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 32, in __call__
    return func(*self.args, **kwa)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 271, in stage_tims
    tims = list(mp.map(read_one_tim, args))
  File "/usr/local/lib/python/astrometry/util/multiproc.py", line 89, in map
    return self.pool.map(f, args, cs)
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 268, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 657, in get
    raise self._value
AssertionError
Something went wrong; please check the logfile.
ERROR: galaxy PGC027949; please check the logfile.
Finished galaxy PGC027949 in 0.075 minutes.
Started working on galaxy PGC027949 at Mon Feb 10 06:17:43 2020
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 146.31578100000002 69.3902517 --width 1069 --height 1069 --pixscale 0.262 --threads 32 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC027949 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run 90prime-mosaic --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC027949/PGC027949-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC027949/PGC027949-runbrick-%%(stage)s.p --unwise-coadds 

runbrick.py starting at 2020-02-10T06:17:44.289704
legacypipe git version: dr9e-pre1-9-g4a51be93
Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '146.31578100000002', '69.3902517', '--width', '1069', '--height', '1069', '--pixscale', '0.262', '--threads', '32', '--outdir', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC027949', '--survey-dir', '/global/cscratch1/sd/desimpp/dr9e', '--run', '90prime-mosaic', '--largegalaxy-preburner', '--write-stage', 'tims', '--write-stage', 'srcs', '--skip-calibs', '--checkpoint', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC027949/PGC027949-runbrick-checkpoint.p', '--pickle', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC027949/PGC027949-runbrick-%%(stage)s.p', '--unwise-coadds']
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 146.31578100000002 69.3902517 --width 1069 --height 1069 --pixscale 0.262 --threads 32 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC027949 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run 90prime-mosaic --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC027949/PGC027949-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC027949/PGC027949-runbrick-%%(stage)s.p --unwise-coadds

WARNING: no WISE sky background maps in /global/cscratch1/sd/desimpp/dr9e/calib/wise/modelsky
Parsed RA,Dec 146.31578100000002 69.3902517
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/home/legacyhalos/.astropy/cache/download/py3/lock' [astropy.utils.data]
Starting process 15492 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071124 MB
Starting process 15493 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071124 MB
Starting process 15494 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071124 MB
Starting process 15495 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071124 MB
Starting process 15496 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071124 MB
Starting process 15497 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071128 MB
Starting process 15498 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071132 MB
Starting process 15499 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071136 MB
Starting process 15500 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071136 MB
Starting process 15501 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071136 MB
Starting process 15502 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07114 MB
Starting process 15503 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07114 MB
Starting process 15504 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071144 MB
Starting process 15505 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071144 MB
Starting process 15506 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071144 MB
Starting process 15507 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071148 MB
Starting process 15508 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 71 MB, VmData: 140 MB, maxrss: 0.071148 MB
Starting process 15509 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07216 MB
Starting process 15510 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07216 MB
Starting process 15511 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071868 MB
Starting process 15512 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071868 MB
Starting process 15513 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071144 MB
Starting process 15514 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.0715 MB
Starting process 15515 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.0715 MB
Starting process 15516 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.0715 MB
Starting process 15517 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.0715 MB
Starting process 15518 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.0715 MB
Starting process 15519 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.0715 MB
Starting process 15520 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.0715 MB
Starting process 15521 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.0715 MB
Starting process 15522 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.0715 MB
Starting process 15523 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071504 MB
Runstage writecat
Runstage wise_forced
Runstage coadds
Runstage fitblobs
Runstage srcs
Runstage largegalaxies
Runstage halos
Runstage outliers
Runstage refs
Runstage tims
Running stage tims
Running stage tims at 2020-02-10T06:17:45.542885
Keeping 15 CCDs:
   ksb_190112_101522_ooi_g_ls9.fits.fz ksb_190112_101522_ooi_g_ls9-CCD1 g exptime 124.63 propid NOTBASS    seeing 1.54 MJD 58495.427 object 
   ksb_190128_105804_ooi_r_ls9.fits.fz ksb_190128_105804_ooi_r_ls9-CCD1 r exptime 163.914 propid NOTBASS    seeing 1.39 MJD 58511.457 object 
   ksb_180411_072612_ooi_g_ls9.fits.fz ksb_180411_072612_ooi_g_ls9-CCD1 g exptime 215.859 propid BASS       seeing 1.54 MJD 58219.310 object 45082
   ksb_190131_114353_ooi_r_ls9.fits.fz ksb_190131_114353_ooi_r_ls9-CCD1 r exptime 139.363 propid NOTBASS    seeing 1.54 MJD 58514.489 object 
   ksb_190128_062918_ooi_g_ls9.fits.fz ksb_190128_062918_ooi_g_ls9-CCD2 g exptime 118.299 propid NOTBASS    seeing 1.36 MJD 58511.270 object 
   ksb_190209_093432_ooi_r_ls9.fits.fz ksb_190209_093432_ooi_r_ls9-CCD2 r exptime 140.004 propid NOTBASS    seeing 1.48 MJD 58523.399 object 
   ksb_180411_072612_ooi_g_ls9.fits.fz ksb_180411_072612_ooi_g_ls9-CCD3 g exptime 215.859 propid BASS       seeing 1.56 MJD 58219.310 object 45082
   ksb_190131_114353_ooi_r_ls9.fits.fz ksb_190131_114353_ooi_r_ls9-CCD3 r exptime 139.363 propid NOTBASS    seeing 1.46 MJD 58514.489 object 
   k4m_170520_050940_ooi_zd_ls9.fits.fz k4m_170520_050940_ooi_zd_ls9-CCD1 z exptime 249.105 propid 2016A-0453 seeing 1.20 MJD 57893.215 object MzLS_713103_z
   k4m_170208_080820_ooi_zd_ls9.fits.fz k4m_170208_080820_ooi_zd_ls9-CCD2 z exptime 124.104 propid 2016A-0453 seeing 1.14 MJD 57792.339 object MzLS_711764_z
   k4m_170208_080820_ooi_zd_ls9.fits.fz k4m_170208_080820_ooi_zd_ls9-CCD4 z exptime 124.104 propid 2016A-0453 seeing 1.17 MJD 57792.339 object MzLS_711764_z
   k4m_160125_093024_ooi_zd_ls9.fits.fz k4m_160125_093024_ooi_zd_ls9-CCD2 z exptime 127.103 propid 2016B-0453 seeing 1.00 MJD 57412.396 object MzLS_712215_z
   k4m_171028_104349_ooi_zd_ls9.fits.fz k4m_171028_104349_ooi_zd_ls9-CCD1 z exptime 158.103 propid 2016A-0453 seeing 1.02 MJD 58054.447 object MzLS_712661_z
   k4m_170520_050940_ooi_zd_ls9.fits.fz k4m_170520_050940_ooi_zd_ls9-CCD3 z exptime 249.105 propid 2016A-0453 seeing 1.09 MJD 57893.215 object MzLS_713103_z
   k4m_160125_092224_ooi_zd_ls9.fits.fz k4m_160125_092224_ooi_zd_ls9-CCD4 z exptime 123.105 propid 2016B-0453 seeing 1.04 MJD 57412.391 object MzLS_712214_z
WARNING: EXPNUM None!=160125093024 in /global/cscratch1/sd/desimpp/dr9e/images/mosaic/CP/V4.3/CP20160124/k4m_160125_093024_ooi_zd_ls9.fits.fz header
WARNING: EXPNUM None!=160125092224 in /global/cscratch1/sd/desimpp/dr9e/images/mosaic/CP/V4.3/CP20160124/k4m_160125_092224_ooi_zd_ls9.fits.fz header
Faked up EXPNUM 85140227
Faked up EXPNUM 85140227
Faked up EXPNUM 85140227
90prime get_fwhm: self.fwhm = 3.39
Faked up EXPNUM 82190101
Faked up EXPNUM 82190101
Faked up EXPNUM 82190101
90prime get_fwhm: self.fwhm = 3.39
Faked up EXPNUM 85140227
Faked up EXPNUM 85140227
Faked up EXPNUM 85140227
90prime get_fwhm: self.fwhm = 3.21
Faked up EXPNUM 82190101
Faked up EXPNUM 82190101
Faked up EXPNUM 82190101
90prime get_fwhm: self.fwhm = 3.45
Faked up EXPNUM 84950057
Faked up EXPNUM 84950057
Faked up EXPNUM 84950057
90prime get_fwhm: self.fwhm = 3.39
mosaic get_fwhm: self.fwhm = 4.51
Faked up EXPNUM 85110243
Faked up EXPNUM 85110243
Faked up EXPNUM 85110243
90prime get_fwhm: self.fwhm = 3.07
mosaic get_fwhm: self.fwhm = 4.19
Swapping in SKY_JOHN values for 1 splinesky cells; 109.86305 -> 109.832535
mosaic get_fwhm: self.fwhm = 4.63
Traceback (most recent call last):
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3605, in <module>
    sys.exit(main())
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3575, in main
    run_brick(opt.brick, survey, **kwargs)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3161, in run_brick
    initial_args=initargs, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  [Previous line repeated 6 more times]
  File "/usr/local/lib/python/astrometry/util/stages.py", line 124, in runstage
    R = stagefunc(stage, **Px)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3150, in mystagefunc
    R = stagefunc(stage, mp=mp, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 40, in __call__
    rtn = super(CallGlobalTime, self).__call__(stage, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 32, in __call__
    return func(*self.args, **kwa)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 271, in stage_tims
    tims = list(mp.map(read_one_tim, args))
  File "/usr/local/lib/python/astrometry/util/multiproc.py", line 89, in map
    return self.pool.map(f, args, cs)
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 268, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 657, in get
    raise self._value
AssertionError
Something went wrong; please check the logfile.
ERROR: galaxy PGC027949; please check the logfile.
Finished galaxy PGC027949 in 0.065 minutes.