Started working on galaxy PGC027688 at Sat Feb  8 08:11:37 2020
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 145.416456 68.382848 --width 399 --height 399 --pixscale 0.262 --threads 8 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/145/PGC027688 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run 90prime-mosaic --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/145/PGC027688/PGC027688-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/145/PGC027688/PGC027688-runbrick-%%(stage)s.p --unwise-coadds 

runbrick.py starting at 2020-02-08T08:11:37.673234
legacypipe git version: dr9e-pre1-7-g54c974a3
Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '145.416456', '68.382848', '--width', '399', '--height', '399', '--pixscale', '0.262', '--threads', '8', '--outdir', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/145/PGC027688', '--survey-dir', '/global/cscratch1/sd/desimpp/dr9e', '--run', '90prime-mosaic', '--largegalaxy-preburner', '--write-stage', 'tims', '--write-stage', 'srcs', '--skip-calibs', '--checkpoint', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/145/PGC027688/PGC027688-runbrick-checkpoint.p', '--pickle', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/145/PGC027688/PGC027688-runbrick-%%(stage)s.p', '--unwise-coadds']
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 145.416456 68.382848 --width 399 --height 399 --pixscale 0.262 --threads 8 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/145/PGC027688 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run 90prime-mosaic --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/145/PGC027688/PGC027688-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/145/PGC027688/PGC027688-runbrick-%%(stage)s.p --unwise-coadds

WARNING: no WISE sky background maps in /global/cscratch1/sd/desimpp/dr9e/calib/wise/modelsky
Parsed RA,Dec 145.416456 68.382848
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/home/legacyhalos/.astropy/cache/download/py3/lock' [astropy.utils.data]
Starting process 30457 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB
Starting process 30458 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB
Starting process 30459 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB
Starting process 30460 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB
Starting process 30461 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB
Starting process 30462 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071176 MB
Starting process 30463 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071176 MB
Starting process 30464 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07118 MB
Runstage writecat
Runstage wise_forced
Runstage coadds
Runstage fitblobs
Runstage srcs
Runstage largegalaxies
Runstage halos
Runstage outliers
Runstage refs
Runstage tims
Running stage tims
Running stage tims at 2020-02-08T08:11:38.986974
Keeping 9 CCDs:
   ksb_190131_113812_ooi_r_ls9.fits.fz ksb_190131_113812_ooi_r_ls9-CCD1 r exptime 116.707 propid NOTBASS    seeing 1.59 MJD 58514.485 object 
   ksb_180411_071357_ooi_g_ls9.fits.fz ksb_180411_071357_ooi_g_ls9-CCD1 g exptime 219.186 propid BASS       seeing 1.69 MJD 58219.301 object 44292
   ksb_190128_062355_ooi_g_ls9.fits.fz ksb_190128_062355_ooi_g_ls9-CCD2 g exptime 118.43 propid NOTBASS    seeing 1.38 MJD 58511.267 object 
   ksb_190209_092322_ooi_r_ls9.fits.fz ksb_190209_092322_ooi_r_ls9-CCD2 r exptime 140.178 propid NOTBASS    seeing 1.56 MJD 58523.391 object 
   ksb_190112_101235_ooi_g_ls9.fits.fz ksb_190112_101235_ooi_g_ls9-CCD1 g exptime 126.55 propid NOTBASS    seeing 1.73 MJD 58495.425 object 
   ksb_190128_105043_ooi_r_ls9.fits.fz ksb_190128_105043_ooi_r_ls9-CCD1 r exptime 165.75 propid NOTBASS    seeing 1.27 MJD 58511.452 object 
   k4m_160317_060522_ooi_zd_ls9.fits.fz k4m_160317_060522_ooi_zd_ls9-CCD3 z exptime 80.104 propid 2016A-0453 seeing 1.23 MJD 57464.254 object MzLS_710389_z
   k4m_170307_072650_ooi_zd_ls9.fits.fz k4m_170307_072650_ooi_zd_ls9-CCD2 z exptime 80.104 propid 2016A-0453 seeing 0.89 MJD 57819.310 object MzLS_709923_z
   k4m_160125_091658_ooi_zd_ls9.fits.fz k4m_160125_091658_ooi_zd_ls9-CCD4 z exptime 124.101 propid 2016B-0453 seeing 1.06 MJD 57412.387 object MzLS_709453_z
Faked up EXPNUM 84950056
Faked up EXPNUM 84950056
Faked up EXPNUM 84950056
90prime get_fwhm: self.fwhm = 3.81
mosaic get_fwhm: self.fwhm = 4.75
WARNING: EXPNUM None!=160125091658 in /global/cscratch1/sd/desimpp/dr9e/images/mosaic/CP/V4.3/CP20160124/k4m_160125_091658_ooi_zd_ls9.fits.fz header
Faked up EXPNUM 85140225
Faked up EXPNUM 85140225
Faked up EXPNUM 85140225
90prime get_fwhm: self.fwhm = 3.51
Faked up EXPNUM 82190098
Faked up EXPNUM 82190098
Faked up EXPNUM 82190098
90prime get_fwhm: self.fwhm = 3.73
mosaic get_fwhm: self.fwhm = 3.43
Traceback (most recent call last):
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3605, in <module>
    sys.exit(main())
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3575, in main
    run_brick(opt.brick, survey, **kwargs)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3161, in run_brick
    initial_args=initargs, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  [Previous line repeated 6 more times]
  File "/usr/local/lib/python/astrometry/util/stages.py", line 124, in runstage
    R = stagefunc(stage, **Px)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3150, in mystagefunc
    R = stagefunc(stage, mp=mp, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 40, in __call__
    rtn = super(CallGlobalTime, self).__call__(stage, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 32, in __call__
    return func(*self.args, **kwa)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 271, in stage_tims
    tims = list(mp.map(read_one_tim, args))
  File "/usr/local/lib/python/astrometry/util/multiproc.py", line 89, in map
    return self.pool.map(f, args, cs)
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 268, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 657, in get
    raise self._value
AssertionError
Something went wrong; please check the logfile.
ERROR: galaxy PGC027688; please check the logfile.
Finished galaxy PGC027688 in 0.062 minutes.
Started working on galaxy PGC027688 at Sat Feb  8 08:34:05 2020
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 145.416456 68.382848 --width 399 --height 399 --pixscale 0.262 --threads 8 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/145/PGC027688 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run 90prime-mosaic --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/145/PGC027688/PGC027688-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/145/PGC027688/PGC027688-runbrick-%%(stage)s.p --unwise-coadds 

runbrick.py starting at 2020-02-08T08:34:05.671300
legacypipe git version: dr9e-pre1-7-g54c974a3
Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '145.416456', '68.382848', '--width', '399', '--height', '399', '--pixscale', '0.262', '--threads', '8', '--outdir', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/145/PGC027688', '--survey-dir', '/global/cscratch1/sd/desimpp/dr9e', '--run', '90prime-mosaic', '--largegalaxy-preburner', '--write-stage', 'tims', '--write-stage', 'srcs', '--skip-calibs', '--checkpoint', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/145/PGC027688/PGC027688-runbrick-checkpoint.p', '--pickle', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/145/PGC027688/PGC027688-runbrick-%%(stage)s.p', '--unwise-coadds']
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 145.416456 68.382848 --width 399 --height 399 --pixscale 0.262 --threads 8 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/145/PGC027688 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run 90prime-mosaic --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/145/PGC027688/PGC027688-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/145/PGC027688/PGC027688-runbrick-%%(stage)s.p --unwise-coadds

WARNING: no WISE sky background maps in /global/cscratch1/sd/desimpp/dr9e/calib/wise/modelsky
Parsed RA,Dec 145.416456 68.382848
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/home/legacyhalos/.astropy/cache/download/py3/lock' [astropy.utils.data]
Starting process 6457 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071168 MB
Starting process 6458 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071168 MB
Starting process 6460 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071168 MB
Starting process 6461 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071168 MB
Starting process 6462 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071168 MB
Starting process 6463 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB
Starting process 6464 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071176 MB
Starting process 6465 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071176 MB
Runstage writecat
Runstage wise_forced
Runstage coadds
Runstage fitblobs
Runstage srcs
Runstage largegalaxies
Runstage halos
Runstage outliers
Runstage refs
Runstage tims
Running stage tims
Running stage tims at 2020-02-08T08:34:06.959155
Keeping 9 CCDs:
   ksb_190131_113812_ooi_r_ls9.fits.fz ksb_190131_113812_ooi_r_ls9-CCD1 r exptime 116.707 propid NOTBASS    seeing 1.59 MJD 58514.485 object 
   ksb_180411_071357_ooi_g_ls9.fits.fz ksb_180411_071357_ooi_g_ls9-CCD1 g exptime 219.186 propid BASS       seeing 1.69 MJD 58219.301 object 44292
   ksb_190128_062355_ooi_g_ls9.fits.fz ksb_190128_062355_ooi_g_ls9-CCD2 g exptime 118.43 propid NOTBASS    seeing 1.38 MJD 58511.267 object 
   ksb_190209_092322_ooi_r_ls9.fits.fz ksb_190209_092322_ooi_r_ls9-CCD2 r exptime 140.178 propid NOTBASS    seeing 1.56 MJD 58523.391 object 
   ksb_190112_101235_ooi_g_ls9.fits.fz ksb_190112_101235_ooi_g_ls9-CCD1 g exptime 126.55 propid NOTBASS    seeing 1.73 MJD 58495.425 object 
   ksb_190128_105043_ooi_r_ls9.fits.fz ksb_190128_105043_ooi_r_ls9-CCD1 r exptime 165.75 propid NOTBASS    seeing 1.27 MJD 58511.452 object 
   k4m_160317_060522_ooi_zd_ls9.fits.fz k4m_160317_060522_ooi_zd_ls9-CCD3 z exptime 80.104 propid 2016A-0453 seeing 1.23 MJD 57464.254 object MzLS_710389_z
   k4m_170307_072650_ooi_zd_ls9.fits.fz k4m_170307_072650_ooi_zd_ls9-CCD2 z exptime 80.104 propid 2016A-0453 seeing 0.89 MJD 57819.310 object MzLS_709923_z
   k4m_160125_091658_ooi_zd_ls9.fits.fz k4m_160125_091658_ooi_zd_ls9-CCD4 z exptime 124.101 propid 2016B-0453 seeing 1.06 MJD 57412.387 object MzLS_709453_z
Faked up EXPNUM 85110241
Faked up EXPNUM 85110241
Faked up EXPNUM 85110241
90prime get_fwhm: self.fwhm = 2.81
mosaic get_fwhm: self.fwhm = 4.75
WARNING: EXPNUM None!=160125091658 in /global/cscratch1/sd/desimpp/dr9e/images/mosaic/CP/V4.3/CP20160124/k4m_160125_091658_ooi_zd_ls9.fits.fz header
Faked up EXPNUM 85140225
Faked up EXPNUM 85140225
Faked up EXPNUM 85140225
90prime get_fwhm: self.fwhm = 3.51
Faked up EXPNUM 84950056
Faked up EXPNUM 84950056
Faked up EXPNUM 84950056
90prime get_fwhm: self.fwhm = 3.81
Faked up EXPNUM 85230154
Faked up EXPNUM 85230154
Faked up EXPNUM 85230154
90prime get_fwhm: self.fwhm = 3.43
mosaic get_fwhm: self.fwhm = 3.43
Traceback (most recent call last):
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3605, in <module>
    sys.exit(main())
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3575, in main
    run_brick(opt.brick, survey, **kwargs)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3161, in run_brick
    initial_args=initargs, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  [Previous line repeated 6 more times]
  File "/usr/local/lib/python/astrometry/util/stages.py", line 124, in runstage
    R = stagefunc(stage, **Px)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3150, in mystagefunc
    R = stagefunc(stage, mp=mp, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 40, in __call__
    rtn = super(CallGlobalTime, self).__call__(stage, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 32, in __call__
    return func(*self.args, **kwa)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 271, in stage_tims
    tims = list(mp.map(read_one_tim, args))
  File "/usr/local/lib/python/astrometry/util/multiproc.py", line 89, in map
    return self.pool.map(f, args, cs)
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 268, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 657, in get
    raise self._value
AssertionError
Something went wrong; please check the logfile.
ERROR: galaxy PGC027688; please check the logfile.
Finished galaxy PGC027688 in 0.060 minutes.
Started working on galaxy PGC027688 at Sat Feb  8 12:46:59 2020
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 145.416456 68.382848 --width 399 --height 399 --pixscale 0.262 --threads 8 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/145/PGC027688 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run 90prime-mosaic --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/145/PGC027688/PGC027688-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/145/PGC027688/PGC027688-runbrick-%%(stage)s.p --unwise-coadds 

runbrick.py starting at 2020-02-08T12:46:59.852778
legacypipe git version: dr9e-pre1-7-g54c974a3
Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '145.416456', '68.382848', '--width', '399', '--height', '399', '--pixscale', '0.262', '--threads', '8', '--outdir', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/145/PGC027688', '--survey-dir', '/global/cscratch1/sd/desimpp/dr9e', '--run', '90prime-mosaic', '--largegalaxy-preburner', '--write-stage', 'tims', '--write-stage', 'srcs', '--skip-calibs', '--checkpoint', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/145/PGC027688/PGC027688-runbrick-checkpoint.p', '--pickle', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/145/PGC027688/PGC027688-runbrick-%%(stage)s.p', '--unwise-coadds']
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 145.416456 68.382848 --width 399 --height 399 --pixscale 0.262 --threads 8 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/145/PGC027688 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run 90prime-mosaic --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/145/PGC027688/PGC027688-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/145/PGC027688/PGC027688-runbrick-%%(stage)s.p --unwise-coadds

WARNING: no WISE sky background maps in /global/cscratch1/sd/desimpp/dr9e/calib/wise/modelsky
Parsed RA,Dec 145.416456 68.382848
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/home/legacyhalos/.astropy/cache/download/py3/lock' [astropy.utils.data]
Starting process 10890 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071176 MB
Starting process 10891 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071176 MB
Starting process 10892 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071176 MB
Starting process 10893 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071176 MB
Starting process 10894 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071176 MB
Starting process 10895 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07118 MB
Starting process 10896 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07118 MB
Starting process 10897 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071184 MB
Runstage writecat
Runstage wise_forced
Runstage coadds
Runstage fitblobs
Runstage srcs
Runstage largegalaxies
Runstage halos
Runstage outliers
Runstage refs
Runstage tims
Running stage tims
Running stage tims at 2020-02-08T12:47:01.180378
Keeping 9 CCDs:
   ksb_190131_113812_ooi_r_ls9.fits.fz ksb_190131_113812_ooi_r_ls9-CCD1 r exptime 116.707 propid NOTBASS    seeing 1.59 MJD 58514.485 object 
   ksb_180411_071357_ooi_g_ls9.fits.fz ksb_180411_071357_ooi_g_ls9-CCD1 g exptime 219.186 propid BASS       seeing 1.69 MJD 58219.301 object 44292
   ksb_190128_062355_ooi_g_ls9.fits.fz ksb_190128_062355_ooi_g_ls9-CCD2 g exptime 118.43 propid NOTBASS    seeing 1.38 MJD 58511.267 object 
   ksb_190209_092322_ooi_r_ls9.fits.fz ksb_190209_092322_ooi_r_ls9-CCD2 r exptime 140.178 propid NOTBASS    seeing 1.56 MJD 58523.391 object 
   ksb_190112_101235_ooi_g_ls9.fits.fz ksb_190112_101235_ooi_g_ls9-CCD1 g exptime 126.55 propid NOTBASS    seeing 1.73 MJD 58495.425 object 
   ksb_190128_105043_ooi_r_ls9.fits.fz ksb_190128_105043_ooi_r_ls9-CCD1 r exptime 165.75 propid NOTBASS    seeing 1.27 MJD 58511.452 object 
   k4m_160317_060522_ooi_zd_ls9.fits.fz k4m_160317_060522_ooi_zd_ls9-CCD3 z exptime 80.104 propid 2016A-0453 seeing 1.23 MJD 57464.254 object MzLS_710389_z
   k4m_170307_072650_ooi_zd_ls9.fits.fz k4m_170307_072650_ooi_zd_ls9-CCD2 z exptime 80.104 propid 2016A-0453 seeing 0.89 MJD 57819.310 object MzLS_709923_z
   k4m_160125_091658_ooi_zd_ls9.fits.fz k4m_160125_091658_ooi_zd_ls9-CCD4 z exptime 124.101 propid 2016B-0453 seeing 1.06 MJD 57412.387 object MzLS_709453_z
mosaic get_fwhm: self.fwhm = 4.75
WARNING: EXPNUM None!=160125091658 in /global/cscratch1/sd/desimpp/dr9e/images/mosaic/CP/V4.3/CP20160124/k4m_160125_091658_ooi_zd_ls9.fits.fz header
mosaic get_fwhm: self.fwhm = 3.43
Faked up EXPNUM 82190098
Faked up EXPNUM 82190098
Faked up EXPNUM 82190098
90prime get_fwhm: self.fwhm = 3.73
Faked up EXPNUM 85110241
Faked up EXPNUM 85110241
Faked up EXPNUM 85110241
90prime get_fwhm: self.fwhm = 2.81
Faked up EXPNUM 85140225
Faked up EXPNUM 85140225
Faked up EXPNUM 85140225
90prime get_fwhm: self.fwhm = 3.51
Faked up EXPNUM 85110140
Faked up EXPNUM 85110140
Faked up EXPNUM 85110140
90prime get_fwhm: self.fwhm = 3.05
Faked up EXPNUM 84950056
Faked up EXPNUM 84950056
Faked up EXPNUM 84950056
90prime get_fwhm: self.fwhm = 3.81
Faked up EXPNUM 85230154
Faked up EXPNUM 85230154
Faked up EXPNUM 85230154
90prime get_fwhm: self.fwhm = 3.43
Traceback (most recent call last):
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3605, in <module>
    sys.exit(main())
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3575, in main
    run_brick(opt.brick, survey, **kwargs)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3161, in run_brick
    initial_args=initargs, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  [Previous line repeated 6 more times]
  File "/usr/local/lib/python/astrometry/util/stages.py", line 124, in runstage
    R = stagefunc(stage, **Px)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3150, in mystagefunc
    R = stagefunc(stage, mp=mp, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 40, in __call__
    rtn = super(CallGlobalTime, self).__call__(stage, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 32, in __call__
    return func(*self.args, **kwa)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 271, in stage_tims
    tims = list(mp.map(read_one_tim, args))
  File "/usr/local/lib/python/astrometry/util/multiproc.py", line 89, in map
    return self.pool.map(f, args, cs)
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 268, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 657, in get
    raise self._value
AssertionError
Something went wrong; please check the logfile.
ERROR: galaxy PGC027688; please check the logfile.
Finished galaxy PGC027688 in 0.063 minutes.
Started working on galaxy PGC027688 at Sun Feb  9 09:55:24 2020
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 145.416456 68.382848 --width 399 --height 399 --pixscale 0.262 --threads 32 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/145/PGC027688 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run 90prime-mosaic --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/145/PGC027688/PGC027688-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/145/PGC027688/PGC027688-runbrick-%%(stage)s.p --unwise-coadds 

runbrick.py starting at 2020-02-09T09:55:25.065490
legacypipe git version: dr9e-pre1-7-g54c974a3
Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '145.416456', '68.382848', '--width', '399', '--height', '399', '--pixscale', '0.262', '--threads', '32', '--outdir', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/145/PGC027688', '--survey-dir', '/global/cscratch1/sd/desimpp/dr9e', '--run', '90prime-mosaic', '--largegalaxy-preburner', '--write-stage', 'tims', '--write-stage', 'srcs', '--skip-calibs', '--checkpoint', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/145/PGC027688/PGC027688-runbrick-checkpoint.p', '--pickle', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/145/PGC027688/PGC027688-runbrick-%%(stage)s.p', '--unwise-coadds']
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 145.416456 68.382848 --width 399 --height 399 --pixscale 0.262 --threads 32 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/145/PGC027688 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run 90prime-mosaic --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/145/PGC027688/PGC027688-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/145/PGC027688/PGC027688-runbrick-%%(stage)s.p --unwise-coadds

WARNING: no WISE sky background maps in /global/cscratch1/sd/desimpp/dr9e/calib/wise/modelsky
Parsed RA,Dec 145.416456 68.382848
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/home/legacyhalos/.astropy/cache/download/py3/lock' [astropy.utils.data]
Starting process 36725 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071168 MB
Starting process 36726 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB
Starting process 36727 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB
Starting process 36728 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB
Starting process 36729 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB
Starting process 36730 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071176 MB
Starting process 36731 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07118 MB
Starting process 36732 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071184 MB
Starting process 36733 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071184 MB
Starting process 36734 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071184 MB
Starting process 36735 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071188 MB
Starting process 36736 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071192 MB
Starting process 36737 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071192 MB
Starting process 36738 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071192 MB
Starting process 36739 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071192 MB
Starting process 36740 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071192 MB
Starting process 36741 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 71 MB, VmData: 140 MB, maxrss: 0.071196 MB
Starting process 36742 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 71 MB, VmData: 140 MB, maxrss: 0.072208 MB
Starting process 36743 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 71 MB, VmData: 140 MB, maxrss: 0.072208 MB
Starting process 36744 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071916 MB
Starting process 36745 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071912 MB
Starting process 36746 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071548 MB
Starting process 36747 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07148 MB
Starting process 36748 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07148 MB
Starting process 36749 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07148 MB
Starting process 36750 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07148 MB
Starting process 36751 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07148 MB
Starting process 36752 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07148 MB
Starting process 36753 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07148 MB
Starting process 36754 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071484 MB
Starting process 36755 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071484 MB
Starting process 36756 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071484 MB
Runstage writecat
Runstage wise_forced
Runstage coadds
Runstage fitblobs
Runstage srcs
Runstage largegalaxies
Runstage halos
Runstage outliers
Runstage refs
Runstage tims
Running stage tims
Running stage tims at 2020-02-09T09:55:26.328543
Keeping 9 CCDs:
   ksb_190131_113812_ooi_r_ls9.fits.fz ksb_190131_113812_ooi_r_ls9-CCD1 r exptime 116.707 propid NOTBASS    seeing 1.59 MJD 58514.485 object 
   ksb_180411_071357_ooi_g_ls9.fits.fz ksb_180411_071357_ooi_g_ls9-CCD1 g exptime 219.186 propid BASS       seeing 1.69 MJD 58219.301 object 44292
   ksb_190128_062355_ooi_g_ls9.fits.fz ksb_190128_062355_ooi_g_ls9-CCD2 g exptime 118.43 propid NOTBASS    seeing 1.38 MJD 58511.267 object 
   ksb_190209_092322_ooi_r_ls9.fits.fz ksb_190209_092322_ooi_r_ls9-CCD2 r exptime 140.178 propid NOTBASS    seeing 1.56 MJD 58523.391 object 
   ksb_190112_101235_ooi_g_ls9.fits.fz ksb_190112_101235_ooi_g_ls9-CCD1 g exptime 126.55 propid NOTBASS    seeing 1.73 MJD 58495.425 object 
   ksb_190128_105043_ooi_r_ls9.fits.fz ksb_190128_105043_ooi_r_ls9-CCD1 r exptime 165.75 propid NOTBASS    seeing 1.27 MJD 58511.452 object 
   k4m_160317_060522_ooi_zd_ls9.fits.fz k4m_160317_060522_ooi_zd_ls9-CCD3 z exptime 80.104 propid 2016A-0453 seeing 1.23 MJD 57464.254 object MzLS_710389_z
   k4m_170307_072650_ooi_zd_ls9.fits.fz k4m_170307_072650_ooi_zd_ls9-CCD2 z exptime 80.104 propid 2016A-0453 seeing 0.89 MJD 57819.310 object MzLS_709923_z
   k4m_160125_091658_ooi_zd_ls9.fits.fz k4m_160125_091658_ooi_zd_ls9-CCD4 z exptime 124.101 propid 2016B-0453 seeing 1.06 MJD 57412.387 object MzLS_709453_z
WARNING: EXPNUM None!=160125091658 in /global/cscratch1/sd/desimpp/dr9e/images/mosaic/CP/V4.3/CP20160124/k4m_160125_091658_ooi_zd_ls9.fits.fz header
Faked up EXPNUM 85140225
Faked up EXPNUM 85140225
Faked up EXPNUM 85140225
90prime get_fwhm: self.fwhm = 3.51
mosaic get_fwhm: self.fwhm = 4.75
Faked up EXPNUM 85110241
Faked up EXPNUM 85110241
Faked up EXPNUM 85110241
90prime get_fwhm: self.fwhm = 2.81
Faked up EXPNUM 85230154
Faked up EXPNUM 85230154
Faked up EXPNUM 85230154
90prime get_fwhm: self.fwhm = 3.43
mosaic get_fwhm: self.fwhm = 3.43
Faked up EXPNUM 84950056
Faked up EXPNUM 84950056
Faked up EXPNUM 84950056
90prime get_fwhm: self.fwhm = 3.81
Traceback (most recent call last):
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3605, in <module>
    sys.exit(main())
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3575, in main
    run_brick(opt.brick, survey, **kwargs)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3161, in run_brick
    initial_args=initargs, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  [Previous line repeated 6 more times]
  File "/usr/local/lib/python/astrometry/util/stages.py", line 124, in runstage
    R = stagefunc(stage, **Px)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3150, in mystagefunc
    R = stagefunc(stage, mp=mp, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 40, in __call__
    rtn = super(CallGlobalTime, self).__call__(stage, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 32, in __call__
    return func(*self.args, **kwa)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 271, in stage_tims
    tims = list(mp.map(read_one_tim, args))
  File "/usr/local/lib/python/astrometry/util/multiproc.py", line 89, in map
    return self.pool.map(f, args, cs)
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 268, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 657, in get
    raise self._value
AssertionError
Something went wrong; please check the logfile.
ERROR: galaxy PGC027688; please check the logfile.
Finished galaxy PGC027688 in 0.057 minutes.
Started working on galaxy PGC027688 at Mon Feb 10 06:12:45 2020
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 145.416456 68.382848 --width 399 --height 399 --pixscale 0.262 --threads 32 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/145/PGC027688 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run 90prime-mosaic --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/145/PGC027688/PGC027688-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/145/PGC027688/PGC027688-runbrick-%%(stage)s.p --unwise-coadds 

runbrick.py starting at 2020-02-10T06:12:45.992727
legacypipe git version: dr9e-pre1-9-g4a51be93
Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '145.416456', '68.382848', '--width', '399', '--height', '399', '--pixscale', '0.262', '--threads', '32', '--outdir', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/145/PGC027688', '--survey-dir', '/global/cscratch1/sd/desimpp/dr9e', '--run', '90prime-mosaic', '--largegalaxy-preburner', '--write-stage', 'tims', '--write-stage', 'srcs', '--skip-calibs', '--checkpoint', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/145/PGC027688/PGC027688-runbrick-checkpoint.p', '--pickle', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/145/PGC027688/PGC027688-runbrick-%%(stage)s.p', '--unwise-coadds']
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 145.416456 68.382848 --width 399 --height 399 --pixscale 0.262 --threads 32 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/145/PGC027688 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run 90prime-mosaic --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/145/PGC027688/PGC027688-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/145/PGC027688/PGC027688-runbrick-%%(stage)s.p --unwise-coadds

WARNING: no WISE sky background maps in /global/cscratch1/sd/desimpp/dr9e/calib/wise/modelsky
Parsed RA,Dec 145.416456 68.382848
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/home/legacyhalos/.astropy/cache/download/py3/lock' [astropy.utils.data]
Starting process 11610 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071132 MB
Starting process 11611 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071132 MB
Starting process 11612 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071132 MB
Starting process 11613 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071132 MB
Starting process 11614 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071132 MB
Starting process 11615 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07114 MB
Starting process 11616 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07114 MB
Starting process 11617 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071144 MB
Starting process 11618 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071144 MB
Starting process 11619 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071144 MB
Starting process 11620 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071148 MB
Starting process 11621 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071148 MB
Starting process 11622 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071152 MB
Starting process 11623 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071152 MB
Starting process 11624 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071152 MB
Starting process 11625 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071156 MB
Starting process 11626 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 71 MB, VmData: 140 MB, maxrss: 0.071156 MB
Starting process 11627 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.072168 MB
Starting process 11628 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.072168 MB
Starting process 11629 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071876 MB
Starting process 11630 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071876 MB
Starting process 11631 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071448 MB
Starting process 11632 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071448 MB
Starting process 11633 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071448 MB
Starting process 11634 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071448 MB
Starting process 11635 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071448 MB
Starting process 11636 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071448 MB
Starting process 11637 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071448 MB
Starting process 11638 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071448 MB
Starting process 11639 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071448 MB
Starting process 11640 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071448 MB
Starting process 11641 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071452 MB
Runstage writecat
Runstage wise_forced
Runstage coadds
Runstage fitblobs
Runstage srcs
Runstage largegalaxies
Runstage halos
Runstage outliers
Runstage refs
Runstage tims
Running stage tims
Running stage tims at 2020-02-10T06:12:47.251773
Keeping 9 CCDs:
   ksb_190131_113812_ooi_r_ls9.fits.fz ksb_190131_113812_ooi_r_ls9-CCD1 r exptime 116.707 propid NOTBASS    seeing 1.59 MJD 58514.485 object 
   ksb_180411_071357_ooi_g_ls9.fits.fz ksb_180411_071357_ooi_g_ls9-CCD1 g exptime 219.186 propid BASS       seeing 1.69 MJD 58219.301 object 44292
   ksb_190128_062355_ooi_g_ls9.fits.fz ksb_190128_062355_ooi_g_ls9-CCD2 g exptime 118.43 propid NOTBASS    seeing 1.38 MJD 58511.267 object 
   ksb_190209_092322_ooi_r_ls9.fits.fz ksb_190209_092322_ooi_r_ls9-CCD2 r exptime 140.178 propid NOTBASS    seeing 1.56 MJD 58523.391 object 
   ksb_190112_101235_ooi_g_ls9.fits.fz ksb_190112_101235_ooi_g_ls9-CCD1 g exptime 126.55 propid NOTBASS    seeing 1.73 MJD 58495.425 object 
   ksb_190128_105043_ooi_r_ls9.fits.fz ksb_190128_105043_ooi_r_ls9-CCD1 r exptime 165.75 propid NOTBASS    seeing 1.27 MJD 58511.452 object 
   k4m_160317_060522_ooi_zd_ls9.fits.fz k4m_160317_060522_ooi_zd_ls9-CCD3 z exptime 80.104 propid 2016A-0453 seeing 1.23 MJD 57464.254 object MzLS_710389_z
   k4m_170307_072650_ooi_zd_ls9.fits.fz k4m_170307_072650_ooi_zd_ls9-CCD2 z exptime 80.104 propid 2016A-0453 seeing 0.89 MJD 57819.310 object MzLS_709923_z
   k4m_160125_091658_ooi_zd_ls9.fits.fz k4m_160125_091658_ooi_zd_ls9-CCD4 z exptime 124.101 propid 2016B-0453 seeing 1.06 MJD 57412.387 object MzLS_709453_z
WARNING: EXPNUM None!=160125091658 in /global/cscratch1/sd/desimpp/dr9e/images/mosaic/CP/V4.3/CP20160124/k4m_160125_091658_ooi_zd_ls9.fits.fz header
mosaic get_fwhm: self.fwhm = 4.75
Faked up EXPNUM 82190098
Faked up EXPNUM 82190098
Faked up EXPNUM 82190098
90prime get_fwhm: self.fwhm = 3.73
mosaic get_fwhm: self.fwhm = 3.43
Faked up EXPNUM 85110241
Faked up EXPNUM 85110241
Faked up EXPNUM 85110241
90prime get_fwhm: self.fwhm = 2.81
Faked up EXPNUM 85140225
Faked up EXPNUM 85140225
Faked up EXPNUM 85140225
90prime get_fwhm: self.fwhm = 3.51
Faked up EXPNUM 84950056
Faked up EXPNUM 84950056
Faked up EXPNUM 84950056
90prime get_fwhm: self.fwhm = 3.81
Faked up EXPNUM 85230154
Faked up EXPNUM 85230154
Faked up EXPNUM 85230154
90prime get_fwhm: self.fwhm = 3.43
Faked up EXPNUM 85110140
Faked up EXPNUM 85110140
Faked up EXPNUM 85110140
90prime get_fwhm: self.fwhm = 3.05
Traceback (most recent call last):
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3605, in <module>
    sys.exit(main())
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3575, in main
    run_brick(opt.brick, survey, **kwargs)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3161, in run_brick
    initial_args=initargs, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  [Previous line repeated 6 more times]
  File "/usr/local/lib/python/astrometry/util/stages.py", line 124, in runstage
    R = stagefunc(stage, **Px)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3150, in mystagefunc
    R = stagefunc(stage, mp=mp, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 40, in __call__
    rtn = super(CallGlobalTime, self).__call__(stage, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 32, in __call__
    return func(*self.args, **kwa)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 271, in stage_tims
    tims = list(mp.map(read_one_tim, args))
  File "/usr/local/lib/python/astrometry/util/multiproc.py", line 89, in map
    return self.pool.map(f, args, cs)
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 268, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 657, in get
    raise self._value
AssertionError
Something went wrong; please check the logfile.
ERROR: galaxy PGC027688; please check the logfile.
Finished galaxy PGC027688 in 0.064 minutes.