o Mh @sddlmZddlZddlZddlZddlmZmZmZm Z ddl Z ddl Z ddl Z GdddZGdddZdd Zd d Zd d ZddZdddZdddZddZdS))PathN)PopenPIPESTDOUTDEVNULLc@s<eZdZdZdZdZdZdZdZdZ dZ d Z d Z d Z d Zd S)colorzzzzzzzzzzzzN)__name__ __module__ __qualname__ZPURPLEZCYANZBLUEZLBLUEZGREENZLGREENZYELLOWZREDLREDZBOLDZ UNDERLINEENDr r S/global/cfs/cdirs/e3sm/www/hass877/share/emis_processing/FlexEmis/src/ggen/utils.pyr src@seZdZddZddZdS) HidePrintcCstj|_ttjdt_dS)Nw)sysstdout_stdoutopenosdevnull)selfr r r __enter__szHidePrint.__enter__cCstj|jt_dS)N)rrcloser)rexc_typeexc_valexc_tbr r r__exit__s  zHidePrint.__exit__N)rr r rrr r r rrs rcCs(|dkrtd}|}|St|}|S)N.)rabsolute)pathpdir_pathr r r get_dir_path!s r$cCs6ttt|dkrttjd|dtjdS)Nr z does not exist!)lenglobstr Exceptionrr r r!r r rchecker)sr+cCs6tj|stdt|dtt|dSdS)Nr%z doesn't exist. Creating one... )rr!existsprintr(makedirsr*r r rmake_dir-s r/cCs"t|jdD]}|d|qdS)Nzgot line from sub: %r)iterreadlineinfo)r"loggerliner r rlog_out2sr6@ cCsd}g}g}d}|D],}|tt|jdd}||kr$||n||d}||g}|d7}q t||ddkrJ|||dt|t|krW|||S)a Splits a list of files based on the file sizes to handle memory issues in multiprocessing. Parameters ---------- filelist : List Original list of files. Returns ------- listoflists : List Split list of lists of files. riN)rstatr(st_sizeappendr&)filelistlimitZ size_of_fileZ listoflistsZ new_filelistifiler r r get_sizes7s$     r@rFc Cstd}t|}|d|d|r't|tttdd}| \}}|j St|tttdd}|j || \}}|d}|D]} || qC|j S)Nzlog.ggenz [cmd]: r%T)rstdinstderruniversal_newlines) logging getLoggershlexsplitr3rrrr communicaterAwrite returncode) cmdinphider4Z cmd_splitr"op_itemsr5r r r exec_shell[s       rQcs|j}t|j}||}tdg|dd|ddkddgf}t|dd|ddk}|j|fddt |ddd|ddddDS)NFr8csg|] \}}||qSr r ).0r>jIr r usz)group_duplicate_index..) valuesnplexsortT concatenateall flatnonzeroindextolistzip)dfasidxbmidxr rUrgroup_duplicate_indexms .0ri)r7)rF)pathlibrrrrD subprocessrrrrrFnumpyrZr'rrr$r+r/r6r@rQrir r r rs"    $