B #=6\-@sddlmZmZmZmZddlmZmZddlZ Gddde Z Gddde Z Gdd d e Z Gd d d e ZGd d d e ZGddde ZGddde ZGddde ZGddde ZdS))absolute_importdivisionprint_functionunicode_literals)coreutilsNc@seZdZdZdZdS)RegularizationByZafter_optimizerZon_lossN)__name__ __module__ __qualname__ZAFTER_OPTIMIZERZON_LOSSr r )kEpsilon)selfr r r __init__szRegularizer.__init__NcCsvt|tjsttt}||ks>td|j ||d|}t ||sbtd|j |t ||||||S)Nz>Regularizer of type {} is called with invalid by={}, not in {}Z_run_z5Regularizer of type {} does not implement function {}) isinstancerZ BlobReferenceAssertionErrorrZEnumClassKeyValsrvaluesformat __class__hasattrgetattr)rnetparam_init_netparamgradZbyZby_enumZrun_funcr r r __call__s   zRegularizer.__call__cCsdS)Nr )rrrrrr r r _run_on_loss'szRegularizer._run_on_losscCsdS)Nr )rrrrrr r r _run_after_optimizer*sz Regularizer._run_after_optimizerFc Cst|dk r|s|r||jn|}|dk r8|s.|r8||jn|}t|tjrV||j|jgn|g} |j| |g||ddS)N)minmax)rrr GradientSliceindicesrZ EnsureClipped) rrrrr r! open_range left_open right_openZ input_blobsr r r _ensure_clipped-s zRegularizer._ensure_clipped)NN)N)NNNFFF)r r r rrrrr'r r r r rs  rcs&eZdZfddZdddZZS)L1Normcs(tt||dkstd||_dS)Nrz6factor ahead of regularization should be 0 or positive)superr(rr reg_lambda)rr*)rr r rKszL1Norm.__init__NcCs<||d}|j|g|gdd|j|g|g|jd|S)NZ_l1_regularization)p)scale)NextScopedBlobLpNormScaler*)rrrrr output_blobr r r rQszL1Norm._run_on_loss)N)r r r rr __classcell__r r )rr r(Js r(cs&eZdZfddZdddZZS)L2Normcs(tt||dkstd||_dS)Nrz6factor ahead of regularization should be 0 or positive)r)r3rrr*)rr*)rr r rYszL2Norm.__init__NcCs<||d}|j|g|gdd|j|g|g|jd|S)NZ_l2_regularization)r,)r-)r.r/r0r*)rrrrrr1r r r r_szL2Norm._run_on_loss)N)r r r rrr2r r )rr r3Xs r3cs&eZdZdfdd ZddZZS)MaxNorm?cstt|||_dS)N)r)r5rnorm)rr7)rr r rgszMaxNorm.__init__cCsL|jdkstdt|tjr@|j||j|jg|gd|jdntddS)Nrznorm should be bigger than 0.T) use_max_normr7z-MaxNorm is not supported for dense parameters) r7rrrr"SparseNormalizer#rNotImplementedError)rrrrrr r r rks   zMaxNorm._run_after_optimizer)r6)r r r rrr2r r )rr r5fsr5cs&eZdZdfdd ZddZZS) ConstantNorm?cstt|||_dS)N)r)r;rr7)rr7)rr r ryszConstantNorm.__init__cCsL|jdkstdt|tjr@|j||j|jg|gd|jdntddS)Nrznorm should be bigger than 0.F)r8r7z2ConstantNorm is not supported for dense parameters) r7rrrr"r9r#rr:)rrrrrr r r r}s   z!ConstantNorm._run_after_optimizer)r<)r r r rrr2r r )rr r;xsr;cs4eZdZdZd fdd Zd ddZdd ZZS) LogBarrierzr Wright, S., & Nocedal, J. (1999). Numerical optimization. Springer Science, 35(67-68), 7. Chapter 19 invNcs>tt||dkstd||_||_|p6ddd|_dS)z discount is a positive weight that is decreasing, and here it is implemented similar to the learning rate. It is specified by a learning rate policy and corresponding options rz6factor ahead of regularization should be 0 or positiveg?)gammapowerN)r)r=rrr*discount_policydiscount_options)rr*rArB)rr r rs zLogBarrier.__init__c Cst||}||d}|j|g|gf|j |jd|j||d}|j|g|g|jd||d}| |g|g||d} | |g| g||d} |j | |g| gdd | S) NZ_log_barrier_discount)Zbase_lrpolicyZ_non_neg)r _logZ_log_sumZ _log_barrierr+) broadcast) rZBuildUniqueMutexIterr.Z LearningRater*rArBZCliprLogZ SumElementsMul) rrrrr iterationZdiscountZ param_non_negZ param_logZ param_log_sumr1r r r rs"  zLogBarrier._run_on_losscCs|j|||ddddS)NrT)r r$)r')rrrrrr r r rszLogBarrier._run_after_optimizer)r>N)N)r r r __doc__rrrr2r r )rr r=s r=cs*eZdZdZdfdd ZddZZS) BoundedGradientProjectionzr Wright, S., & Nocedal, J. (1999). Numerical optimization. Springer Science, 35(67-68), 7. Chapter 16 NFcstt||dk rt|nd}|dk r2t|nd}|dk rFt|n|j}|dksdtdj|d|dks|dks||r~|nd||r|ndkstdj|||rdnd|rdnd |d ||_||_||_||_ ||_ dS) Nrz2Bounded Gradient Projection with invalid eps={eps})epsgzLBounded Gradient Projection with invalid {lp}ub={ub}, lb={lb}{rp}, eps={eps}([)])lbublprprK) r)rJrfloatrrrr%r&rPrQ)rrPrQr%r&epsilon)rr r rs*    z"BoundedGradientProjection.__init__c Cs$|j||||j|j|j|jddS)N)r r!r%r&)r'rPrQr%r&)rrrrrr r r rsz.BoundedGradientProjection._run_after_optimizer)NNFFN)r r r rIrrr2r r )rr rJs rJcs,eZdZdZdfdd Zd ddZZS) GroupL1Norma Scardapane, Simone, et al. "Group sparse regularization for deep neural networks." Neurocomputing 241 (2017): 81-89. This regularizer computes l1 norm of a weight matrix based on groups. There are essentially three stages in the computation: 1. Compute the l2 norm on all the members of each group 2. Scale each l2 norm by the size of each group 3. Compute the l1 norm of the scaled l2 norms rcsFtt||dkstdt|ts0td||_||_||_dS)a Args: reg_lambda: The weight of the regularization term. groups: A list of integers describing the size of each group. The length of the list is the number of groups. Optional Args: stabilizing_val: The computation of GroupL1Norm involves the Sqrt operator. When values are small, its gradient can be numerically unstable and causing gradient explosion. Adding this term to stabilize gradient calculation. Recommended value of this term is 1e-8, but it depends on the specific scenarios. If the implementation of the gradient operator of Sqrt has taken into stability into consideration, this term won't be necessary. rz-regularization weight should be 0 or positivezgroups needs to be a listN) r)rVrrrlistr*groupsstabilizing_val)rr*rXrY)rr r rs zGroupL1Norm.__init__Nc Cs||}|j|dgdd}|||jgdt|jg|jdg}|jrl|j||jgd|jdg|gdd| |}| ||j gt|jgt |j|jdgdg} |j| dgdd } | S) a Args: param: The input blob to regularize. It should be a weight matrix blob with shape (output_dim, input_dim). input_dim should be equal to the sum of self.groups. Returns: group_l1_norm: The output blob after applying regularization. These are the steps of computation: 1. square all elements 2. sum by row 3. lengthssum by group 4. square_root all elements 5. normalize each group based on group size 6. compute l1 norm of each group 7. scale the result with the regularization lambda r)ZaxesZkeepdimsr+)shaper)value)rEZnormalized_l2_norm_scaledZ group_l1_nrom)r,)ZSqrZ ReduceSumZ LengthsSumZGivenTensorIntFilllenrXrYZAddZ ConstantFillZSqrtrGZGivenTensorFillnpsqrtr*r/) rrrrrZsquaredZ reduced_sumZ lengths_sumr^Z l2_scaledZ group_l1_normr r r rs*   zGroupL1Norm._run_on_loss)r)N)r r r rIrrr2r r )rr rVs rV) __future__rrrrZ caffe2.pythonrrZnumpyr]objectrrr(r3r5r;r=rJrVr r r r s<.3