Demos Applications Components Optimizers Experiments Datasets

Optimization Components

Java 8 Neural Networks with CuDNN and Aparapi

Project maintained by SimiaCryptus Java, CuDNN, and CUDA are others' trademarks. No endorsement is implied.

Optimization Components

These notebooks document the components related to optimization and training.

  1. com.simiacryptus.mindseye.eval.DeltaHoldoverArrayTrainable

    Description: Experimental. The idea behind this class is to track for each row the change in value for an objective function overthe course of a single training epoch. Rows which are observed to have a high delta are inferred to be "interesting"rows, subject to retention when re-sampling training data between epochs.

    1. HoldoverStochasticGradientDescentTest

      Description: The type Holdover stochastic gradient descent run.

      Status: OK

  2. com.simiacryptus.mindseye.eval.L12Normalizer

    Description: Abstract base class for a trainable wrapper that adds per-layer L1 and L2 normalization constants. It allows theimplementing class to choose the coefficients for each layer.

    1. L1NormalizationTest

      Description: The type L 1 normalization run.

      Status: OK

    2. L2NormalizationTest

      Description: The type L 2 normalization run.

      Status: OK

  3. com.simiacryptus.mindseye.eval.SampledArrayTrainable

    Description: This type handles the data selection part of stochastic gradient descent training. Between each epoch, a "reset"method is called to re-sample the training data and pass it to the inner Trainable implementation.

    1. SimpleStochasticGradientDescentTest

      Description: The type Simple stochastic gradient descent run.

      Status: OK

  4. com.simiacryptus.mindseye.opt.line.BisectionSearch

    Description: An exact line search method which ignores the quantity of the derivative, using only sign. Signs are sufficient tofind and detect bracketing conditions. When the solution is bracketed, the next iteration always tests the midpoint.

    1. BisectionLineSearchTest

      Description: The type Bisection line search run.

      Status: OK

  5. com.simiacryptus.mindseye.opt.line.QuadraticSearch

    Description: This exact line search method uses a linear interpolation of the derivative to find the extrema, where dx\/dy = 0.Bracketing conditions are established with logic that largely ignores derivatives, due to heuristic observations.

    1. QuadraticLineSearchTest

      Description: The type Quadratic line search run.

      Status: OK

  6. com.simiacryptus.mindseye.opt.line.StaticLearningRate

    Description: A very basic line search which uses a static rate, searching lower rates when iterations do not result inimprovement.

    1. StaticRateTest

      Description: The type Static rate run.

      Status: OK

  7. com.simiacryptus.mindseye.opt.orient.GradientDescent

    Description: The most basic type of orientation, which uses the raw function gradient.

    1. GDTest

      Description: The type Gd run.

      Status: OK

  8. com.simiacryptus.mindseye.opt.orient.LBFGS

    Description: An implementation of the Limited-Memory Broyden\u2013Fletcher\u2013Goldfarb\u2013Shanno algorithmhttps:\/\/\/wiki\/Limited-memory_BFGS

    1. LBFGSTest

      Description: The type Lbfgs run.

      Status: IterativeStopException

  9. com.simiacryptus.mindseye.opt.orient.MomentumStrategy

    Description: A simple momentum module which uses a cumulative decay algorithm to add a momentum term to any orientation strategy(if it yields a SimpleLineSearch cursor)

    1. MomentumTest

      Description: The type Momentum run.

      Status: OK

  10. com.simiacryptus.mindseye.opt.orient.OwlQn

    Description: Orthant-Wise Limited-memory Quasi-Newton optimization This is a modified L-BFGS algorithm which uses orthant trustregions to bound the cursor path during the line search phase of each iteration

    1. OWLQNTest

      Description: The type Owlqn run.

      Status: OK

  11. com.simiacryptus.mindseye.opt.orient.QQN

    Description: Quadratic Quasi-Newton optimizationThis method hybridizes pure gradient descent with higher-order quasinewton implementations such as L-BFGS. Duringeach iteration, a quadratic curve is interpolated which aligns with the gradient’s direction prediction andintersects with the quasinewton’s optimal point prediction. A simple parameteric quadratic function blends both innercursors into a simple nonlinear path which should combine the stability of both methods.

    1. QQNTest

      Description: The type Qqn run.

      Status: OK

  12. com.simiacryptus.mindseye.opt.orient.RecursiveSubspace

    Description: An recursive optimization strategy which projects the current space into a reduced-dimensional subspace for asub-optimization batch run.

    1. Baseline

      Description: The type Baseline.

      Status: OK

    2. Demo

      Description: The type Demo.

      Status: OK

    3. Normalized

      Description: The type Normalized.

      Status: OK

  13. com.simiacryptus.mindseye.opt.region.AdaptiveTrustSphere

    Description: This trust region uses recent position history to define an ellipsoid volume for the n+1 line search

    1. TrustSphereTest

      Description: The type Trust sphere run.

      Status: OK

  14. com.simiacryptus.mindseye.opt.region.LinearSumConstraint

    Description: This constrains a weight vector based on a single hyperplane which prevents immediate increases to the L1 magnitude.(Note: This region can allow effective L1 increases, if at least one weight changes sign; this allows for our entiresearch space to be reachable.)

    1. LinearSumConstraintTest

      Description: The type Linear sum constraint run.

      Status: OK

  15. com.simiacryptus.mindseye.opt.region.SingleOrthant

    Description: A Single-orthant trust region. These are used in OWL-QN to proven effect in training sparse models where an exactvalue of zero for many weights is desired.

    1. SingleOrthantTrustRegionTest

      Description: The type Single orthant trust region run.

      Status: OK