Solver Refactor: Separate files and Change Solver's Type to String

[ronghanghu]

Resolve #2890.

Split solver.cpp into each solver per file, allow self-contained solver development, and provide abstraction for adaptive gradient solvers.

Caffe has individual cpp files for each layer, and in #1694, layer types were changed to strings to allow self-contained layer development. Similar work should be done for solvers, especially given that right now all solver implementations are in solver.cpp and their tests in test_gradient_based_solver.cpp, both 1200+ lines.

Part 1: Solver Refactor

• Split solver implementation into each file per solver, and add solver factory
• Change solver type to strings, add layer-style solver factor and solver registry mechanism
• Implement automatic upgrade for existing solver files
• Update examples and docs

Note: The old enum field solver_type is changed to the new string field type in SolverParameter. Automatic upgrade provided.

Part 2: Solver Abstraction (in another PR)

• Extract common parts in adaptive gradient-based solvers and provide abstract adaptive gradient solver class
• Rewrite solver tests. Split and provide abstraction for solver tests to remove duplicated code.

[ronghanghu]

To provide correct level of solver abstractions, I hope the hierarchy would be

/Solver (abstract class)
/Solver/BaseSGDSolver (abstract class)
/Solver/BaseSGDSolver/SGDSolver
/Solver/BaseSGDSolver/NesterovSolver
/Solver/BaseSGDSolver/AdaptiveSGDSolver (abstract class)
/Solver/BaseSGDSolver/AdaptiveSGDSolver/AdamSolver
/Solver/BaseSGDSolver/AdaptiveSGDSolver/RMSPropSolver
/Solver/BaseSGDSolver/AdaptiveSGDSolver/AdaGradSolver
/Solver/BaseSGDSolver/AdaptiveSGDSolver/AdaDeltaSolver
/Solver/HessianFreeSolver (abstract class)
/Solver/HessianFreeSolver/SomeToBeImplementedHessianFreeSolver

Any suggestions are welcome!

[ronghanghu]

@shelhamer Do you prefer that I address all the above tasks in one PR, or should I split them into separate PRs (I am in favor of the latter)?

[shelhamer]

@ronghanghu you are welcome to do it as a progression but certain tasks need to be done together such as

• Change solver type to strings, add layer-style solver factor and solver registry mechanism
• Implement automatic upgrade for existing solver files

so that each PR yields a correct master. Thank you for bringing order to the solver code!

[ronghanghu]

I feel it is better to split this PR into two separate PRs, one for splitting files and one for solver abstraction. Part 1 should be (almost) completed.

[ronghanghu]

A rebase is needed here after #3204.

[shelhamer]

@ronghanghu sorry about that! I'm happy to do a last review and merge after the rebase so we can have orderly solver code.

[ronghanghu]

Rebased to the latest master

[shelhamer]

Thanks for splitting up the solvers and decoupling development Ronghang!