Skip to content

Comments

solver trace feature to trace weights, loss, diffs and much more#3569

Open
cseward wants to merge 14 commits intoBVLC:masterfrom
elezar:feat_solver_trace
Open

solver trace feature to trace weights, loss, diffs and much more#3569
cseward wants to merge 14 commits intoBVLC:masterfrom
elezar:feat_solver_trace

Conversation

@cseward
Copy link

@cseward cseward commented Jan 19, 2016

This pull request allows a running history to be created of any solver. Only a very small user selected subset of values are saved, but these are enough to visualize the progress of training.

With this pull request, plots tracing the progress of learning can be created that look like this:

test_trace

weight_traces

The changes made were the following:

  • messages for saving solver history are added to caffe.proto
  • options for how solver history is saved (how often, how much) is added to caffe.proto
  • a SolverTrace class is created to manage solver history
  • Solver class is modified to save its pass its state to SolverTrace at appropriate times
  • caffe executable is updated to allow resumption of training with a solver trace file
  • mnist example that creates a solver trace added
  • python script to visualize a solver trace file added

@elezar
Copy link
Contributor

elezar commented Feb 8, 2016

This PR supersedes #2327. @cseward, could you close the previous one.

@elezar
Copy link
Contributor

elezar commented Feb 8, 2016

Looking at #2836, it may be useful to switch to the new (hdf5) format. @erictzeng could you (or someone else) comment on whether the changes that @cseward proposes here are better suited to the new format?

@cseward cseward force-pushed the feat_solver_trace branch from 19cf0a8 to 04af006 Compare April 14, 2016 15:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants