Allow flag or file based configuration by rolandshoemaker · Pull Request #620 · google/trillian

rolandshoemaker · 2017-05-17T01:56:08Z

Initial implementation, slightly less hacky than I originally anticipated.

Fixes #578.

pphaneuf · 2017-05-17T10:50:04Z

Could you please rebase this on top of master, rather than merge it in? Thanks!

codingllama

Looks good to me, I have mostly nits / suggestions.

Would you mind applying similar changes to server/vmap/trillian_map_server/main.go too? It's fairly small and the only missing entry point.

codingllama · 2017-05-17T10:42:17Z

examples/ct/ct_server/main.go

-	etcdServers       = flag.String("etcd_servers", "", "A comma-separated list of etcd servers")
-)
+type config struct {
+	Host                 string


nit: As the struct is private, the fields should possibly be private too. (It seems the large majority of the existing code follows this convention, apart from 1 outlier that I found.)

The Golang JSON package requires that fields be public if you want to unmarshal into them.

Right, I missed that. Would you mind adding a comment? I.e.:

// config represents all configurable settings of {CT,Log,Map} server. // Fields may be set either via flag or a configuration file in JSON format. // Fields must be public due to a requirement of json.Unmarshal.

Not everyone is going to use config files, so a "stylistic" change might break you and leave us none the wiser.

Ditto for others.

codingllama · 2017-05-17T10:43:22Z

examples/ct/ct_server/main.go

+	if *configFlag != "" {
+		// Only allow either --configFlag OR any other flags
+		if flag.CommandLine.NFlag() > 2 {
+			glog.Exit("Cannot use --configFlag with any other flags")


nit: --config instead of --configFlag

codingllama · 2017-05-17T10:46:34Z

server/trillian_log_server/main.go

-	etcdService         = flag.String("etcd_service", "trillian-log", "Service name to announce ourselves under")
-)
+type config struct {
+	MySQLURI            string


Ditto for private struct / public fields.

Same for other files.

codingllama · 2017-05-17T10:47:05Z

server/trillian_log_server/main.go

+	if *configFlag != "" {
+		// Only allow either --configFlag OR any other flags
+		if flag.CommandLine.NFlag() > 2 {
+			glog.Exit("Cannot use --configFlag with any other flags")


Ditto for --config.

Same for other files.

codingllama · 2017-05-17T10:50:36Z

util/config.go

+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+package util


WDYT about moving this to a configs package? I kind of dislike util on principle.

codingllama · 2017-05-17T10:50:51Z

util/config.go

+}
+
+// UnmarshalJSON parses a time.Duration field from a string
+func (cd ConfigDuration) UnmarshalJSON(data []byte) error {


Would you mind including a test for this?

codingllama · 2017-05-17T10:57:41Z

examples/ct/ct_server/main.go


-	if *maxGetEntriesFlag > 0 {
-		ct.MaxGetEntriesAllowed = *maxGetEntriesFlag
+	if *configFlag != "" {


Have you considered moving this block to a shared function, possibly under the configs package I suggested below?

codingllama · 2017-05-17T10:59:12Z

util/config.go

+
+// ConfigDuration is an abstraction used to simplifying unmarshalling
+// json into structs that contain time.Duration fields
+type ConfigDuration struct {


Maybe rename to JsonDuration?

Please document the expected marshaled format as well (a mention to time.ParseDuration would be good enough, IMO).

rolandshoemaker · 2017-05-17T18:24:41Z

I've run into a bit of a problem while testing these changes, github.com/golang/glog adds a set of flags related to logging which break the test that checks if you are providing --config and other flags because they aren't part of the per binary configs. It's entirely plausible that you'd want to provide --config and --log_dir or something but with the current flag.CommandLine.NFlags() > 1 test you are prevented from doing this.

Not entirely sure what the best way around this is at the moment, going to drink some coffee and see if I can come up with a good solution.

rolandshoemaker · 2017-05-17T18:37:03Z

(this idea doesn't work)

Post caffeine: easiest solution is probably to just make the per-binary flags use their own flag.FlagSet and let glog use the default flat.CommandLine set so we can still test for --config and other binary specific flags.

Will require some rewording of the --config usage text to specify you can use it and the glog flags but that should be fine. It's a little annoying that you won't be able to specify everything via the config file but meh ¯_(ツ)_/¯.

gdbelvin · 2017-05-17T19:07:28Z

I know I'm a bit late to this conversation, but while we're discussing flags and configurations, have we thought about configuring through environment variables? Many cluster management products use a .env file to supply environment variables to server binaries ala the 12 factor app model

rolandshoemaker · 2017-05-17T21:59:07Z

Blerp, current solution doesn't work: I forgot that flag doesn't like parsing flags it doesn't know and the only other option is basically 'just allow anything, even badly formatted flag arguments'. There are some really hacky ways around this by for instance changing the error handling for flag.CommandLine and then adding the glog flags to the per-binary flag.FlagSet but, guh, it's not very clean.

One super simple option I can think of that would make life significantly easier (and allow all configuration to happen via flags or files) would be to fork github.com/golang/glog and remove the flags it registers and move the configuration to a Setup or Init method that has to be called before logging happens. This would preserve the same properties the package already has (i.e. you currently have to call flag.Parse before you can do any logging) and remove all of the annoying side effects of a package magically registering its own flags that you don't have control over...

I know I'm a bit late to this conversation, but while we're discussing flags and configurations, have we thought about configuring through environment variables? Many cluster management products use a .env file to supply environment variables to server binaries ala the 12 factor app model

@gdbelvin This would be a viable solution but I personally don't super like it as a method of configuration, you get the benefits of having everything in a file but have to worry about when the last time you sourced it was/if there the current ENV is out of sync with what is in the file etc. It would also require writing some reflect code or something to simplify pulling from vars unless each binary just had a big section for populating variables.

codingllama · 2017-05-18T11:06:30Z

The case of flags defined outside the Trillian codebase is an interesting one, because it seems to force us to a mixed configuration. @rolandshoemaker, what's your deployment plan in regards to those flags? Should they be defined in your configuration object as well?

About implementation options, I don't think this is a strong enough reason to fork glog. I would suggest something in those lines instead:

allowedFlags := map[string]bool{
  "logtostderr":      true,
  "alsologtostderr":  true,
  "stderrthreshold":  true,
  "log_dir":          true,
  "log_backtrace_at": true,
  "v":                true,
  "vmodule":          true,
}
for _, arg := range os.Args[1:] {
  var flag string
  switch {
  case strings.HasPrefix(arg, "--"):
    flag = arg[2:]
  case strings.HasPrefix(arg, "-"):
    flag = arg[1:]
  default:
    continue // Ignore non-flags
  }
  if index := strings.Index(flag, "="); index != -1 {
    flag = flag[0:index]
  }
  if _, ok := allowedFlags[flag]; !ok {
    return fmt.Errorf("flag not allowed in conjunction with --config: %v", arg)
  }
}

(Warning: I haven't tested / executed this.)

Another option, if we'd like other flags to be configurable via config files, is to add them to the configuration and call flag.Set for each flag. The caveat is that logging won't work as expected up until that point.

I'm OK with dropping the --config "exclusiveness" requirement if people think those options are not worth it, though.

codingllama

Please have a look at the Travis failure as well.

@RJPercival had an interesting suggestion: we could use flag.CommandLine.Parse directly on the config file, which would preclude all the json-parsing logic and config struct intermediates. That would change the format from json to a "flags file", though (one flag per line, possibly).

At that point, IMO, we might just have a .sh file wrapping the entry points and set the flags there, which would avoid the entire config file business. IDK much about your deployment environment, so I'll leave that to you.

codingllama · 2017-05-18T11:08:47Z

configs/configs_test.go

+		Dur JSONDuration
+	}
+
+	a := []byte(`{"dur":"10s"}`)


Would mind refactoring the test into a table-driven style?

https://github.com/golang/go/wiki/TableDrivenTests

codingllama · 2017-05-18T11:10:15Z

configs/configs.go

+// UnmarshalJSON parses a time.Duration field from a string
+func (jd *JSONDuration) UnmarshalJSON(data []byte) error {
+	s := ""
+	err := json.Unmarshal(data, &s)


nit: Scope err in the if? if err := json.Unmarshal(data, &s); err != nil { ... }

codingllama · 2017-05-18T11:20:19Z

examples/ct/ct_server/main.go

-		ct.MaxGetEntriesAllowed = *maxGetEntriesFlag
+	if *configFlag != "" {
+		// Only allow either --config OR any other flags
+		if flags.NFlag() > 1 {


Could we move the --config "exclusiveness" check to the shared function as well? (Whatever form it might take, it is.)

codingllama · 2017-05-18T11:22:38Z

examples/ct/ct_server/main.go

-	etcdServers       = flag.String("etcd_servers", "", "A comma-separated list of etcd servers")
-)
+type config struct {
+	Host                 string


Right, I missed that. Would you mind adding a comment? I.e.:

// config represents all configurable settings of {CT,Log,Map} server. // Fields may be set either via flag or a configuration file in JSON format. // Fields must be public due to a requirement of json.Unmarshal.

Not everyone is going to use config files, so a "stylistic" change might break you and leave us none the wiser.

Ditto for others.

rolandshoemaker · 2017-05-18T22:28:35Z

@RJPercival had an interesting suggestion: we could use flag.CommandLine.Parse directly on the config file, which would preclude all the json-parsing logic and config struct intermediates. That would change the format from json to a "flags file", though (one flag per line, possibly).

At that point, IMO, we might just have a .sh file wrapping the entry points and set the flags there, which would avoid the entire config file business. IDK much about your deployment environment, so I'll leave that to you.

I wouldn't be explicitly opposed to this, generating a file is generating a file, the only real difference in terms of management is that it's a little simpler to verify that the file has been properly formatted when it's JSON since we can use basically any parser on the config management node to verify the file.

I think this approach does have a few downsides though, mainly it would make the config take precedence over flags when both are provided (which I think is basically the opposite of expected behavior) and it would require either a bunch of extra parsing code (to properly split up the flags/flag arguments in the way that flag.Parse expects them to be provided or strict rules about how flags should be provided (i.e. each flag must be on a new line and use the -flag=value format etc etc) in the files.

codingllama

Many thanks for addressing all comments, Roland. Could you rebase on master again after you address the next round?

codingllama · 2017-05-19T09:31:13Z

configs/configs.go

+// be set from a file.
+func SetGLogFlags(gf GLogFlags) error {
+	if gf.LogToStderr {
+		if err := flag.Set("logtostderr", fmt.Sprintf("%t", gf.LogToStderr)); err != nil {


Suggestion: Just fmt.Sprint(gf.LogToStderr) should work.

Ditto for alsologtostderr and v.

codingllama · 2017-05-19T09:32:42Z

configs/configs_test.go

+		wantErr      string
+		wantDuration time.Duration
+	}{
+		{[]byte(`{"dur":"10s"}`), "", time.Second * 10},


nit: Please use named literals on the struct fields. This way you don't have to specify default/unrelated values as well.

codingllama · 2017-05-19T09:33:05Z

configs/configs_test.go

-	var s struct {
-		Dur JSONDuration
+	testCases := []struct {
+		json         []byte


Suggestion: You could make this a string and move the cast to the test body.

codingllama · 2017-05-19T09:34:35Z

configs/configs_test.go

 	"time"
 )

 func TestJSONDuration(t *testing.T) {


Maybe TestJSONDuration_ UnmarshalJSON, so it's clear which method is being tested?

codingllama · 2017-05-19T09:36:50Z

configs/configs_test.go

-		Dur JSONDuration
+	testCases := []struct {
+		json         []byte
+		wantErr      string


This is tying the test to error messages we don't own. I'd be ok with an wantErr bool and a err != nil check. We have a bunch of tests like that, eg, noop_test.go.

codingllama · 2017-05-19T09:42:52Z

configs/configs_test.go

+			var s struct {
+				Dur JSONDuration
+			}
+			err := json.Unmarshal(tc.json, &s)


Suggestion:

err := json.Unmarshal(tc.json, &s) switch hasErr, wantErr := err != nil, tc.wantErr != ""; { case hasErr != wantErr: t.Errorf(...) case hasErr && err.Error() != tc.wantErr: t.Errorf(...) }

Although, if you take the comment above about the error string, the case comparing the messages goes away, so we could simplify it further to an if.

codingllama · 2017-05-19T09:44:53Z

configs/configs_test.go

+			} else if err == nil && tc.wantErr != "" {
+				t.Errorf("json.Unmarshal didn't fail: want %q", tc.wantErr)
+			}
+			if tc.wantDuration != 0 && tc.wantDuration != s.Dur.Duration {


I think you can drop the tc.wantDuration != 0 condition and it'll still work, as zero should be the default on s.Dur.Duration.

codingllama · 2017-05-19T09:46:23Z

examples/ct/ct_server/main.go

 	"google.golang.org/grpc/naming"
 )

+// config represents all configurable settings of {CT,Log,Map} server and Log signer.


nit: Replace "{CT,Log,Map} server and Log signer" with just "CT server". This particular config is exclusive to CT server (and so are the others to their respective binaries).

Ditto for others.

codingllama · 2017-05-19T09:55:49Z

examples/ct/ct_server/main.go

 		if err := configs.LoadConfig(*configFlag, &c); err != nil {
 			glog.Exit(err)
 		}
+		if err := configs.SetGLogFlags(c.GLogFlags); err != nil {


Suggestion: Could we move this to LoadConfig?

If we make changes similar to the ones below I reckon it should work:

// Change SetGLogFlags signature. func (gf *GLogFlags) SetGLogFlags() error { ... } type glogFlagsSetter interface { SetGLogFlags() error } func LoadConfig(path string, obj interface{}) error { // ... if setter, ok := obj.(glogFlagsSetter); ok { if err := setter.SetGlogFlags(); err != nil { return err } } }

codingllama · 2017-05-19T09:59:31Z

configs/configs.go

+
+// GLogFlags contains the values of the flags that github.com/golang/glog
+// registers.
+type GLogFlags struct {


Suggestion: I'd be tempted to remove call this "CommonFlags" (or something in that spirit), so if we add non-glog flags to it later we won't have to rename it. (Ditto for SetGlogFlags).

RJPercival · 2017-05-19T16:49:02Z

To illustrate my proposal of reading flags from a file, see RJPercival@c6d3fc8. This supports things like glog without any special code being required. Parsing is off-loaded to a shellwords package (admittedly, this means an extra dependency). It supports either letting command-line arguments take precedence or raising an error if any command-line arguments other than "--config" are supplied.

Having said all that, it doesn't give you much over just having a wrapper script around the binary with whatever flags you'd like. Can you explain to me what this PR gives us? You mentioned it allows you to "use basically any parser on the config management node to verify the [config]". What sort of verification were you planning to do?

rolandshoemaker · 2017-05-19T19:44:53Z

To illustrate my proposal of reading flags from a file, see RJPercival/trillian@c6d3fc8. This supports things like glog without any special code being required. Parsing is off-loaded to a shellwords package (admittedly, this means an extra dependency). It supports either letting command-line arguments take precedence or raising an error if any command-line arguments other than "--config" are supplied.

Yup, this is a MUCH cleaner implementation. Looking at the code I think this is probably the best way forward if we do want to support some kind of file based configuration.

Can you explain to me what this PR gives us? You mentioned it allows you to "use basically any parser on the config management node to verify the [config]". What sort of verification were you planning to do?

By verification I really just meant a super basic sanity check, for instance in our existing deployment of other software we are able to simply parse the JSON config after it has been generated on the config management node from a template in order to verify that it is at least properly formatted.

Having said all that, it doesn't give you much over just having a wrapper script around the binary with whatever flags you'd like.

After having deployed trillian in its current form a bunch recently I've pretty much come around to this being a basically fine solution. I think the only thing I'd really like file based configuration for at this point is basic secret isolation so that we don't have PEM key passwords or MySQL credentials in the process list but ¯_(ツ)_/¯.

Conclusion: I think I'm convinced at this point that this JSON-based solution ends up being very heavyweight for relatively little benefit (we could have native lists or maps! but... nowhere uses that now so eh). I still think having the separation of config from service is nice to have and provides for basic secret isolation but is not necessary.

At this point I think there are two real choices: (a.) close this and #578 out and go on our merry way and keep everything as-is or (b.) implement the solution from RJPercival/trillian@c6d3fc8. I think I'd probably go with (b.) but could really go either way to be honest.

RJPercival · 2017-05-22T08:52:58Z

I can appreciate not wanting certain flags visible in the process list; that's a good reason to have a config file. Would you like to pick up RJPercival@c6d3fc8, finish it off and see how it works for your use case?

rolandshoemaker · 2017-05-22T19:26:18Z

I can appreciate not wanting certain flags visible in the process list; that's a good reason to have a config file. Would you like to pick up RJPercival/trillian@c6d3fc8, finish it off and see how it works for your use case?

Sounds good to me.

…erver

RJPercival · 2017-05-23T13:52:14Z

cmd/flags.go

+	"flag"
+	"io/ioutil"
+
+	"github.com/mattn/go-shellwords"


Having looked at what is available within Google, could we swap this for https://bitbucket.org/creachadair/shell instead please? We'll lose the ability to use environment variables in the flag file, which is a bit of a shame, but it saves having to get https://github.com/mattn/go-shellwords reviewed by our security team. I can go down that road if supporting environment variables is important to you though.

Works for me.

…ir/shell

RJPercival · 2017-05-23T22:07:08Z

server/vmap/trillian_map_server/main.go

+	if *configFile != "" {
+		if err := cmd.ParseFlagFile(*configFile); err != nil {
+			fmt.Fprintf(os.Stderr, "Failed to parse %v: %v\n", *configFile, err)
+			os.Exit(1)


glog.Exitf() rather than fmt.Fprintf(), os.Exit(1) would be more consistent with the rest of the file.

RJPercival · 2017-05-23T22:08:17Z

server/trillian_log_signer/main.go

+	if *configFile != "" {
+		if err := cmd.ParseFlagFile(*configFile); err != nil {
+			fmt.Fprintf(os.Stderr, "Failed to parse %v: %v\n", *configFile, err)
+			os.Exit(1)


glog.Exitf() would be better here.

RJPercival · 2017-05-23T22:08:32Z

server/trillian_log_server/main.go

+	if *configFile != "" {
+		if err := cmd.ParseFlagFile(*configFile); err != nil {
+			fmt.Fprintf(os.Stderr, "Failed to parse %v: %v\n", *configFile, err)
+			os.Exit(1)


glog.Exitf() would be better here.

RJPercival · 2017-05-23T22:13:32Z

cmd/flags.go

+// path. Re-calls flag.Parse() after parsing the flags in the file
+// so that flags provided on the command line take precedence over
+// flags provided in the file.
+func ParseFlagFile(path string) error {


This func could do with some tests.

RJPercival · 2017-05-23T22:16:10Z

cmd/flags.go

+	}
+
+	// Expand any environment variables in the file
+	flagsString := os.ExpandEnv(string(file))


Expanding envvars before splitting might lead to unintended splitting if the envvar value contains spaces, quotation marks, etc. Unfortunately, you might have to instead call os.ExpandEnv() on each of args. A test case could confirm this.

RJPercival · 2017-05-23T22:16:48Z

cmd/flags.go

+	}
+
+	err = flag.CommandLine.Parse(args)
+	if err != nil {


These two lines could be combined, e.g. if err := flag.CommandLine.Parse(args); err != nil {

RJPercival

Once these comments have been addressed, it should be good to merge.

RJPercival · 2017-05-25T08:23:30Z

cmd/flags_test.go

+		},
+	}
+
+	initalArgs := os.Args[:]


typo: s/initalArgs/initialArgs/

RJPercival · 2017-05-25T08:30:50Z

cmd/flags_test.go

+			}
+		}
+		err := parseFlags(tc.contents)
+		if err != nil {


Could combine the above two lines.

RJPercival · 2017-05-25T08:32:58Z

cmd/flags_test.go

+	for _, tc := range tests {
+		a, b = "", ""
+		os.Args = initalArgs[:]
+		if len(tc.cliArgs) > 0 {


No need for this check - os.Args = append(os.Args, tc.cliArgs...) will simply be a noop if len(tc.cliArgs) == 0.

RJPercival · 2017-05-25T08:34:13Z

cmd/flags_test.go

+	initalArgs := os.Args[:]
+	for _, tc := range tests {
+		a, b = "", ""
+		os.Args = initalArgs[:]


Lines 80-83 could be condensed into os.Args = append(initialArgs, tc.cliArgs...).

RJPercival · 2017-05-25T08:38:12Z

cmd/flags_test.go

+
+	flag.CommandLine.Init(os.Args[0], flag.ContinueOnError)
+
+	tests := []struct {


Please add a name field and include it in all of the t.Errorf() calls so that it's easy to identify which test case failed.

RJPercival

To save time, and because my comments were minor, I've taken the liberty of addressing them myself and pushing a commit to your branch. Feel free to amend that commit as you like, then you can squash and merge this PR.

Superseded by a more recent review.

rolandshoemaker · 2017-05-25T18:49:36Z

Changes look fine to me, thanks! Side-note: looks like TestUpdateRoot may be flaky.

rolandshoemaker · 2017-05-25T18:52:48Z

(Oh also, I don't have perms on this repo so someone else should merge/kick the travis build)

RJPercival · 2017-05-25T21:32:24Z

I've given it a kick. Yes I've noticed client_test.go being pretty flaky, likely because of the sleep call it's relying on. @gdbelvin, any chance you could take a look at that?

RJPercival · 2017-05-25T22:54:39Z

The second build hit issue #617 (I'll look into that). Third time is hopefully the charm...

googlebot added the cla: yes label May 17, 2017

codingllama self-requested a review May 17, 2017 10:42

codingllama suggested changes May 17, 2017

View reviewed changes

rolandshoemaker force-pushed the conf-files branch from 04b4c90 to 8b48ccf Compare May 17, 2017 18:19

codingllama assigned rolandshoemaker and codingllama May 18, 2017

RJPercival self-requested a review May 18, 2017 10:11

codingllama previously requested changes May 18, 2017

View reviewed changes

codingllama reviewed May 19, 2017

View reviewed changes

Rob Percival and others added 2 commits May 22, 2017 17:56

Example of loading flags from a file

4712f74

Expand config file functionality to log_server, log_signer, and map_s…

ddcc786

…erver

rolandshoemaker force-pushed the conf-files branch from 7dd8a47 to ddcc786 Compare May 23, 2017 00:58

Roland Shoemaker added 3 commits May 22, 2017 18:00

Add comment to ParseFlagFile

4c06b2e

Add cmd/flags.go license header

a6656ac

Re-add go-shellwords import

cc9922c

rolandshoemaker force-pushed the conf-files branch from 6c2ff65 to cc9922c Compare May 23, 2017 01:23

RJPercival reviewed May 23, 2017

View reviewed changes

Roland Shoemaker added 2 commits May 23, 2017 11:40

Switch from github.com/mattn/go-shellwords to bitbucket.org/creachada…

4fd46e5

…ir/shell

Expand env vars in flag file if present

d2cc80d

RJPercival reviewed May 23, 2017

View reviewed changes

RJPercival suggested changes May 23, 2017

View reviewed changes

Roland Shoemaker added 2 commits May 24, 2017 14:38

Add flag file parsing tests and env cleanups

54850a1

Use glog.Exitf instead of fmt.Fprintf and os.Exit

661daaf

RJPercival reviewed May 25, 2017

View reviewed changes

cmd/flags_test.go Outdated

},

}

initalArgs := os.Args[:]

Copy link

Contributor

RJPercival May 25, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typo: s/initalArgs/initialArgs/

RJPercival reviewed May 25, 2017

View reviewed changes

cmd/flags_test.go Outdated

}

}

err := parseFlags(tc.contents)

if err != nil {

Copy link

Contributor

RJPercival May 25, 2017

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could combine the above two lines.

RJPercival reviewed May 25, 2017

View reviewed changes

Address review comments

c49bc08

RJPercival approved these changes May 25, 2017

View reviewed changes

RJPercival merged commit 6713433 into google:master May 26, 2017

RJPercival mentioned this pull request May 26, 2017

Move towards file based configuration instead of flag based #578

Closed

This was referenced Jun 5, 2017

Docker swarm google/keytransparency#578

Merged

Allow file based configuration google/keytransparency#597

Closed


		flag.CommandLine.Init(os.Args[0], flag.ContinueOnError)

		tests := []struct {

Comments

Conversation

rolandshoemaker commented May 17, 2017

Uh oh!

pphaneuf commented May 17, 2017

Uh oh!

codingllama left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rolandshoemaker commented May 17, 2017

Uh oh!

rolandshoemaker commented May 17, 2017 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gdbelvin commented May 17, 2017

Uh oh!

rolandshoemaker commented May 17, 2017

Uh oh!

codingllama commented May 18, 2017

Uh oh!

codingllama left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

rolandshoemaker commented May 18, 2017

Uh oh!

codingllama left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

RJPercival commented May 19, 2017

Uh oh!

rolandshoemaker commented May 17, 2017 •

edited

Loading

RJPercival May 23, 2017 •

edited

Loading