Closes #379: Add a task to monitor data source health.

[emtwo]

This still needs some tests, a bit of code refactoring, and changing the task schedule to be less frequent than 30 seconds.

Otherwise, it satisfies the basic requirement to save data source info to redis and output to status.json

[arikfr]

All the code used here except for the change in monitor.py and the schedule for the task, can be placed in its own module. I think it will be easier for you when syncing with upstream (and later easier to move to a plugin).

[emtwo]

@arikfr good point, thank you!

[emtwo]

One important note here is that right now this code assumes that a data source is query-able with basic SQL and that the schema is returned by get_schema() in the same format. I want to look into these things because I suspect there are non-sql data sources which may need to be handled differently.

[emtwo]

I've updated the PR to leverage the existing test_connection() function here: https://github.com/mozilla/redash/blob/master/redash/query_runner/__init__.py#L105

So now it uses the existing no-op queries hardcoded for all data sources by default and has the additional ability to customize queries used for a data source.

In status.json the new field will look something like this:

  "data_sources": {
    "postgres": {
      "SELECT 1": {
        "last_run": 1526571509.069719, 
        "last_run_human": "2018-05-17 15:38:29", 
        "runtime": 0.003139972686767578, 
        "status": "SUCCESS"
      }, 
      "select * from api_keys;": {
        "last_run": 1526570795.251458, 
        "last_run_human": "2018-05-17 15:26:35", 
        "runtime": null, 
        "status": "FAIL"
      }
    }
  }

@robotblake @jezdez what do you think of this approach?

[emtwo]

Added a commit which changes the status.json format a bit, allowing for metadata and keying on data source by ID as well as setting custom health queries by data source ID. Here is what the new format looks like:

  "data_sources": {
    "1": {
      "metadata": {
        "name": "postgres"
      }, 
      "queries": {
        "SELECT 1": {
          "last_run": 1527265788.569606, 
          "last_run_human": "2018-05-25 16:29:48", 
          "runtime": 0.0092010498046875, 
          "status": "SUCCESS"
        }, 
        "select * from users": {
          "last_run": 1527265818.619137, 
          "last_run_human": "2018-05-25 16:30:18", 
          "runtime": null, 
          "status": "FAIL"
        }
      }
    }
},

[jasonthomas]

Can we perform an AND operation on all the query statuses for a given datasource and expose that as an overall status for that datasource? My main reason for doing this is so to reduce the complexity of the monitoring code. From the ops perspective we should consider a datasource in a failed state if one or more queries have failed.

[emtwo]

thanks @jasonthomas for the feedback! Yes I like that idea! How about keeping the data that's currently visible in the json example I showed above for some bonus context, with an additional higher-level status field? For example:

  "data_sources": {
    "1": {
      "status": "FAIL",
      "metadata": {
        "name": "postgres"
      }, 
      "queries": {
        "SELECT 1": {
          "last_run": 1527265788.569606, 
          "last_run_human": "2018-05-25 16:29:48", 
          "runtime": 0.0092010498046875, 
          "status": "SUCCESS"
        }, 
        "select * from users": {
          "last_run": 1527265818.619137, 
          "last_run_human": "2018-05-25 16:30:18", 
          "runtime": null, 
          "status": "FAIL"
        }
      }
    }
},

[jasonthomas]

That works for me. Thanks!

[emtwo]

@jasonthomas I've made that change and also set it to run the heartbeat once every 12 hours (for now). Does that sound reasonable?

[jasonthomas]

I was thinking more of every 5-15 minutes. Can we make it so that is configurable via environment variable? We can make the default 12 hours and ops can override it.

[emtwo]

@jasonthomas I've updated to include an environment variable for the heartbeat frequency. Please let me know if this all looks good or if there's anything else you'd like to see! Thanks!

[jasonthomas]

lgtm!