Discussion of risk scores

[tomschenkjr]

To frame the discussion, there are a few different types of risk metrics we can use for the project. I think we have been considering either an odds ratio or relative risk ratio or one of the many permutations.

The problem is the above ratios are mostly designed around experimental data, for instance, that uses treatment and control groups or cohort studies of individuals. However, our use case is for point predictions--the risk of an individual compared to the group. The typical intent of odds ratios are looking whether or not the relative risk is higher for one group compared to the other. If relatively higher, then it is bad.

This may be hard to translate this to an individual prediction. The incidence of lead poisoning is low. So, what is an "unacceptable" odds ratio? Anything more than the average? But why accept that -- aren't we trying to eliminate any incidence? We need to explicitly define our objective here.

Instead of relative risks, we could use an absolute risk such as the raw probability. Again, this may not be useful for medical staff where they will likely and always see low numeric numbers (e.g., 5%) that represent higher-than-average risk.

Of course, we could interpret raw scores into quantiles (e.g., top 25%). But again, will need a business rule to determine thresholds on when follow-ups should be conducted.

We could potentially reference known resource limitations. It is expensive and burdensome on families to undergo lead testing. Do we suspect there is a maximum number of additional inspections that could be completed this year? This could correspond to quantiles where we recommend action.

[tomschenkjr]

Below are some notes extracted from the phone conversation during the weekly meeting today.

There are three potential scores to communicate to medical staff:

• Raw score
• Probabilities
• Relative risk
• Quantiles

Raw scores provide no meaningful interpret ability, so that should not be used. Probabilities provide a human-understandable interpretation (e.g., 20%), but clinical staff--and we agreed--that the numbers are too low to be understood as "alarming".

Relative risk and Quantiles are useful because they contextualize it against a baseline. The highest relative risk is likely 5. But, it can be hard to understand at which threshold the relative risk is truly "elevated" because there is no natural maximum value nor easily understood what the "typical" value is. Finally, Quantiles are also desirable because it may be easier to understand and interpret the risk.

We will approach the medical directors to ask them if they prefer Relative Risk or Quantiles. We will use their feedback to finalize the output of risk_score field. In the meantime, @potash will create some charts showing the distribution of scores so we can start understanding the shape of risk.

The risk_score_notes is still subject to more discussion. We have decided there will be two outputs always included: reminder to order blood lead screening tests and to distribute literature on lead paint.

However, the third action on whether to recommend a visual inspection is less clear. Eric's descriptive statistics will help us determine if there is a natural threshold within the data.

Please continue this discussion within this thread.

[RaedMan]

@jtwalsh0 and I discussed quartiles and sketched it out briefly but think the threshold and the output communicated may work and will discuss at next meeting.