Improvements to "Discrepancy Source Tier" Pie Charts

[samm82]

I just noticed that the standards tier is the only one with discrepancies between documents by the same author(s) or standards organization(s) (see image below), and I wonder if SWEBOK V3 and V4 could fall under this umbrella. They have different sets of editors, and this field is what appears when citing them with natbib (Bourque and Fairley, 2014 vs. Washizaki, 2024), but since I internally label them as SWEBOK2014 and SWEBOK2024 it wouldn't be hard to consider this edge case. I would likely change the label of this group of discrepancies to "organization(s)" instead of "standards organization(s)". Thoughts?

[JacquesCarette]

I would drop the "same author(s)" part as being too fine a distinction, and make the wording a bit more generic, while keeping the actual definition the same.

[smiths]

The figure looks nice @samm82. I haven't read your text yet where you discuss this figure, but I wonder what conclusions you can draw?

I'll give myself the challenge of looking for some conclusions. 😄 There is a higher percentage of discrepancies within standards (your within a single document category). Is this just a consequence of standards being longer? Since it is a percentage, this is also related to the fact that only 3 of your 6 categories appear in standards. Is this because of how you count discrepancies? Textbooks show discrepancies with standards, but standards don't show discrepancies with textbooks. Is this because the discrepancy is assigned to the textbook because the standard is more trusted? If I remember correctly, that is how you count discrepancies. This way of counting makes sense, but it also (I think) makes comparing between the pie charts not particularly interesting/meaningful. Maybe I'm missing something. All of the above is just a stream of consciousness writing; I haven't thought deeply about what the figure means. 😄

In your writing you need to know exactly what the figure says. You need to have clear conclusions that you can draw from the data presented in the figure. If you already have this, you can ignore my comment. 😄 If not, you need to look at each figure and determine what conclusion it is supporting. It isn't enough just to provide the data in figures; we need to interpret the data for the reader. In many cases the interpretation comes first and then you figure out the figure that will support your conclusion.

For instance, if you really do want to say that standards have more internal inconsistencies (your first category in the legend), then this figure doesn't do the job. We would need to normalize somehow by the size of the document to see if they have more internal inconsistencies per page.

I'm rambling now. I mention this because it is a common problem for student writers to show the data and leave it to the reader to make conclusions from it. 😄

[samm82]

@JacquesCarette I'm really struggling to think of a better generic description; the best I've come up with is "Between documents with the same set of authors".

@smiths Now that I'm looking closer at it, I realize that I explain how this figure is build in the Tools section but don't actually describe what it means in the Discrepancies section where it appears! 😅 The purpose of this figure is really just to provide a different view on where these discrepancies originate (in contrast to the tables which divide them based on rigidity and semantics/syntax). You are right in saying that each source tier is only compared to those at least as trusted as itself; I'll also explicitly include this in my explanation.

I think that putting raw counts in the legends would be more helpful for comparing these data than percentages to at least partially account for the "magnitude" issue you noted. These data may also be better represented by counts in a table with the bottom diagonal omitted (similar to what is used for logic puzzles; ChatGPT calls this a "triangular comparison matrix" or "reduced pairwise comparison matrix"). Would this merit an more in-depth conversation? I can wait on actually building one if so, but can also quickly throw together a mock-up to help visualize it!

[smiths]

@samm82 we can discuss the discrepancy source count in our meeting. We don't want to just show the counts, we want to interpret the results. If we say there are more internal discrepancies in standards then we want the figure to reinforce this. For the reasons we discussed, I don't think the current pie charts convey this correctly. Visually they look like the largest piece of the pie, but some pieces of the pie are excluded from the standards because of the "most trusted source" rule.

We should present figures where it is the reader's responsibility to figure out what conclusions can be drawn. We should make conclusions and use figures to support our conclusions. The first thing for you to decide is what conclusions you wish to draw, and then the figure comes. Of course, you need to find ways to visualize the data yourself to help formulate the conclusions, but in the writing you should have a plan for the conclusions you wish to draw from the data.

[samm82]

I found the original intent of these counts/pie charts in #83: to make it more clear "how many inconsistencies are within a document and how many are between different documents". Considering our discussion in #133, I think a table would be a better way to indicate this. @smiths was the first to suggest this (I think in written comments in a review of our paper), but my interpretation of the intent of this is to show that:

1. "standardized" documents aren't actually standardized
2. even when they are correct, other documents differ from/disagree with these standards
3. documents also disagree with less standardized (but still useful) sources (i.e., terminology collections and textbooks)
4. issues occur even within the same document and between documents with the same author(s)!

I'm thinking more and more that a table would be a better way to present the data supporting these, as with my other discrepancy tables. How much explanation of conclusions would be necessary in this case, where the data is presented more objectively? Would explicitly stating the above points be sufficient?