How to improve/shrink example of recording test approach information?

[samm82]

In a previous review, you suggested that I should include an example of how I record information on test approaches to give the reader a more tangible idea of what I mean. @JacquesCarette mentioned that "as this is the methodology section, we really care much more about process and illustration of process[] than the data obtained." If we end up following through with #197, this information will be more "isolated" which may make it less overwhelming, but I'm still wondering if I went a little bit overboard on this specific example 😅:

For example, ISO/IEC and IEEE (2022, p. 1, 36) define “A/B testing” (see Figure 3.2), giving “split-run testing” as a synonym and describing it as “statistical testing approach”. We therefore create a new row in our test approach glossary for “A/B Testing” with the Category of “Approach”, a Synonym of “Split-Run Testing”, and a Parent of “Statistical Testing” (based on Section 2.1.3), all with their relevant citations. We then abstract away this already-captured information from its Definition before recording it as “testing ‘that allows testers to determine which of two systems or components performs better’ ” with the same citation information (ISO/IEC and IEEE, 2022, pp. 1, 36). The latter page provides more information that we capture as Notes in our glossary: it “can be time-consuming, although tools can be used to support it”, “is a means of solving the test oracle problem by using the existing system as a partial oracle”, and is “not a test case generation technique as test inputs are not generated” (p. 36). Interestingly, this explicitly rules out “Technique” as a possible Category for this approach, which is consistent with its categorization as a test practice (Fig. 2); we replace our previous Category of “Approach” with the more specific “Practice” (see Section 2.1.1). As we investigate other sources, we learn more about this approach. Firesmith (2015, p. 58) gives it as a child of usability testing, so we include this Parent alongside “Statistical Testing” without issue. However, since usability testing is a test type (ISO/IEC and IEEE, 2022, pp. 22, 26-27; 2021, pp. 7, 40, Tab. A.1; implied by its quality; Firesmith, 2015, p. 53), A/B testing may inherit this categorization. We document this by adding “Type” as a Category alongside “Practice” with the source “inferred from usability testing”. Since this violates our assumption that categories are orthogonal (see Section 2.1.1), we consider this to be an inferred flaw, defined in Section 2.3 and documented later in Section 5.3.1 and Table 6.6.

Figure 3.2: ISO/IEC and IEEE (2022, p. 1)’s glossary entry for “A/B testing”.

---

I also realized that in the introduction, I include an excerpt of our test approach glossary for clarity; do you think pointing back to this would be sufficient, or should I also walk through the methodology as I do above (potentially more concisely 😅)?

[smiths]

Yes, the specific example seems like too much detail, and too much data, for the methodology section. The big picture view of the methodology will get lost in the details.

The information on how one should interpret your methodology is useful; it just shouldn't be presented at the same time as the methodology or it will be confusing. You could have a separate section, probably in the chapter that presents your data, illustrating how the methodology was realized. Your paragraph about A/B testing is still really dense. You might want to present it as an enumerated list of the steps that were followed.

I don't understand your second question (which really should be in a separate issue) 😄

"I also realized that in the introduction, I include an excerpt of our test approach glossary for clarity; do you think pointing back to this would be sufficient, or should I also walk through the methodology as I do above"

Where are you pointing back to the introduction? Where would you be walking through the methodology?

[samm82]

I currently do not point back to the Table 1.1, which is in the Introduction. My question was when I'm giving an example of how to perform our methodology, should I include a pointer back to this table as the "final" product? If so, should I also give a (briefer) description of the steps I followed? My thinking is that seeing the final product will make the methodology more clear, especially if we include the information we used from the source, but I also don't want to fall into the trap (again) of making the reader deduce what we're trying to communicate, so we should still probably say something. 😅

Your suggestion of enumerating the steps makes a lot of sense and should hopefully make this more digestible. 😂

[smiths]

Ah, I get it now. Yes, point back to Table 1.1 as an example of the output of this process. You should probably also point to the full output of the process. I guess that will be online, or in an Appendix. You don't need to keep pointing to it, but I agree that a pointer at the beginning will help the reader see what the methodology is leading to.

[samm82]

In part from my work on #197, I realize this is a symptom of the larger issue: I have a dense paragraph outlining our procedure for recording information, then a dense paragraph outlining how we apply it to this specific example. I still agree that enumeration would help, and we could do this for both instances. However, there are some decision points to be made (e.g., we could include discussion on "explicitness" here) that could make enumerating this as a "simple" list challenging.

Could a flowchart be a better representation of our procedure? If so, how could we use it to simplify the example of A/B testing without being repetitive? Just include subsets of the flowchart with our relevant information filled in? We could always tackle this piecewise instead of making all the decisions upfront. I could even start by enumerating our procedure (or at least trying to).

[JacquesCarette]

Flowcharts are often a good means to outline a procedure. Especially when the text in the boxes is sparse.

[samm82]

I made a first draft of a procedure flowchart (see below; the "X"s are an artifact of the website I used). I don't know if it's too "hand-wavey", or if it's abstracted at the correct level of detail. The main areas I foresee issues arising:

1. Details for "Given X Matches Present" may be ambiguous, including the name of these decisions, the qualifications for each decision outcome, and the process to perform based on these outcomes.
2. As a subset of this, "Document as Flaw" may be too vague, but I think explaining this in more details makes more sense to do in an explanatory paragraph as opposed to this diagram.
3. I should likely be more clear about what I mean by "Given" (found in the source being investigated) and "Present" (found in our glossary); should this also be done in paragraph form?
4. "Remaining Given Data" is intentionally left vague as to abstract away details, but is it clear that the relevant decisions would be made for each piece of additional data (e.g., synonyms and parents)?

Any other glaring issues with this before I start digging into it more, or am I on the right track?

[JacquesCarette]

One minor thing: please don't have 2 'End' nodes! Also, it is very weird to have two 'Category Given?' nodes. I don't understand why the flow for something not present in the Glossary should be different (after you create a new row) -- shouldn't it be the same?

[samm82]

I have two "Category Given?" nodes since the action I take depends on if the approach is new to our glossary or not; if it is and the category isn't given, I give it a default category of "Approach". I guess I could rearrange this a little bit as to create the new row with a default category of "Approach", then proceed through the "normal" steps. This could be a little clunky because then we might add a category of "Approach" just to immediately replace it, but it would also resolve having two "End" nodes as a side effect! 🤩