Shortcomings of ISO 25010

Dr. Gernot Starke

Before we dive into the details, please allow me some nitpicking.

Adjectives versus Nouns

The ISO 25010 standard[1] (and most of the historical quality models too) is writing about the “properties of a system or product”. Such properties are crucial in everyday life, therefore our (natural) language provides adjectives as a specific category of words: In real life, people call things delicious, fast, usable or secure. Product owners want or require systems to be responsive, reliable, testable, or easy to change.

Too bad, computer scientists plus the ISO 25010 standard committee, as well as the inventors of traditional quality models, preferred to use nouns to describe properties: They use “flexibility” instead of flexible, “time-behaviour” instead of fast and many more “…ilities” instead of the beautiful, pragmatic, and understandable adjectives our languages provide.

You might consider this nitpicking, but it’s definitely a linguistic downside, which makes the practical day-to-day work with quality characteristics and quality-requirements more difficult than it needs to be.

ISO 25010 - An Overview

Due to ISO 25010, software product quality is composed of eight top-level characteristics, which in turn consist of several sub-characteristics. While such a hierarchy is structurally simple, it gives rise to serious disadvantages - but more on that in a moment. Altogether, the standard attempts to establish a widely usable (“generic”) taxonomy of almost 40 terms in total. Presumably, all stakeholders should be able to recognize their goals and requirements represented there. In reality, this is not the case, as you will see right away.

During development of a specific system or product, we have to refine that generic model for our system anyhow: We need to describe or demand those characteristics (aka qualities) that are relevant for our specific system.

The diagram shows the top-level characteristics of ISO 25010 from the 2011 version, which is still the official version.

Holes in the conceptual carpet

Developers of embedded or real-time systems in medicine, aerospace, the automotive industry, chemistry, mechanical engineering, and others will immediately notice an omission: “Safety” (safety of life and health, physical integrity, etc.) is missing from this overview. And no, it does not appear as a subfeature in the ISO standard either.

Now for the teams that develop modern web or mobile applications: They will be surprised that operational or DevOps topics like deployability are missing from the standard. Oh yes, scalability, which is usually a challenge in online applications, is missing too.

Well – if there are any product owners among you, they will likely miss “Time-to-Market” (TTM) because you surely want to bring your product to the market or to your users in a timely manner. If you think that something like TTM is not a quality attribute at all: It is, however, an attribute that some stakeholders want from a system, and thus it counts as a quality attribute for them.

My conclusion so far: A few topics relevant from a practical standpoint are missing in the current version of the ISO 25010 standard. This is certainly not dramatic because we can add them to our system-specific model.

However, in a standard document as detailed as ISO 25010 presents itself in many parts, there certainly would have been room for those missing pieces.

Confusion under the Hood

Let’s dive into the details, and head over to some of the about 40 so-called sub-characteristics of ISO-25010.

Their sheer number might seem overwhelming at first, but let’s ignore this for now[2].

Functionality, no lo comprende

Sidenote[3] Let’s start on the left of figure 2: The functional suitability subtree causes some confusion, at least in my head: At first, this term is quite uncommon, and definitely exceeds my everyday vocabulary.

For me, “suitability” refers to the entire system, not just its function. A system needs suitable performance, availability and so on, not just suitable functionality. It goes further: One of the subfeatures is functional adequacy. I, at least, have failed to understand the difference between “suitability” and “adequacy”.

Adapt, Maintain, Change?

Let’s look further, to maintainability and transferability: conceptually, I see no real difference between “changeability” (subcharacteristic of maintainability) and “adaptability” (subcharacteristic of transferability). I want to adapt systems to changed requirements, not only adapt them for transfer to another operating system. This distinction might lead to confusion in practice.

Limits of the Strict Hierarchy

From my perspective, a tree structure is not suitable to structure a multi-faceted term like “quality”, as a strict hierarchy has its limits. Consider some examples:

• “Testability”: ISO positions testability under maintainability, which is understandable: When maintaining or changing systems, testability is an important property, to reduce risks. But: From my experience, people also consider testability an indispensable part of reliability. Therefore, testability should occur in both subtrees, not only under maintainability.
• “Availability” is located below reliability. However, if a system is not available, it is not usable. Therefore, availability also belongs to usability.
• “Adaptability” belongs to both portability and maintainability, but ISO decided to put it below portability only.

A strictly hierarchical placement of system qualities will always be disputable. Another approach would be to allow detailed qualities to be “taggable”, so a specific quality might be tagged (or labelled) with more than just one top-level quality property. The recently published arc42 quality model allows exactly that (see [2]).

My conclusion so far: A lot of mass, but important nutrients (e.g. conceptual consistency and clarity) are missing. These and other problems with ISO 25010 have initially been pointed out by Len Bass et al.[4]

Does the 2022-Draft Improve the Situation?

When ISO announced an update to the 2011-version in a mailing list in mid-November 2022, my expectations were high. After all, ISO has had 11 years to come up with an improved proposal. At first glance, it is noticeable that the top-level features have grown from 8 to 9, see the diagram below.

Safety has made it into this draft. Definitely a massive step forward. Still, I see overlapping terms in the areas of “maintainability” and “flexibility”. This remains dubious even when considering the ISO definitions:

• Flexibility: ability to meet different or extended requirements, the ease with which changes can be adapted to requirements or the system environment.
• Maintainability: ability to be changed effectively and efficiently.

From my perspective, these two sentences are too similar, and leave way too much room for interpretation. That will lead to useless discussions in practice.

Below the usability category, aesthetics and accessibility are replaced by engagement and assistance. Again, we could argue about this — but in my perception, the term “accessibility” has become established in the UI/UX field, which is why this replacement rather bothers me.

In the new version, I consider it positive that:

• scalability has been added, as a subcharacteristic of flexibility, and
• maturity has now become_faultlessness_.

Isn’t Software made up from Code?

Although internal structure, legibility and conciseness have been part of previous quality models (Boehm[5] mentioned these qualities in his model from 1978!), the ISO refused (or forgot?) to include such properties in their model.

In my opinion, development teams are important stakeholders, and structure and readability of code will often be essential goals for them. Ignoring code quality and component structure as quality requirements is simply not acceptable from my experience, and I consider their absence a serious deficit in the ISO 25010 standard.

Practical Use?

Although one can use the ISO 25010 standard as a checklist for specific quality requirements, a smaller but well-chosen selection of terms would be sufficient for this. For example, the arc42 quality model (Q42) consists of just eight key properties, instead of the 35 terms from ISO 25010.

Maybe the most serious disadvantage of ISO 25010 is the complete lack of specific examples of how to apply the model to practical, real-world systems. That leaves development teams and other system stakeholders out in the cold. Len Bass et al.[4] have proven in their book that such examples can be systematic and useful. arc42 followed suit, providing numerous tagged and cross-referenced examples of quality requirements.

Another potential source of unwanted complexity is the ISO metamodel[6], which distinguishes Quality-in-Use, Data Quality, Service Quality and Product Quality. In my opinion, this makes the ISO 25010 model overly difficult to digest for real-world projects.

My final complaint is the paywall: The price of almost 140 Swiss Francs (approx. €140 and US-$140) might prevent people from reading the complete document.

Fast Food or Gourmet Dinner?

ISO 25010 is neither. Even the 2022 draft update of ISO 25010 doesn’t go beyond canteen food: Versatile, but difficult to access due to the multitude of overlapping and sometimes confusing terms Formality-lovers will adore the metamodel, but practitioners will lack practical guidance. The often abstract definitions of terms build up superfluous hurdles. Specific, usable examples of practical applications are missing (again, arc42 to the rescue).

In my opinion, less would have been more: Instead of nearly 40 terms at various levels, a few top-level features would be sufficient, coupled with suggestions or examples for each of thems.

In this respect, Len Bass and Co. have done an outstanding job with their fundamental work [4]. They had already invented the concept of “quality scenarios” many years ago. In their fourth and latest edition of their book, they followed up with a multitude of practical examples. You can find a brief overview of their work in another article.

Acknowledgements

Thanx to “m”, Anja Kammer, Daniel Lauxtermann, Joachim Praetorius and Ben Wolf for bug fixing and dramatically improving the quality of this post. Thanx to Eberhard Wolff for encouragements.

References