Image Annotation

With just one class the segmentation masks are binary; all pixels surrounded by a drawn polygon correspond to corrosion, whereas all remaining pixels are to be interpreted as a neutral class, i.e. background. For implementation reasons it is useful to also count the background class as a distinct class. It will in the following therefore be referred to two classes, but keep in mind we are just considering corrosion versus not corrosion.

For semantic segmentation, all corroded pixels can be segmented as one instance, i.e.

as the union of a set of polygons. For instance segmentation, however, we need to divide the damages into multiple instances. In typical segmentation problems this is straight

forward. A person, for instance, is very well defined; the annotated instance should contain the apparent parts of a head, a body, two arms and two legs. Similarly, a ball should have a circular shape and a door is an (often skewed) rectangle with well-defined edges.

Corrosion damages, on the other hand, is much more challenging to annotate.

Challenges Related to Annotating Corrosion Damages

First and foremost, it can be challenging to decide what is really corrosion and what is dirty brown/red-like surfaces. An example is shown in Figure 4.2a. Furthermore, corro-sion has a tendency to smudge onto surrounding areas, for instance covering concrete with corrosion. The smudged areas are not damaged themselves but are still visually corroded.

An example is shown in Figure 4.2b. Similarly, the boundaries between corroded and un-corroded areas are often smooth/gradually changing. An example is shown in Figure 4.2c.

With no sharp edges, it is very difficult to decide where the annotation should start and end.

Next, corrosion damages are not "objects" as in the typical interpretation of the word.

Consequently, there are few to none shape characteristics defining a corrosion damage.

The damages can take on all sorts of shapes and sizes, making it difficult to distinguish individual instances. In fact, a general definition of an instance of corrosion is practically impossible to make. Figure 4.2g shows an example in which it is obvious to consider three distinct instances, whereas in the other examples there are many ways to separate instances.

Finally, a corroded area is often not completely and evenly corroded. It may contain scattered corrosion, as shown in Figure 4.2h, and creating detailed masks only contain-ing true corrosion is tedious and impractical. Similarly, some images have areas heavily corroded, with lighter corrosion surrounding it. An example is shown in Figure 4.2i. This will either lead to instances with varying degree of intensity or instances surrounding other instances, depending on how annotation is performed.

Annotation Solutions Opted for

There is no one correct solution to the above issues. It is important, however, to be consis-tent and to choose segmentation masks useful for machine learning algorithms.

Dirty/brown/red surfaces were excluded as best as possible. Since missing a damage is worse than reporting a few excess damages, a fairly liberal approach was chosen in which cases of doubt were annotated as corrosion. For the same reason, mask boundaries were drawn to contain as much corrosion as possible. This was also chosen to ensure light corrosion was detected.

Areas with smudged corrosion were annotated as corrosion for two reasons: First, it was difficult to determine where the damage ends, and the smudge begins. Second, as smudged areas are visually similar to true corrosion, it would likely be confusing for machine learning algorithms to learn the difference.

Separating corroded areas into instances was perhaps the most challenging task. Due to the wide range of different shapes and sizes, it was difficult to be consistent. In fact, consistency is not well defined in this case. Corroded areas with lots of background in

(a) Red/Brown Surfaces. (b) Smudged Corrosion. (c) Smooth Boundaries.

(d) Corresponding Mask to (a). (e) Corresponding Mask to (b). (f) Corresponding Mask to (c).

(g) Well-Defined instances. (h) Scattered Corrosion and Un-defined Number of Instances.

(i) Heavy Corrosion Surrounded by Light Corrosion.

(j) Corresponding Mask to (g). (k) Corresponding Mask to (h). (l) Corresponding Mask to (i).

Figure 4.2: A selection of different types of images annotated and corresponding instance segmentation masks.

Colors are randomly assigned to instances for visibility, and have no further interpretation.

between were separated as distinct instances. Corroded areas were also considered sepa-rate instances if their sizes were similar and boundaries sharp. Continuous corroded areas were considered one instance, even if the resulting shape was very wide or tall (e.g. metal beams), unless it was evident from the image that the damage is spread over different objects (e.g. across two neighboring beams).

Regarding scattered corrosion, small uncorroded areas within larger-region segmenta-tion masks were accepted. This was necessary to maintain a reasonable annotasegmenta-tion time and was considered not harming training of machine learning models. Very small corro-sion dots not part of a cluster or scattered region were ignored for the same reason and to reduce training time for instance segmentation.

Finally, heavy corrosion was annotated as one instance, with surrounding light cor-rosion as a separate instance, but only when the difference/boundary was strong/obvious.

This decision was made as it was considered the best for machine learning algorithms (e.g.

edges are very important features).

Figure 4.2 shows examples of the above-mentioned challenges with corresponding segmentation masks indicating the solutions chosen. Note that for semantic segmentation, all instance masks were merged to one, eliminating some of the aforementioned problems.