QR/Barcode scanning feature

Every part that is inspected has a barcode that can be used for identification. That is why selecting a part by scanning a QR/Barcode is one of the features (XRI-154) of the second product release version, that was researched and evaluated by the team.

Selecting a part by looking at a barcode also makes the application more user friendly than selecting it from a scroll-menu or saying long ID-numbers to select it with a voice command (see Sec. 6.1.1).

The envisioned functionality is described by the acceptance criteria in Fig. 40.

Figure 40: Acceptance criteria for the QR/Barcode scanning feature

QR/Barcode recognition is not something that is supported by HoloLens natively, at least not by the 1st generation. Microsoft offers a Software Development Kit for QR code detection and tracking on Windows Mixed Reality devices but it is supported only by the Hololens 2, as mentioned in Sec. 3.3.3Image Recognition and Tracking Software.

There is a QR/barcode library that is made specifically for decoding and generation of barcodes within images called ZXing. It was originally implemented in Java and later ported to support different platforms, including UWP and should, in theory, be possible to implement with HoloLens. [80] The main challenge with this is that documentation is near nonexistent and the library is rather old. There are some references of successful implementation of QR scanning with HoloLens using ZXing library with Unity, most of them are online forums, and are considered to be unreliable sources. However, there is one blog that is referenced to in almost all forums regarding the subject and is worth mentioning.

Mike Taulty works in Microsoft’s Commercial Software Engineering group that builds Mixed Reality/HoloLens applications on UWP, and has been working with development languages, tools, frameworks and platforms for Microsoft for twenty years and in the industry since 1991. [81] In Desember 2016 Mr. Taulty wrote a blogpostWindows 10, UWP, QR Code Scanning with ZXing and HoloLens [82] that received a lot of atten-tion on forums like Windows Mixed Reality Developer Forum [83] and Stack Overflow [84]. It seems that many developers were able to use ZXIng with HoloLens by following Taulty’s guidance at the time it was written and until around year 2017.

The challenge with Taulty’s implementation today is that is outdated and there are dependencies that prevent porting the solution directly to a newer version of Unity.

Taulty’s project was build in Unity 5.5, a version of Unity released in 2016 [85], and uses HoloToolKit, an older version of MRTK (see Sec. 3.3.1) from 2016. A lot of things have changed since then and in addition to Unity and tool kit dependencies there are building tools dependencies in Visual Studio as well.

Even though Taulty’s solution seems as a good starting point, implementing QR scan-ning with HoloLens using ZXing today requires more expertise and human resources than our team could provide in the short amount of time given. Considering that the XRI project will need to be ported to HoloLens 2 which supports QR code detection with a SDK provided by Microsoft [21] it does not seem as a good use of time and resources either.

With all that being said, QR/Barcode scanning is a process of image detection/recog-nition. Since this project requires image detection and tracking to achieve holographic guidance through measuring points on a part/panel in real world, it should be possible to use image processing functionality of the same software (see Sec. 3.3) to recognize QR/barcodes if it does not contradicts it’s licensing.

6.2.1.1 ARToolKit markers

ARToolKit was used to demonstrate the concept of barcode recognition with HoloLens.

There are two marker images designed to work with ARToolKit for HoloLens by de-fault: Hiro and Kanji. The markers can be found in the GitHub repository of the HoloLensARToolKit [24].

The toolkit can recognize two types of markers: pattern markerslike Hiro and Kanji and barcode/matrix code markers, described by an ID.

The pattern markers can be customized and there are some trivial guidelines/rules for developing new ones, found in the ARToolkit documentation or as result of recognition performance (see Sec. 3.3.3.1):

• The marker should be asymmetric for correct marker orientation.

• The marker should have a border for correct detection and not to blend in with the surroundings.

• The marker should not be too complex (a bit ambigious, as no definition on what is considered complex has been given).

• The marker can be colored since ARToolKit translates it to grayscale internally but a high contrast marker (black and white) is preferable since it is considered less complex.

After designing a marker it can be used with ARToolKit by translating it to a binary file (for example by using a program called mk patt [86]) and then indicating the file-name of the new pattern marker description in theARUWPMarker.csscript of the toolkit.

Barcode/matrix code markers recognized by ARToolKit can be generated using several online tools. Marker Generator by Eden Networks Ltd [87] is one of the tools that can be used and provides many options for generation matrix codes with up to 4194303 pos-sibilities (with 5x5 barcode dimension). 3x3 barcode dimension (without error checking and correction) gives 63 possibilities and is the best option since is trivially less complex.

Due to delivery deadline of the project this feature was tested as a stand alone concept and not integrated into the main application.