Data Collection and Methods

Figure 9.1: Block scheme of 3d CC modeling using Microsoft Kinect. IIR(i, x, y) and IDM(i, x, y) are provided by the Kinect.

9.2 Data Collection and Methods

The CC modeling is performed using Microsoft Kinect for Xbox One⁸ and reflective markers to track the shoulders, elbows and wrist points in 3D.

The Kinect device uses IR time-of-flight technology to create depth maps where the map pixel values corresponds to the distance in millimeters to

8https://www.xbox.com/en-US/xbox-one/accessories/kinect

objects visible to the IR sensor. To capture the IR frames and the depth maps from the Kinect, we have used a modified version of theKin2 Matlab toolbox created by Terven⁹. The modifications includes frame rate control and frame capturing.

The setup for the experiment with an additional block scheme for the main tracking algorithm is shown in Fig. 9.1. From the captured IR frames and depth maps we track the reflective markers and measure the bystanders CC movement in world coord., X_w,Y_w andZ_w, for different CC depths.

The truth data,CC_true, is collected by performing the CCs on a Resusci Anne manikin.

9.2.1 Distance, Z, from Kinect camera in world coord.

As can be seen in the depth map, I_DM(i, x, y), shown in Fig. 9.1, the markers appears as black spots due to the fact that the highlights either prevent the infrared reflection back to the kinect sensor or causes the sensor to saturate [118]. To obtain the depth map information in the area around each marker the following steps, shown in Fig. 9.1, are carried out: In step 1, only the information in bright reflective spots are kept in the IR frames, I_IR(i, x, y), where iis the frame number and x, y the image coordinates, by thresholding the frames:

I_IRT(i, x, y) =

⎧⎨

⎩

1 if I_IR(i, x, y)> T_m

0 otherwise (9.1)

Step 2 removes small bright spots caused by noise by discarding all ar-eas with a number of pixels< T_sbs. Containing only information in the reflective markers, the resulting filtered framesI_IRF(i, x, y) are dilated in step 3 with a 5-by-5 matrix of ones and all pixel values>0 in the dilated imageI_IRD(i, x, y), are set to one. Next, the Hadamard product between I_IRD(i, x, y) and the depth maps,I_DM(i, x, y), is found, resulting in frames, I_DMF(i, x, y), with only depth information in the area around each marker.

Further we define index-sets, A_m = {x^(m)_l , y_l^(m)}, where x^(m)_l and y_l^(m) represents the pixel positions included in the region of each marker, m, wherem∈1 : 6. TheZ_w position of each marker and frame can then be found by:

9https://github.com/jrterven/Kin2

Figure 9.2: Median movement,MQ(p, d, DG,Q2), [mm] as a function of CC depth [mm] for each test person‘s (TP) CCs.

Z_w(i, m) = 1 n_m

l∈Am

I_DMF(i, x^(m)_l , y_l^(m))>0 (9.2) where n_m is the number of pixels in the index-set of marker, m.

9.2.2 X and Y position in world coord.

In step 4, Fig. 9.1, the markers centroid coordinates, (x_c, y_c)ⁱ_m, are found inI_IRF(i, x, y) by:

(x_c, y_c)ⁱ_m =cent(I_RF(i, A_m)>0), (9.3) and in step 5 we convert these image coordinates to world coordinates. By calibrating the IR camera, the camera matrix, K_IR, can be found, and together with a rotation matrix,R_k2w, a translation vector, T_k2w and the depth information, Z_w, from section 9.2.1, K_IR allows us to convert IR image coordinates (x, y) to world coordinates X_w and Y_w. By defining the matrixC_k2w =K_IR[R_k2w|T_k2w], and choosing the center of the world coordinate system to be the same as the camera coordinates system, the conversion can be written [119]:

λ

where λ=Z_w,α andβ the focal length of the camera andx₀ andy₀ the principal point offset in pixels. From Eq. 9.4 we can find theX_w andY_w coordinates for eachm and i:

X_w(i, m) = (xⁱ_c,m−x_o)Z_w(i, m)

α (9.5)

Y_w(i, m) = (yⁱ_c,m−y_o)Z_w(i, m)

β (9.6)

Further, for each marker, m ∈ 1 : 6 and direction, d ∈ X_w, Y_w, Z_w, we define position in world signals,S_m,d(i), as a function of discrete time,i, e.g. S_1,Xw(i) =X_w(i,1).

9.2.3 Movement analysis

FromS_m,d(i) and the reference data,CC_true, we can measure a person‘s movement as a function of different CC depths, summarized in Algorithm 2.

The output is the measured bystander movement, M_Q(p, d, DG, Q) [mm], where p∈shoulder, elbows, wrists in the directionsd∈X_w, Y_w and Z_w, DG the depth groups and Q the quartile measurements, Q1 (25%), Q2 (median) and Q3 (75%). CCs in the CC rate range of 95-125 cpm is here being measured and sorted ingroupCCD according to the reference CC depths. The first group 0-15 mm and the following 9 groups divides the range 15 to 60 into depth intervals of 5 mm. Further we find the motion vector for the median movement inY_w andZ_w direction for the bystander‘s shoulders. These vectors are used to estimate how the motion would be observed by a smartphone camera placed on the floor next to the patient, see Fig. 9.4, and to investigate if it is possible to create a conversion model based on the method for CC depth measurment proposed in Meinich-Bache [93], where we measure the movement‘s motion band size in the image frames. To convert the movement in world coord. to smartphone camera image coord., we use Eq. 9.4 and substitute K_IR, R_k2w and T_k2w with smartphone to world matrices K_SP, R_sp2w and T_sp2w. Fig. 9.4 shows the rotation and translation, -555 mm in Y_SP-direction and 475 mm in Z_SP-direction, between the coord. systems, and we get:

[R_sp2w|T_sp2w] =

Algorithm 2 Movement measurement of shoulders, elbows, and wrists for different CC depths.

Input: S_m,d(i), CC_true, Output: M_Q(p, d, DG, Q) Detecting CCs using the Y_w signal of left wrist:

[pks lcs]←f indpeaks(S_5,Yw(i))

Measuring mov. for each CC, marker and direction:

form=1:6 do

ford=X_w, Y_w, Z_wdo

o=1;forj=length(lcs):-1:2 do

if 95[cpm]< CC_true(j)<125[cpm]then M_m,d(o)←

|max(S_m,d(lcs(j)) : S_m,d(lcs(j − 1)) − min(S_m,d(lcs(j)) : S_m,d(lcs(j−1))|o=o+ 1;

end end

Grouping CCs in CC depth groups (DG):

M(m, d, DG)←groupCC(M_m,d(o), CC_true) end

end

forshoulders, elbows and wrists in all directions: do Combining L&R measurements

M_L&R(p, d, DG)←[M(Lef t, d, DG), M(Right, d, DG)]

Estimating Q1,Q2,Q3

M_Q(p, d, DG, Q)←Q(M_L&R(p, d, DG)) end