Before presenting the main results and answering the research questions, I will present the demographics of the two questionnaires. It must be noted that with only 10 and 5 respondents, respectively, these results are not generalizable and are included to contextualize the main results.
In Figures 4.1 and 4.2 we can see how respondents in each of the questionnaires rated themselves in terms of experience level with Microsoft Excel. Both source questions are a 5-point Likert scale and all available options are shown. Overall, we see that respondents of Questionnaire B reported a slightly higher level of experience with Microsoft Excel.
Figure 4.1:Questionnaire A: Level of experience with Microsoft Excel
Note that in all of the tables pertaining to demographics, I have adjusted the y-axis to one higher than the most popular option.
Figure 4.2:Questionnaire B: Level of experience with Microsoft Excel
Figures 4.3 and 4.4 show how respondents in the two questionnaires rated Mi-crosoft Excel’s suitability as a coding tool for the IG. Both source questions are a Likert scale of 7 points, and all available options are shown. Comparing the two questionnaires, the results are quite similar. Both groups of respondents reported a fairly strong positivity toward the tool.
Figure 4.3:Questionnaire A: Microsoft Excel’s suitability as a coding tool Finally, Figures 4.5 and 4.6 show the Questionnaire B respondents’ level of ex-perience with and perceived suitability of INCEpTION. The source questions and options were identical to those on Microsoft Excel. Overall level of experience with INCEpTION in this group is slightly lower than that of Microsoft Excel but again the results are very similar. INCEpTION’s reported suitability is also slightly lower than that of Microsoft Excel in the same questionnaire.
Figure 4.4:Questionnaire B: Microsoft Excel’s suitability as a coding tool
Figure 4.5:Questionnaire B: Level of experience with INCEpTION
Figure 4.6:Questionnaire B: INCEpTION’s suitability as a coding tool
Moving on, the ranked findings in each of the six categories, without distinguish-ing between categories 5 and 6, are shown in Tables 4.2 to 4.6. Accorddistinguish-ing to the weighting method described in Section 4.2, they are weighted roughly by the
num-ber of people who reported a sentiment or idea in each cluster, giving emphasis to the questionnaires. In the source lists, "Quest" is short for questionnaire.
Since the categories outlined in Section 4.2 map to research questions, we can now answer the research questions.
RQ1: What are the strengths and weaknesses of current coding tools? Since I in Section 1.1 deemed strengths and weaknesses to directly translate to RQ1’s respective sub-questions, this question is answered by those sub-questions.
RQ1a: What features from existing tools should the new coding tool retain?
This research question is answered by Tables 4.2 and 4.3. In the two question-naires, I asked participants "In your opinion what makes Excel / INCEpTION a useful coding tool?" I believe this formulation helps participants think about the advantages of the tools, which is what I am after. This question came after asking participants to rate their level of experience with the tool and to rate it as a policy coding tool.
As we can see, there is a higher number of clusters related to Microsoft Excel than to INCEpTION. I remind the reader that the data pertaining to Microsoft Excel is significantly richer than that on INCEpTION, and that this applies to the next research question as well (RQ1b).
Cluster Count Sources
Represents data in an organized fashion 8 QuestA (6), Notes (1), In-terview (1)
Easy to navigate, simple 4 QuestA (4)
The parsed information is easy to see, in-terpret, compare, and manipulate
4 QuestB (2), Interview (1), Notes (1)
Coded data can almost immediately be used for analysis
3 Interview (1), Notes (1), QuestB (1)
"Everyone" knows how to use a spread-sheet, easy to share
2 Interview (1), QuestA (1) Quick and easy to populate cells that
contain the same information by copy-pasting
2 QuestA (1), QuestB (1)
Ability to freeze cells is useful 1 QuestA (1) The tabular format enforces thinking
about each category, which is good for less experienced coders
1 QuestB (1)
Very good educational tool for the IG, easy to understand for beginners
1 QuestB (1)
Table 4.2:Identified advantages of Microsoft Excel as a coding tool
Cluster Count Sources Allows cross-statement annotation 2 QuestB (2)
Allows duplicate annotations 1 QuestB (1)
Allows sharing and collaboration 1 QuestB (1) Coding directly on the document is
intu-itive
1 QuestB (1) Easy to create new layers/tag sets 1 QuestB (1)
Free and open source 1 QuestB (1)
Layers/tag sets are specifically designed for coding in the IG
1 QuestB (1)
Table 4.3:Identified advantages of INCEpTION as a coding tool
RQ1b: What features from existing tools should the new coding tool discard?
This research question is answered by Tables 4.4 and 4.5. In the two question-naires, participants were asked "What challenges have you encountered coding in Excel / INCEpTION?" I chose to use the word "challenges" because it helps the participants think about their own experience with the tools, as opposed to asking something akin to "What are the weaknesses of the tools?". This question immediately followed the question asking about advantages with the tool.
RQ2: What features do coders need in a coding tool for the IG? This overar-ching research question is answered by Table 4.6. In both of the questionnaires, I asked participants "If you could design a new coding tool for IG 2.0 (i.e., new software), what three capabilities should that tool possess?" My intent with asking for three features was to avoid "intimidating" the respondents, because without the "three" qualifier, this question might as well have asked for a complete re-quirements document for the application. Several respondents thus listed multiple ideas, which I extracted and organized as explained in Section 4.2. One response was hard to split up: "Accountability, accuracy, efficiency". This was left as is be-cause it describes non-functional requirements, which are also useful to me in requirements engineering for IG Coder.
In both of the questionnaires, the above question immediately followed the one asking about disadvantages with the tool (INCEpTION in the case of Questionnaire B). This may have had an effect on the responses, as respondents by this point had thought about advantages and disadvantages with the tool and were now asked to come up with features a coding tool should have. This should be taken into consideration when inspecting the identified needs.
Note that the interview (data source 3) yielded no data points pertaining to needs.
This is because the interview focused on the coding experience in Microsoft Excel, and coding tools in general were not discussed.
Cluster Count Sources Time-consuming and tedious to code by
copy-pasting
5 QuestB (3), Notes (1), QuestA (1)
Confusion and loss of overview in larger tables
4 QuestA (2), Interview (1), Notes (1)
Difficult and cumbersome to code and keep track of nested statements
4 Interview (1), Notes (1), QuestA (1), QuestB (1) Annoying to move back and forth
be-tween far apart columns
2 QuestA (2) No ability to capture inter-statement or
inter-document linkages
2 QuestB (2) No coding completeness or quality
con-trols
2 Notes (1), QuestB (1)
Hard to rephrase a statement 1 Notes (1)
Hierarchies between components and properties cannot be represented
1 QuestB (1) Limited cell space, words are often cut
off
1 QuestA (1) Metadata has to be manually populated 1 QuestA (1) No standard way of indicating "not yet
coded"
1 Interview (1) Not a good tool for big projects with
sev-eral documents and sevsev-eral team mem-bers
1 QuestB (1)
Not computationally tractable 1 Notes (1) Rows full of "n/a" are hard to read 1 Interview (1)
Table 4.4:Identified disadvantages of Microsoft Excel as a coding tool
RQ2a: What are the essential features a coding tool for the IG should possess?
According to the threshold defined in Section 4.2, needs with a count of three or higher are deemed essential features. As we can see in Table 4.6, three needs were identified that belong to this category:
• Ability to export coded text in different formats (e.g. Excel CSV)
• An overview area underneath each statement with organized fields for IG components to visually double-check one’s coding
• Color coding for components distinguishing between regulative and consti-tutive
All of these had a count of three and are sorted alphabetically. They will have priority in defining the requirements for IG Coder.
RQ2b: What are the fringe features a coding tool for the IG should possess?
According to the threshold defined in Section 4.2, needs with a count of two or
Cluster Count Sources Editing an annotation requires deleting
and recreating it
2 QuestB (2) Graphic representation of coded data is
unreadable
2 QuestB (2) Missing output/export of coded text into
a usable format
2 QuestB (2) Missing validation of correctness and
completeness
2 QuestB (2) Changing layers/tag sets during coding
is impractical
1 QuestB (1) Coding in the IG requires the use of
sev-eral layers/tag sets
1 QuestB (1)
Table 4.5:Identified disadvantages of INCEpTION as a coding tool
lower are deemed fringe features. Refer to Table 4.6; 27 needs were identified that belong to this category. Eight of these had a count of two, meaning they were reported by two people. All of the needs will be considered during requirements engineering for IG Coder but the ranking will help me prioritize what features to invest in.
Table4.6:IdentifiedneedsinacodingtoolfortheIG(1/2) ClusterCountSources Abilitytoexportcodedtextindifferentformats(e.g.ExcelCSV)3QuestB(3) AnoverviewareaunderneatheachstatementwithorganizedfieldsforIGcomponentsto visuallydouble-checkone’scoding3QuestB(2),QuestA(1) Colorcodingforcomponentsdistinguishingbetweenregulativeandconstitutive3QuestA(2),Notes(1) Abilitytodirectlyconnectrelatedinstitutionalstatementswithinthesamedocument,e.g. usingsectionmarkers2Notes(1),QuestB(1) Abilitytodragcomponentsfromoriginalstatementintocategoryfield(e.g.,selecttheactor anddragitintoAttributefield)2QuestA(1),QuestB(1) Abilitytoflagexternalpolicydocumentsreferencedinaninstitutionalstatement,e.g.using sectionmarkers2Notes(1),QuestB(1) Abilitytogenerateanoutputfileforaccessibleviewingofcodeddata2QuestB(2) Accommodateimplicit(tacit)components2Notes(1),QuestB(1) Clearrepresentationofcodedstatementsforreview2QuestB(2) Fastnon-redundanthandlingofnested/multiplestatementsandlogicalrelationships2QuestA(2) Morespaceforeachcomponent2QuestA(2) Abilitytochangenestinglevelswithoutlosingannotationcontent(e.g.,ifapropertyisshifted tolevel3,itshouldnotloseitscontent)1QuestB(1) Abilitytoreviewearliercodedstatements1QuestB(1) AbilitytotagIGcomponentsdirectlywithinthedocument(withouthavingtocopyandpaste)1QuestB(1) Abilitytouploadtextdocumentsforcoding,andabilitytomanipulateuploadedtext1QuestB(1) Accountability,accuracy,efficiency1QuestA(1)
Table4.6:IdentifiedneedsinacodingtoolfortheIG(2/2) ClusterCountSources Context-sensitive"codecompletion"(e.g.,Attributesfieldshoulddrawonpreviouslyentered AttributesandpossiblyevenObjects)1QuestB(1) Detectionofstopwordsforremovalinannotationprocess(e.g.,"Thefarmer"shouldbecoded as"Farmer")1QuestB(1) Discouragemanualtypingandofferalternatives1Notes(1) Facilitiestosupportinter-coderreliabilityassessments(oratleastpreparationofdatato facilitatethis)1QuestB(1) IGisalreadyquitecomplicated,soacodingtoolmustnotaddtotheconfusion1Notes(1) Notforcecoderstomakeanupfrontdecisionaboutwhetherastatementisregulativeor constitutive1Notes(1) Prepopulatedcategories/tagsforcapturingIGsyntacticcomponents1QuestB(1) Quizzesforself-study1QuestA(1) Remindertoexportdatabeforeclosingbrowserwindow1QuestB(1) Separateregulativeandconstitutivestatements1QuestA(1) SupportallthreelevelsofIGcoding1QuestB(1) Validationofcodedstatements1QuestB(1) Visualizehierarchyofstatementandcomponents1QuestA(1) Visuallylinksimilarcategories1QuestA(1)