9.3 First Part: Task UI
9.3.2 Discussion
The first part of the evaluation was supposed to find an answer toRQ1.2. However, as this was a pilot evaluation, the main focus was to find errors in the artifact and the evaluation method.
9.3 First Part: Task UI
ID PID Feedback Fixed
FPF01 E,F,G Suggestion: Remove “run loop code” button. 3 FPF02 E,F,H Did multiple subtasks before running the code.
Suggestion: Either run code automatically or make
“run code” button more visible.
3
FPF03 E,F,H Suggestion: Place code editor in the middle. 3 FPF04 E,F Loop code button did not run the new loop code. 3 FPF05 E,G Using ”,” for decimal does not give an understandable
error.
3 FPF06 E,G Suggestion: Make it possible to change variables in the
variable listing.
3 FPF07 F,G Some section titles was too long resulting in a layout
bug. Suggestion: Require short titles.
3 FPF08 F,G Outline looks more important than it should. Also too
many elements in the text. Suggestion: Move the outline out of the main content.
3
FPF09 G,H Suggestion: Instructions and error messages about variables should first mention the name, then the value, not value, then name.
3
FPF10 G,H Task typos. 3
FPF11 E Hard to see when section instructions are changing.
Suggestion: Use different colors or highlight changes for a second.
3
FPF12 E Confusion auto-zoom. Suggestion: Make zoom more static.
FPF13 E Suggestion: Do not change code between subtasks.
FPF14 F Hard to see where to write the code. Suggestion:
Highlight where to input.
3 FPF15 F Suggestion: Make it possible to navigate between tasks. 3 FPF16 F Suggestion: Add hints.
FPF17 F Suggestion: Do not duplicate info in the task title and description.
FPF18 F Suggestion: Collapse outline sections. 3
FPF19 F Suggestion: Show hidden code. Use collapsible fields in the editor.
FPF20 H Hard to see where to look. Suggestion: More pointers and highlighters.
3 FPF21 H Suggestion: Have an initial tutorial for the task UI.
Table 9.1: Feedback from the first part of the pilot evaluation.
Errors in the artifact
Given the feedback listed in Table 9.1, a few of them pointed out errors in the artifact. Two participants found a logical error with the “run loop code” button (FPF04). However, as three participants already suggested that the “run loop
code” button should have been removed (FPF01), it got removed, also resolving FPF04. Two of the participant also tried to use ”,” instead of ”.” for decimals.
This resulted in a not understandable error for the user (FPF05). Due to the nature of Python, this type of error is very hard to detect. To resolve this error, the code editor was modified to convert all numbers with ”,” into numbers with
”.” automatically. As this feedback was never brought up again after this change, it would seem to work. However, this change may cause side effects as the learners may get used to writing ”,” and getting an automatic correction.
Method
The first interviews were more unstructured than the last ones. This was due to getting more experience with both the preparation and process routine. Thus, overall, the routines worked as they were supposed to.
Elements
To begin answeringRQ1.2,Table 9.2contains all the elements that were placed in the artifact on purpose as well as elements that were suggested or not implemented on purpose. Those who did not exist in the artifact at the time were marked with ”-” in the “Used by” column. Each element has either been placed in the
“Should be included” section or the “Optional” section, based on whether they are important to include in an OLP used to introduce programming in an introductory physics course. If they received a complaint, the complaint was noted in the
“Feedback” column. As it is important to know if the element was useful for the tasks, they were either noted with “ALL” in the “Used by” column, for being used by all, or with “SOME”, for being used by one or more participants, but not all. If it was noted with “NONE”, none of the participants used it.
For an element to be included in the “Should be included” section, the element must be used by all and be told to be beneficial by more than one participant.
However, if the element was not implemented in the artifact, it must have been suggested by more than one participant.
Changes
The main change that needed to be done was to make it the focus of the task clearer, including removing elements that proved to distract the participants from the task at hand. All the changes can be seen in Table 9.3.
9.3 First Part: Task UI
E04 Code editor ALL Hard to find the first
time E05 Graphical result output ALL
E06 Section title ALL Too long
E07 Section level instructions ALL Hard to see changes E08 Automatic tested subtasks ALL
E09 Preexisting code in editor ALL E10 Understandable error feedback ALL
Optional E11 Subtask level instructions ALL
E12 Outline SOME
E13 See progress SOME
E14 Separate code editor for initial conditions and loop
ALL E15 Textual feedback from teacher
trough tests
NONE Hard to spot E16 Underlines on syntax errors SOME
E17 Autocomplete in code editor SOME
E18 Textual output SOME
E19 Graphical solution output SOME
E20 Current variables NONE No use cases
E21 Graphs -
-E22 Get solution after X attempts ALL
E23 Navigate between tasks - Wanted
E24 See hidden code - Wanted
E25 Change variables in variable list - Wanted
E26 Highlight where to look - Wanted
E27 Initial tutorial for task UI - Wanted E28 Save progress, continue later - Wanted
Table 9.2: Feedback on specific elements from the pilot evaluation. The yellow ones were added from suggestion.
ID Change
FPF01 Removed “run loop code” button.
FPF02 “Run code” button only appears when a change is done as well as not listing the next subtask titles. See Figure 9.1a.
FPF03 Placed code editor in the middle.
FPF04 Removed “run loop code” button.
FPF05 Automatically correct ”,” in numbers to ”.” in the code editor.
FPF06 Make it possible to click a variable to change it.
Display a text that it is changeable on mouse hover.
FPF07 Shorten long section titles as well as make the titles wrap.
FPF08 Move outline to the whole left side of the screen. See Figure 9.2.
FPF09 Change all occurrences where value comes before the variables name. Also changed the task creator UI to generate feedback with this format too. See Figure 9.3.
FPF10 Changed tasks.
FPF11 Highlight section for a second when it changes. See Figure 9.4.
FPF12 Not fixed as it has benefits as well. Needs a better solution.
FPF13 Not fixed as teachers may want different dynamics of their tasks. Thus, it was a reasonable suggestion.
FPF14 Pulsating highlight around code editor when the user approaches a new task. See Figure 9.1b.
FPF15 Suggestion: Make it possible to navigate between tasks.
FPF16 Not fixed.
FPF17 Changed tasks.
FPF18 Collapse outline to only show subtasks belonging to the current section. See Figure 9.2.
FPF19 Not fixed.
FPF20 Fixed by FPF02, FPF08, FPF11, and FPF14.
FPF21 Not added. Thus, it was a reasonable suggestion to add.
Table 9.3: Changes from feedback inTable 9.1.
9.3 First Part: Task UI
(a) Fix for FPF02.
(b) Fix for FPF14.
Figure 9.1: Fixes for code editor.
Figure 9.2: Fixes for FPF08 and FPF18.
(a) Before change, in teacher feedback. (b) After change, in teacher feedback.
(c) After change, in instructions.
Figure 9.3: Fix for FPF09.
Figure 9.4: Fix for FPF11.