Furthermore, novice females were superior to expert females, whereas expert males to novice males (Figure 12). However, none of these differences occurred among gender were significant F(1,55)= 0.032, p=0.859).
The effect of expertise and gender on colour illustration was also assessed by two-way ANOVA test and no significant interaction occurred, F(1,55)= 0.109, p=0.742.
Figure 12. Scores on colour of four user groups.
Figure 13 shows the performances of experts and novices based on shape, size, colour, and presence & location. We clearly see that the lowest overall performances for both groups occurred for presence & location. This proves that drawing a map element on a correct location is the hardest part of the memory task compared to describing its shape, size, and colour.
Figure 13. Summary of performances (dark colour: experts, light colour: novices)
Aggregation presence & accuracy (1), shape (2), size (3) & colour (4). Presence & accuracy, shape, size and colour of drawn elements show “how well” the sketch maps were drawn. Until here, we tried to evaluate the influence of each criterion individually. However, aggregation of all criteria used for scoring the drawn elements can offer a more objective measure to compare the quality of sketch maps. Inherently, the quality of sketch maps reflects the performance of participants. We treated each four parameters as they have equal importance on overall performance of a participant, so that we assigned each parameter the same weight. Overall performance scores were calculated as the average of individual performances for four different groups (expert females, expert males, novice females and novice males) in a 0-100 scoring scale.
Expertise. According to the aggregated analysis, the average score of experts was 71.8 (st. dev. 19.2) with a minimum of 39.9 and a maximum of 92.8, whereas it was 68.2 (st. dev. 19.1) with a minimum of 21.5 and a maximum of 92.2 for novices. The difference of 3.6% on expertise was not statistically significant, F(1,55)= 0.689, p=0.410). The findings correspond to what was found by Thorndyke & Stasz (1980) and also by Gilhooly et al. (1988) in their studies on experts and novices’ ability to learn and remember information presented via maps. As mentioned in literature review, learning and remembering a map feature more likely depends on the general map knowledge and map experience does not predict learning of planimetric information (Kulhavy & Stock, 1996). The original map shown to participants was a simplified 1:10k topographic map and did not contain any familiar places (or names) to eliminate or minimize the degree of familiarity. Thus, both experts and novices saw the map for the first time and we presume that the maplikeness of stimulus had a great influence on their map learning (study and recall) process. The more maplike a stimulus, the more likely it is that the participants consult their general map knowledge as they execute the memory task. Although the amount of information learned related to a map stimulus do not change much, the learning strategies of experts and novices may differ. For instance, drawing order may change which is a sign of how spatial information hierarchically constructed is.
More detailed findings point out expert females were the most successful ones overall with a score of 74.2. Expert females were followed by novice females (69.9), then expert males (69.3) and lastly novice males (66.5). However, as mentioned above, these slight differences were not significant and we can conclude that experts and novices show no difference in map learning, unless the stimulus requires specific map knowledge that only an expert possesses.
Gender. The average score of females was 72.1 (st. dev. 18.7) with a minimum of 39.9 and a maximum of 92.8, whereas it was 67.9 (st. dev. 19.6) with a minimum of 21.5 and a maximum of 92.2 for males. Females outperformed males with a 4.1% difference and both expert and novice females were favored in their groups. However, these findings stemmed out of little differences in the performances and they were not significant F(1,55)=0.942, p=0.336).
Lastly, conducting two-way ANOVA showed that there was no statistically significant interaction between the effects of gender and expertise on drawing map elements, F(1,55)= 0.029, p=0.866).
Eye tracking
It can be clearly seen from the focus map of all participants that users’ gaze activity reflects the main structuring elements of the map stimulus (Figure 14). When visually interpreted, focus map highlighted the main road construction, water bodies and large settlements belonging to the stimulus. Because of the data collection issues (e.g. ET calibration, low tracking ratio), two recordings were discarded from the ET data (in total 54 participants).
Figure 14. Focus map for all participants
As mentioned in methodology, eye tracking statistics we considered were trial durations for both study and drawing part, the average duration of the fixations and the number of fixations per second for only study part of the task.
Trial durations
We assessed the trial durations for memory task in two aspects; study time of the map stimulus, and the drawing time of the sketch map. Figure 15 illustrates a general overview of study and drawing performances of participants regarding four main groups. The graph clearly shows that drawing took approximately twice -or in some cases more than twice- time as studying did for every user group.
Figure 15. Trial durations of four user groups during memory task
Study time. Average time for studying the map was 102.6 s (stdev 61.7 s) for experts with a minimum of 27.1 s and a maximum of 226.6 s (Figure 16a) and 81.5 s (stdev 57.6 s) for novices with a minimum of 23.2 s and a maximum of 292.8 s (Figure 16b) (Table 3). Although having had values that are close together for both experts and novices, there are differences among groups. If we classify the performances of participants regarding to study time, 17% of experts spent 0-50 s, 41%; 50-100 s, 21%; 100-150 s, and 21%; 150 s and more. On the other hand, 35% of novices spent 0-50 s, 46%; 50-100 s, 6%; 100-150 s, and 4%; 150 s and more. Results confirm that most of the experts allocated more time in studying than novices did.
Having gender differences explored in detail shows that novice males studied the map for 82.5 s, while novice females spent less time 78.3 s. On the other hand, the average study time for expert females was 112.2 s and for males, it was 91.5 s. Accordingly, both female and male experts allowed themselves more time than the novices did. Due to Cohen’s d effect size calculations, no significant difference occurred in terms of expertise or gender (Table 1).
Figure 16a. Study time of experts (green line: average, blue: male, pink: female)
Table 1. Study time
User groups | M | SD | ANOVA | Cohen’s d |
F | p | d | Effect size |
Experts (N=24) | 102.680 | 61.709 | | | | |
Novices (N=30) | 81.539 | 57.643 | 1.567 | 0.216 | 0.355 | Small effect |
| | | | | | |
Female (N=20) | 100.327 | 54.724 | | | | |
Male (N=34) | 85.410 | 62.813 | 0.018 | 0.894 | 0.249 | Small effect |
Figure 16ba. Study time of novices (green line: average, blue: male, pink: female)
Drawing time. As well as the study part of the memory task, there was not any time limitation for the drawing part. Average drawing time for novices was 195.4 s (stdev 75.6s) with a minimum of 76.5 s, with a maximum of 356.1 s (Figure 17a) whereas it was 253.5 s (stdev 263.0 s) for experts with a minimum of 50.2 s and maximum of 1169.4 s (Figure 17b) (Table 2).
Figure 17a. Drawing time for experts (green line: average, blue: male, pink: female)
Time spent during sketch map creation might correspond to the richness of details depicted in the sketch map, the difficulties encountered due to the lack of experience (i.e. unfamiliarity of the task and with the drawing tool), or remembering issues. Since the average drawing time of experts is greater than of novices, some experts spent the longest time for the task. Extreme values occurred in expert group can be explained by the richness of main structuring elements on the sketch maps. These sketch maps were detailed, contained larger amount of structuring elements and scored higher than the average among their group. Unlikely in expert group, there was more balanced trend in novices (Figure 17b). However, within one-third time of experts, some novices –those who spent the longest time- got equal scores as experts did with their sketch maps.
Figure 17b. Study time of novices (green line: average, blue: male, pink: female)
Table 2. Drawing time
User groups | M | SD | ANOVA | Cohen’s d |
F | p | d | Effect size |
Experts (N=24) | 253.470 | 262.949 | | | | |
Novices (N=30) | 195.433 | 75.654 | 8.032 | 0.007 | 0.316 | Small effect |
| | | | | | |
Female (N=20) | 245.085 | 243.837 | | | | |
Male (N=34) | 207.193 | 184.348 | 3.201 | 0.079 | 0.182 | Small effect |
The average duration of the fixations. While studying the map, the average duration of the fixations was 230.0 ms for experts and 244.1 ms for novices. This value was 234.0 ms for females, and 240.1 ms males (Table 1). Although the average duration of fixations of novices was slightly higher than it was for experts, there was only a little difference between expert and novice group, as well as between females and males. None of the differences were significant (Table 3).
Table 3. Average Fixation Duration [ms] while studying the map
User groups | M | SD | ANOVA | Cohen’s d |
F | p | d | Effect size |
Experts (N=24) | 229.933 | 50.104 | | | | |
Novices (N=30) | 244.133 | 48.439 | 0.074 | 0.787 | -0.123 | Small effect |
| | | | | | |
Female (N=20) | 233.980 | 56.352 | | | | |
Male (N=34) | 240.082 | 45.283 | 1.001 | 0.322 | -0.289 | Small effect |
The number of fixations per second. The mean number of fixations per second for the stimulus was 3.561 (stdev=0.765) with 3.519 (stdev=1.015) for experts and 3.596 (stdev=0.500) for novices. Having interpreted in more detail, this value was 3.351 for expert females, 3.717 for expert males, 3.505 for novice females, and 3.624 for novice males. The mean number of fixation of novices and experts slightly differed, as well as it did for females and males. Regarding to Cohen’s d effect size calculations, no significant differences emerged (Table 4).
Table 4. Number of fixations per second while studying the map
User groups | M | SD | ANOVA | Cohen’s d |
F | p | d | Effect size |
Experts (N=24) | 3.519 | 1.015 | | | | |
Novices (N=30) | 3.596 | 0.500 | 2.398 | 0.128 | -0.100 | Small effect |
| | | | | | |
Female (N=20) | 3.405 | 1.111 | | | | |
Male (N=34) | 3.654 | 0.454 | 6.298 | 0.015 | -0.326 | Small effect |
Post-test questionnaire
Having completed the memory task, participants answered questions about personal characteristics such as age, gender, native language, ethnicity and education. They also indicated if it was their first time participating such user test and left a testimony about what they think about the experiment. Eight female and eight male experts participated in such user experiment with ET before. Although it was for different purposes, two novice males indicated that they participated in a user study before. Remaining 38 participants took part in such user testing for their first time. Table 5 gives an overview about the participants’ profile.
The age of 96% of participants ranged between 18 and 34, which corresponds to quite a young user group. As majority of the participants were Belgian, there were six Asian expert participants. The native language of most participants was Dutch and Chinese was the second most spoken language within the participants.
Feedback of both experts and novices about the experiment showed similarities such as close amount of people found the task hard, interesting or either uncomfortable.
Table 5. User profile based on post-test questionnaire.
| Novices (N= 32) | Experts (N=24) |
| | | |
Age | 18-24 | 29 | 1 |
| 25-34 | 3 | 21 |
| 35-44 | 1 |
| 45-54 | 1 |
|
Etnicity | White / Caucasian | 31 | 18 |
| Asian | 6 |
| Mixed | 1 |
|
Native language | Dutch | 28 | 17 |
| Dutch & French | 2 |
| English & Dutch | 1 |
| Chinese | 5 |
| Turkish | 1 | 1 |
| Mongolian | 1 |
|
Highest level | High school graduate | 28 |
of education | Bachelor's degree | 4 |
| Master's degree | 22 |
| Doctoral degree | 2 |
|
Thoughts about | Interesting | 20 | 18 |
experiment | Odd | 2 |
| Hard | 4 | 4 |
| No specific thoughts | 2 |
| Discomfort (hard to stay still or concentrate) | 2 | 2 |
| Stating the purpose of the experiment | 2 |
| | | |
Conclusion
We used digital sketch maps to understand the cognitive abilities and limitations of map users during a memory task via drawing. On one hand, we tried to assess the quality of sketch maps based on the drawn elements, which, we think, will reflect the performances of different user groups and may reveal significant insights about their cognitive processes and strategies of retrieving a spatial information. On the other hand, we integrated ET statistics to quantify the cognitive processes to be able to put forward time-related, gaze activity-related (especially fixations) analyses. We also derived the order with which the sketched objects were drawn from ET data. The order of drawing offers a significant insight of hierarchical construction of cognitive map and may unveil the differences in the retrieval strategies of experts and novices, if there are any.
The task required of recalling the main structuring elements belonging to a screen map. This retrieval act involved WM-LTM transitions, such as retrieving spatial information stored in WM through LTM or strategies for constructing hierarchy in between map elements.
We regarded visual variables such as location, shape, size and colour as they were equally important on the drawing order. There were several reasons behind it. Although Bertin (1983) pointed out that colour is useful for forming the instant images required for effective cartographic communication, our findings did not fully support the Bertin’s assumption. The order of drawing differed for experts and novices, so that experts drew map elements in red (roads) in the first place, whereas novices drew the ones in blue (hydrography). However, the common characteristic of the first drawn elements was that they both contained linear objects. This finding shows that the shape of the map elements is at least as important as colour. In fact, some studies revealed that colour usually has little effect on recalling map information (Patton and Slocum 1985; Potash, Farrell, and Jeffrey 1978; Shurleff and Gieselman 1986; Thorndyke & Stasz, 1980). Additionally, hydrography category included lakes, which were areal representations. Drawing areal elements instead of linear ones (or in our case, polygons (lakes) and lines (rivers) were a part of a whole (hydrography)) in the first place is a sign that the size also mattered when recalling map information.
Based on assessment of sketch maps considering aggregated analysis of presence & location, shape, size, and colour of drawn elements, we found out that neither expertise, nor gender differences was influential on retrieval of spatial information. The findings that indicate males and females do not differ on map performance correspond to what was found by Beatty and Bruellman 1987; Golledge et al. 1995; Lloyd and Steinke 1984; Patton and Slocum 1985. The same applies for the fact that the influence of expertise was not significant (Thorndyke & Stasz, 1980; Gilhooly et al. 1988). The fact that novices and experts do not differ much in terms of how they learn and remember map-related information can be explained with the general map knowledge that steps in when both user groups saw a typical planimetric map stimulus. Hence, various levels of map experience may result in modest differences (Kulhavy & Stock, 1996). Besides the maplikeness and the simplicity of the map, the task to be executed is influential on performance. It is important to keep in mind that if the task required domain-specific knowledge about geography or related areas, experienced users would perform superior compared to novices (Kulhavy & Stock, 1996; Thorndyke & Stasz, 1980). Although the individual factors, other than expertise and gender, might likely to have an effect on results, the sample size was not sufficient to draw conclusions regarding to ethnicity or native language.
Furthermore, eye tracking statistics (trial duration, average fixation duration and the number of fixations per second) also revealed that there is no significant difference among expert and novice groups, as well as female and male groups.
This research can be extended to explore additional insights offered by brain imaging techniques such as electroencephalogram (EEG). The reason is that the cognitive processes emerge from both overt and covert attention. As we can easily detect overt attention with ET method, covert attention can be observed by EEG as a subsidiary resource to understand the complete cognitive process with the recall task of spatial memory.