Applied side of my dissertation UX Research on Digital Concept Mapping

Approach

Integrated in the roadmap of OASYS, an assessment platform developed by the University of Luxembourg (LUCET team), the applied side of my Ph.D. focused on the human-centred design of a digital concept mapping tool, building on prior work by Katja Weinerth and Eric François. This approach had two objectives:

• publish an easy-to-use concept mapping tool for schools in Luxembourg, and
• integrate the digital concept mapping tool as a task type for assessment

I will outline the main approach of my human-centred design work below.

Benchmarking and Contextual Research

Based on research on the web, the app stores, scientific literature, and user research, my benchmarking included 34 digital tools for concept mapping and related knowledge visualisation methods. Furthermore, I performed extensive literature research on knowledge visualisation methods, and included a variety of different concept mapping variants in my follow-up studies.

User research

• Co-design studies with 67 students in classrooms contributed to identifying what the concept mapping tool needs to do and how it should appeal. Furthermore, I derived information about the context of use, for example devices used for digital concept mapping or expectations and use cases about digital concept mapping experiences.
• Storytelling allowed diverse participants to share and discuss how they envision using digital concept mapping in learning and teaching activities, as well as which pain points and desired outcomes they expect.
• I performed 17 interviews about experiences with concept mapping and other methods of knowledge visualisation. A particular focus was on psychological needs in concept mapping, investigating with UX cards (Lallemand 2015). Furthermore, I used these interviews to gain further insights into the context of use and expected functionalities of a digital concept mapping tool.
• Together with Bo Raber, a doctoral researcher in the related "School Futures" project, we performed extended observations of learners during collaborative, concept-mapping-based activities, both on paper and in digital tools. We used these insights to identify further requirements and context of use information.

Definition of Requirements

I derived requirements and collected ideas from user research into cards and performed a workshop with stakeholders and developers to gather their insights on technical feasibility and other important considerations, such as whether the requirement has potential as a differentiator compared to other benchmarked tools. I developed a template to collect key information about each requirement on one card.

I organised several requirements prioritisation workshops with different stakeholder groups. The results were included in a product requirement document (PRD) and scheduled along a roadmap, together with other features in the toolset of OASYS.

Prototyping, testing, and iterative improvements

Based on a earlier work by Katja Weinarth and Eric François, I created and iteratively validated three prototypes of a digital concept mapping tool:

• First lab-based qualitative user experience test with 35 participants
• Second lab-based qualitative user experience test with 31 participants
• Third quantitative user experience test with 71 participants as a field study
• Fourth remote qualitative user experience test (with 6 participants) and expert review (with 4 experts) of a published alpha version

All tests used different concept mapping tasks to investigate different uses of digital concept mapping. Methodologically, I used both quantitative (e.g. UEQ) and qualitative methods (e.g., interviews). Results were used for iterative improvements of follow-up versions. Finally, I derived design recommendations that I submitted to the University of Luxembourg for potential future enhancements.

Further activities

Although the user experience design of the digital concept mapping tool was the main applied outcome of my work at the University of Luxembourg, it is not sufficient to successfully implement concept mapping in education. Consequently, my work also covered further activities:

• I created pedagogical materials (including sample lesson plans) for teachers, containing a wealth of information about concept mapping tasks and assessment methods with concept maps.
• I also worked on guidelines and a scoring rubric for using visual features in concept mapping, which was a frequent user need throughout the studies.
• Together with Susanne Backes, I published results of a study in the National Education Report 2022.
• Finally, I was involved in dissemination activities to promote the developed concept mapping tool and connected closely to the development team to support future enhancements along the defined roadmap.

Results

Besides academic outcomes, my research project provided a solid, human-centered foundation for a digital concept mapping tool. In particular, the results were:

• more than 70 usability and user experience tests, using various methods and focus areas
• more than 10 classes involved in various design activities
• close collaboration with School Futures research project and three partner schools
• 5 documents with user experience results and design guidelines for the development team (e.g., test reports, contextual research, benchmarking)
• 4 design workshops with human-computer interaction experts

Finally, there were some take-aways and learning. During my Ph.D. work, I could not cover every aspect of a successful digital concept mapping tool, especially due to further academic tasks, teaching opportunities, and a strict time limit of my doctoral and postdoctoral work. For example, my studies focused on the user interface for creating concept maps, but the user interface for creating concept mapping tasks would need further work. Also, combining applied UX research and academic research comes at the cost of making compromises: For example, by building on prior work and an existing code base, my project did not consider design possibilities outside these guardrails. More work would also be needed in exploring collaborative concept mapping, but my Ph.D. focused on individual concept mapping. Finally, I would have loved to investigate concept mapping as a method for UX work, as I outlined briefly in an article in t3n.