A+

The end-of-year school report card…

How can one document bring so many strong emotions?

As a provider of educational software systems, my client wanted to identify opportunities to do report cards better. How could they make them easier for teachers to prepare, and make them more meaningful for parents and students. A report card on report cards, if you will.

Factually speaking, academic reports are not popular. Teachers don’t like writing them; and parents and students don’t get much value from them. What would a better academic report look like? What even is a report card meant to do?

Those questions aren’t necessarily simple to answer. Consider the four groups which are affected by academic reports in very different ways:

• Teachers and their schools
• Students and their parents

And then the significance of an academic report varies, depending on whether the school itself is:

• Primary, secondary, or K-12
• Public, private, or independent
• Metropolitan, regional, or rural

And that’s not to mention other considerations, such as the number of children in a family; household income; and cultural or ethnic factors…

Why does this matter?

Academic reporting affects virtually every school student from every walk of life across the country (and potentially beyond). There are real commercial opportunities if my client can make academic reporting:

• Easier for teachers
• More meaningful for students and their parents
• Ideally both!

It’s equally risky, though. It would be catastrophic to roll out software which makes life harder for teachers or less helpful for families.

So how does a client make sure they’re solving people’s problems in the right way? To start with, by listening far and wide.

The brief

At a glance

Key objectives

Most teachers dread writing academic reports, because it’s time consuming and painful.

Academic reports have no impact on a student's learning journey.

Parents don’t care about academic reports, as they are very generic.

The client

For reasons of commercial confidence, I won't name my client in this case study.

Project team

UX researcher, Askable's Enterprise Success Manager

Project involvement

10 days

Project duration

Four weeks

All three of those hypotheses are steeped in frustration.

Frustrations are born from a mismatch between a person’s expectations and their progress towards a goal. The client’s hypotheses suggest there’s a mismatch between academic reports and improving student outcomes.

As a result, the client wanted to go back to basics, to understand what teachers and parents expected from academic reporting in the first place.

The project's foundations

In UX research, our challenge is to analyse things that are fuzzy, like peoples' feelings. To do this, we ask a lot of questions like, 'is it this or is it that?'. As a result, when we describe our own work, we use a lot of specific 'this or that' labels. For this project, my approach used the following labels –

What did people expect from academic reporting, and why? Hearing people talk about their feelings would be important. Therefore I would use interviews to collect qualitative evidence.

The client wanted to discover the origins of expectations and frustrations regarding academic reporting. Also, the research was being conducted before the client started making changes to their product. That meant I'd use a generative approach to understand peoples' experiences.

Academic reporting is a complex and open-ended topic. I would need to tailor the interview questions on-the-fly. The research interviews would therefore be moderated.

Because academic reporting is so complex and open-ended, and affects different groups in different ways, the client knew they needed to listen to lots of people. Their brief called for 50 participants.

Interviewing that many people would be prohibitively expensive. But something like a survey would only gather numbers (or long-form answers of wildly unpredictable quality). How would the client discover helpful sentiment insights?

Listening at scale

Let’s come back to my client’s study...

At the time, Askable’s AI moderated interviews were in beta testing. The client agreed to participate.

The study would be split in two – one set of questions for teachers, and another for parents. For each group, I’d run four moderated interviews, and then the AI system would conduct 25 interviews per group.

Their study would get the volume of responses needed, while avoiding the expense of conducting a large number of interviews.

How does it work?

Askable’s system incorporates a small number of moderated interviews, which act as guides for the AI system. My interview questions and style shape the AI’s understanding of the study’s intentions.

After that, the research participants can begin a video call with the AI system at their convenience. They are asked questions by the AI system. Questions are presented both verbally and visually, which means participants are remarkably comfortable answering in a natural and thoughtful way.

The AI can ask intelligent follow-up questions, and can distribute the frequency of a given question or theme across the participant population.

Recruitment

The client's brief asked for research participants to come from two groups:

• Teachers, including at least one head teacher
• Parents

All participants needed to be distributed between public and private schools; primary and secondary schools; small, medium, and large schools; and be Australian-based.

For the human-moderated interviews, I recruited 8 participants across the two groups. I held 45-minute online sessions with each of them.

For the AI-moderated interviews, I recruited 50 participants across the two groups. The system used a trimmed-down set of objectives, and its sessions averaged about 8 minutes per person.

Discussion guide

Fun fact: I used to be a secondary school teacher. I’ve written my share of academic reports.

The discussion guide for teachers explored three themes:

• Writing reports – Prevailing mood towards reports; the effort required; the role of automated report systems; and shortcuts and hacks teachers use to make the task easier.
• Parents’ response to reports – Teachers’ perceptions of how parents see and use academic reports; improvements which parents have suggested to teachers over the years.
• Students’ response to reports – Teachers’ perceptions of how students see and use reports.

The discussion guide for parents explored the flipside of the teachers’ themes:

• Communication with teachers – Prevailing mood towards parent-teacher interactions; frequency of parent-teacher interactions.
• Parent’s response to reports – Perceptions of academic reports; most valuable elements in academic reports.
• Parent’s response to students’ progress – How do parents engage with their children about their academic progress? Are their children invested in their own academic performance?

Interviews

As a researcher, I tend to write literal, word-for-word interview scripts. When I facilitate an interview, I tend to follow the script closely, depending on the needs of a project.

But for this project, the questions were intentionally exploratory and open-ended. I wanted teachers and parents alike to let their answers wander (within reason).

This decision was made knowing that the AI system would perform most of the thematic coding grunt work. If I were analysing 50 sessions, I would have been more prescriptive about following the script.

Analysis

Askable's platform automatically transcribes and time-codes its interview recordings.

Normally after each session, I would trawl through each transcript, and mark comments which were applicable to each research objective. For many past research studies, I’d also copy comments into a giant spreadsheet, to track each participants’ responses to each question.

For this study, that wouldn’t be necessary. Instead, the AI system parsed through the transcripts to ‘learn’ my research intentions.

The AI system generated a list of themes, and a list of insights.

Themes included:

• Parent engagement
• Value of academic reports
• Administrative burden

Insights included:

• Mixed Perceptions on Academic Report Value and Parent Engagement
• Balancing Teacher Workload and Meaningful Reporting
• Limitations of Traditional Grading in Academic Reports

Each insight appears to have been triangulated from the intersection of one or two themes. For instance, the Mixed Perceptions insight was constructed from comments which matched the themes "parent engagement" and "value of academic reports".

For each insight, the system generated a summary of participants’ comments, followed by an exhaustive list of direct quotes from participants.

Insights

For this research study, I was contracted to recruit participants and then facilitate the human-moderated interviews. And so, this is where my formal involvement ends.

But then… curiosity hit. How would I use the insights generated by the AI system to derive valuable outcomes for the client?

Askable’s system automatically creates a list of quotes, categorised by themes. This is incredibly valuable for me – it’s the slowest part of my work. This AI system frees up my time to run more analysis.

As I review participants’ comments, I’m often looking for trends across the whole cohort. I especially look for unexpected consensus or conflict in the context of a specific question. And then, I’ll look for commonality within various demographic groups, to find reasons why people express the sentiments they’re sharing.

Several investigative leads quickly emerged.

I’m so eager for Askable to add filtering tools – for instance, I want to see comments made by gender, or by age, by their answer to a screener question.

The outcome

Askable knew I could roll with the ambiguity and new-ness of its beta AI system. It’s full of promise for a researcher like myself.

As a one-off, standalone research study grappling with a complex, open-ended topic… this study might have pushed the capabilities of the system.

I can see that an ongoing program of continuous discovery research would absolutely benefit from a cost-effective solution like this.

Imagine being able to quickly and cheaply gather sentiment data from a large population. Picture being able to listen to the reasoning behind people’s sentiments. Imagine being able to build up a library of answers over time, and being able to deliver deeply relevant insights for your business.

On the basis of these sentiments, I’d give that future a very high grade.