‘A double-edged sword’: Potential uses of AI in the Children’s Hearings System

New research from the Scottish Children’s Reporter Administration (SCRA) tackles questions many in our sector currently face regarding how AI could and should safely support our work. In the context of the Children’s Hearings System, Dr Annie Taylor and Jennie Laurenson explored the opportunities and challenges AI poses and a sought a way forward that prioritises the safety and rights of children and young people. Here, they share insights from their timely research. 

AI is everywhere. It’s in everything from Spotify recommendations to weather forecasts, and now it’s increasingly popping up in our workplaces. At the Scottish Children’s Reporter Administration (SCRA) we’ve been thinking about using AI in the Children’s Hearings System, part of the child protection and youth justice system in Scotland.

Previous research has found that although people can see the potential benefits of AI, they worry about how it could affect a whole range of things, including privacy and safety online, misinformation, and how it could change human relationships and jobs, quality of life, and inequalities.

At SCRA, we wondered how AI would fit into our job to keep children safe and protected. Children, young people and families in the Children’s Hearing System are often going through a difficult time, so it’s crucial that we prioritise their safety and rights. This means thinking carefully about any potential risks before using AI. To do this properly we needed to hear from people who are involved in the system. 

What did we do?

We held research workshops with 163 people, including professionals in and around the Children’s Hearings System, and young people and families. We asked them what they thought about AI, how they already used it, and what they thought the positives and negatives might be of using it in the Children’s Hearings System. 

What did we find out?

Knowledge uses and perceptions

People were often surprised by how much AI they already used. Many understood AI in the context of increasingly popular large language models like ChatGPT and Copilot, or personal assistants like Alexa, but often didn’t realise that AI was embedded in their favourite websites and apps.

Adults often assumed that young people were more interested and comfortable with AI then they were. The young people we spoke to weren’t as excited about AI as adults expected them to be, and mainly used it for fun, not for serious tasks.

People debated whether AI was more neutral than humans. Some said AI doesn’t have the baggage of human bias, but others pointed out that AI is programmed by humans and built upon human data, meaning it acts as a mirror, reflecting the bias, inequality and harm that already exist in our society.

While some felt that AI use might be okay for admin tasks, there was a strong feeling that AI couldn’t replace human interaction, relationships, connection and decision-making – a human would always need to check and take responsibility for anything it produced.

AI’s impact on children and young people

We know it can be difficult for children and young people to participate in hearings. Everyone said the Children’s Hearings System could be better at helping children and young people participate but didn’t always agree about whether and how AI should be used for this. AI could potentially be used to help gather information from case files, and make reports easier to read, but would it really be better at this than humans? Some thought it could have a role, but only with human involvement, to make sure everyone’s needs were met.

And what about children using AI online? Participants were very aware of the risks it could bring to young people, including through deepfake photos or videos, or exposing them to grooming, potentially leading to real life abuse.

We need to make sure that any AI we use makes things better for children, rather than just saving money.

Benefits and risks of use

Although people could see the potential benefits of AI in the Children’s Hearings System, they also had concerns. People said AI could: make some tasks quicker so staff could spend more time building relationships; allow easier access to information; improve accessibility; and analyse huge amounts of data quicker than humans. But questions were raised about the risks:

• AI can be inaccurate, so staff would need to check (and potentially correct) everything AI created. Could AI accidentally increase, rather than decrease, workloads?
• AI tools can’t understand nuance, so could AI outputs make trauma worse for vulnerable children, young people and families?
• How would personal and sensitive information be kept safe and private?
• Would AI make decisions, and would these be based on incorrect or incomplete data?
• How would people know whether AI or humans were making decisions about them?

 Moving forward

People said that to keep children, young people and families safe if AI is used: humans should always check and control AI work; AI should not make decisions about or do relational work with children, young people and families; and AI tools should be carefully planned, with rules about their use created before they’re introduced, and humans monitoring them while they’re used.

What does this mean for the Children’s Hearings System?

So what do we do now? We’ve used our research to begin creating a set of ethical principles that can be used whenever we consider using an AI tool. We’re still working on these, but they will likely include:

• not using AI for anything replacing human interaction or decision-making;
• always checking to see if the benefits are worth the cost involved;
• making sure privacy guidance is followed;
• working with people who will be affected by/ using the tool;
• being transparent about how and why each tool will be used and managed;
• ongoing planning, regulation and monitoring.

Sticking to these ethical principles should help ensure that any AI tool we use is ethical, safe, and upholds the rights of those we’re here to help.