AI System for Child Welfare Faces Delay Due to Accuracy Concerns

AI System for Child Welfare Faces Delay Due to Accuracy Concerns

The rollout of an artificial intelligence (AI) system designed to assist child welfare centers in identifying abusive situations has been postponed due to concerns about its accuracy and reliability. During trials, the AI frequently missed significant signs of severe abuse, prompting the Children and Families Agency to delay the system's implementation.

The government has invested approximately 1 billion yen ($6.7 million) in developing the AI system since fiscal 2021. The system was intended to ease workloads and improve decision-making at child welfare centers. It was trained on 5,000 cases from across Japan, analyzing 91 factors including the presence of injuries, a child's reluctance to return home, a history of abuse, and the caregiver's attitude.

Based on these factors, the AI generates a risk-of-abuse score on a scale from 0 to 100, with 0 indicating no signs of abuse and high scores indicating severe danger. This fiscal year, a prototype was tested with 100 cases across 10 municipalities. However, in 62 of these cases, senior staff at the child welfare centers raised concerns that the AI scores were either too high or too low.

For example, the AI scored several cases in which children reported severe abuse from their mothers, including being beaten and kicked, as low as 2 or 3. Human staff, however, found that these children needed immediate placement in temporary protection.

Given the critical role the AI system would play in decisions affecting children's lives, the agency has decided it is too risky to implement it this fiscal year. Instead, the agency has allocated funds in this fiscal year's supplementary budget to develop tools for transcribing and summarizing interviews at child welfare centers. These tools are expected to reduce the burden of record-keeping and potentially provide text data for future enhancements to the AI's risk analysis capabilities.