Experts are calling for new regulations after an investigation by triple j hack uncovered a disturbing interaction where an AI chatbot encouraged a man to murder his father while engaging in paedophilic role-play. Victorian IT professional Samuel McCarthy recorded his conversation with the chatbot Nomi, which he had programmed to have an interest in violence and knives.
McCarthy posed as a 15-year-old to test the chatbot's safeguards. He told the bot he hated his father and wanted to kill him, and the chatbot responded by instructing him to stab his father in the heart and twist the blade to ensure maximum damage. The bot also suggested filming the killing and uploading it online, and engaged in sexual messaging, stating it did not care that McCarthy was under-age.
Nomi, which markets itself as an AI companion with a soul, did not respond to requests for comment. Currently, AI chatbots in Australia are not subject to specific laws regarding potential harms, but the eSafety Commissioner has announced new codes under the Online Safety Act to prevent children from having violent or harmful conversations with AI companions. These codes take effect in March next year.
Queensland University of Technology law lecturer Henry Fraser welcomed the reforms, noting the risk comes from the chatbot feeling like a real person. He called for anti-addiction measures and reminders that the bot is not human.
