Roblox Rolls Out AI-Powered Real-Time Rephrasing Of Profanity Within Chat
The power of the latest generation of AI systems is such that previously impractical applications are not just possible, but scalable. For example, moving beyond basic early AI text translation tools, it is now possible to use live translation to communicate in another language in real time. For many people that will be a real boon, especially when they are traveling. But here’s something that is likely to prove more controversial: real-time rephrasing of profanity within chat. It’s a new AI-powered feature from Roblox that is designed to “keep gameplay fluid while maintaining civility within chat”:
Roblox is leveraging AI to automatically rephrase profanity. Rather than displaying only hashmarks, filtered text will be translated into more respectful language that remains closer to the user’s original intent. For example, a message that violates Roblox’s profanity policies, such as “Hurry TF up!” would previously have appeared as “####” within experience chat. That will now be rephrased to “Hurry up!” This new layer is designed to maintain civility by rephrasing the language and replacing “stop signs” with real-time guidance.
Specifically:
When a message violates Roblox’s profanity policy, everyone in the chat is notified that the text has been rephrased to keep things civil. While rephrasing reduces some of the disruption in the chat, Roblox’s multilayered safety system remains in effect for more serious behavior. Rephrasing is available exclusively for in-experience chat between age-checked users in similar age groups and is supported in all languages currently available through Roblox’s automatic translation tools.
Alongside this new AI-based capability, Roblox is also tweaking its text filtering system:
Early results from Roblox’s testing show significant improvements in detecting leet-speak, or letters replaced with numbers or symbols, and more sophisticated attempts to bypass filters.
Parents may applaud real-time rephrasing as a way for the service to nudge younger users away from bad language in their interaction with others, without stopping them playing altogether. But it creates a dangerous proof of concept that others may build on, particularly in jurisdictions that want stricter controls on what people say online.
It’s easy to imagine situations where Chinese AI systems, for example, rephrase people’s language on social media in real time to promote “social harmony”. Not only the style but even the content’s details could be subtly changed away from controversy towards conformity. It would be possible for rephrasing to be visible only to others, so the person making a comment might not even be aware that their words were being subverted in this way. Something similar is already happening with Chinese AI chatbots that censor their own answers, without acknowledging that fact. As Chinese AI companies become increasingly important players in the online world, this kind of covert rephrasing by them — and others — is another issue people will need to watch out for in our brave new AI world.