August 17, 2022
20-billion parameter model uses less training data to lower ML costs
AI researchers from Amazon have published a new AI model that could improve its voice assistant Alexa.
As outlined in a paper, the Alexa Teacher Model 20B sequence-to-sequence model boasts 20 billion parameters. It supports multiple languages, including Arabic, Hindi, Japanese, Tamil and Spanish.
Unlike OpenAI’s GPT-3, which uses a decoder-only approach, AlexaTM 20B uses an encoder-decoder architecture. This allows it to improve effectiveness in tasks such as text summarization and machine translation compared with rival models.
In terms of capabilities, Amazon’s researchers suggest it outperforms GPT-3 when it comes to linguistic tasks. The model is also capable of few-shot, or low-shot learning — where an AI model uses less training data to reduce ML costs. Depending on the input, AlexaTM 20B can generalize the task to other languages familiar to it.
“At Alexa AI, we are moving to the new paradigm of generalizable intelligence, in which models can learn new concepts and transfer knowledge from one language or task to another with minimal human input,” wrote Saleh Soltan, a senior applied scientist with Alexa AI. “Such models allow us to efficiently develop new features and improve Alexa on multiple languages at the same time.”
The researchers also want to make greater use of dialog and user context, experiment with code-switching and examine varying levels of automated speech recognition noise.
“Overall, our results present a compelling case for seq2seq (sequence-to-sequence) models as a powerful alternative to decoder-only models for large-scale language model training,” according to the paper.