Creators of ‘Attention is All You Need’ explore AI possibilities beyond Transformers at Nvidia GTC, emphasizing the necessity for an improved solution.

The creators of the groundbreaking ‘Attention is All You Need’ paper, which introduced Transformers, recently gathered for a panel discussion at the Nvidia GTC conference. This group of seven out of the eight authors, including Noam Shazeer, Aidan Gomez, Ashish Vaswani, Llion Jones, Illia Polosukhin, Jakob Uskhoreit, and Lukasz Kaiser, came together to discuss the future possibilities of AI models beyond Transformers. Niki Parmar, the co-founder of Essential AI, was unable to attend the event.

Transformers, a neural network NLP breakthrough developed by the team at Google Brain in 2017, revolutionized natural language processing by capturing the context and meaning of words more accurately than previous models. The Transformer architecture has since been used in various language applications, including GPT-4 and ChatGPT by OpenAI and AlphaFold by DeepMind.

However, the creators of Transformers are now looking ahead to what comes next for AI models. Aidan Gomez from Cohere expressed that “the world needs something better than Transformers” and hoped for a new model that would push the boundaries of performance. He emphasized the need for innovation and progress in AI models to provide everyone with access to superior models. Gomez highlighted the inefficiencies in memory usage and architectural components of Transformers that need to be re-explored and reconsidered.

While the panelists agreed that there is a need for a superior AI model, they also acknowledged that it is crucial for the new model to be clearly and obviously better than Transformers. Llion Jones from Sakana AI stated that it’s not enough to be technically powerful; the new model must excite and captivate the community to drive momentum and encourage a shift to the next architecture. Gomez echoed this sentiment, emphasizing the importance of both a good model and architecture as well as excitement from the community.

The discussion at the Nvidia GTC conference highlighted the collective desire among AI researchers and developers for continuous progress and improvement in AI models. The creators of Transformers recognize the limitations of the current model and are eager to explore new possibilities. As the AI industry continues to evolve, it is clear that innovation and excitement from the community are essential drivers for advancing AI technology.

Overall, the panel discussion at the Nvidia GTC conference revealed the aspirations of AI researchers to develop models that surpass Transformers in terms of performance and efficiency. The creators of ‘Attention is All You Need’ emphasized the need for innovation, re-exploration of architectural components, and the importance of excitement from the community to drive progress in the field of AI. With their collective expertise and determination, it is only a matter of time before a new and improved AI model takes center stage, pushing the boundaries of what is possible.