The rapidly advancing field of artificial intelligence (AI) is witnessing the emergence of large language models (LLMs) with unprecedented capabilities. Recently, a newly released LLM demonstrated a potential understanding of its own thought processes, sparking discussions about AI’s potential for self-awareness. However, the true significance lies in the sheer power of these models as they continue to grow larger and more sophisticated.
As LLMs become more advanced, so do their capabilities and costs. The training costs associated with the latest models are soaring, with some approaching $200 million. This exponential growth in performance also means exponential growth in costs. In fact, it is predicted that by 2025 or 2026, training the latest models will cost $5 to 10 billion dollars, making it unattainable for all but the largest tech giants and their partners.
The rising costs can be attributed to the increasing complexity of these models. Each new generation has a greater number of parameters, enabling more complex understanding and query execution. Additionally, more training data and larger amounts of computing resources are required. These escalating costs mirror the trajectory of the semiconductor industry, where the costs for each new generation of equipment and fabrication plants grew alongside exponential improvements in chip performance.
In response to these rising costs, AI companies are following a similar path to the semiconductor industry. Just as many semiconductor companies chose to outsource their manufacturing to reduce costs, AI companies may also opt for outsourcing. This trend can already be seen with smaller language models (SLMs) that offer several billion parameters instead of the trillion parameters found in larger LLMs like GPT-4. Microsoft’s Phi-3 is an example of an SLM with 3.8 billion parameters. While these smaller models may not offer the same level of performance as their larger counterparts, they are cost-effective alternatives for certain applications.
However, the increasing costs pose risks to AI innovation and diversity in the field. The high entry barriers resulting from these costs may limit AI development to a few dominant players, stifling creativity and reducing the range of ideas and applications. To counterbalance this trend, it is crucial to support smaller, specialized language models that provide critical capabilities for niche applications. Promoting open-source projects and collaborative efforts can democratize AI development and ensure a more inclusive and diverse future for the technology.
In conclusion, the integration of AI in business is a complex and rapidly evolving field. The advancement of large language models brings both new capabilities and skyrocketing costs. Understanding the trajectory of the AI industry in relation to the semiconductor industry provides valuable insights into the potential future scenarios. By supporting smaller language models and fostering collaboration, the industry can ensure that AI development remains accessible and beneficial to a wide range of participants.