ElevenLabs, an AI voice startup, recently introduced its Sound Effects text-to-sound AI offering. Now, the company has taken it a step further by releasing an open-source tool to demonstrate the potential of this technology. This application allows creators to generate sound effect samples for their videos by analyzing the imported clip and providing multiple options.
While developers can access the app’s code on GitHub, ElevenLabs has also created a website for the public to try out its Sound Effects API. The process is simple: when a video is uploaded, the Video to Sound Effects app extracts four frames at one-second intervals on the client side. These frames, along with a prompt, are sent to OpenAI’s GPT-4o, which uses them to create a custom text-to-sound effects prompt. The generated prompt is then used with ElevenLabs’ Sound Effects API to produce a sound effect. Finally, the video and audio are combined into a single file ready for download.
According to Ammaar Reshi, ElevenLabs’ design lead, this tool serves as a proof of concept for their SFX API. It aims to speed up the workflow for AI video creators by suggesting the best output based on the frames in their videos. Reshi emphasizes that the company is excited about the potential for dynamic experiences that can be built using this API. For example, immersive video games could generate sounds based on a player’s interaction.
The Sound Effects API allows developers to build fully custom AI sound effects using a short description. The pricing is based on character count, with options for 100 characters per generation with automatic duration or 25 characters per second with a set duration.
In a brief test, the video-to-sound effects app appeared simple yet effective. When an audio-free movie of a vehicle navigating an all-terrain environment was imported, ElevenLabs’ AI generated four options that all sounded like a car driving on a gravel road. While applying sound effects to clips is entertaining, the true potential lies in integrating this capability into larger systems to derive real benefits.
As the AI video generation space becomes increasingly competitive, ElevenLabs is committed to staying ahead by developing new audio solutions that it knows will be in high demand among developers, filmmakers, and creators. By showcasing the power and versatility of their Sound Effects API, the company is positioning itself as a leader in AI-driven audio technology.