In an innovative and fun move, Adobe has launched new AI filmmaking tools that allow users to create custom sound effects through simple voice recordings, opening up new horizons for creators and setting a new standard in controlling audio and video generation.
Introduction: A revolution in the audio industry
Adobe is launching new generative AI filmmaking tools, offering fun ways to create sound effects and control generated video outputs. Alongside familiar text prompts that typically allow users to describe what Adobe’s Firefly AI models should create or edit, users can now use onomatopoeia-like vocal recordings to generate custom sounds, and use reference footage to guide movements in videos generated by Firefly.
This step represents a major development in the field of creative tools, as input is no longer limited to text, but now includes sound and motion, making the creative process more intuitive and interactive. Through these tools, Adobe seeks to maintain its position at the forefront of creative software, even if it lags behind competitors in developing the generative models themselves.
“Generate Sound Effects” tool: From “clip-clop” to horse hooves
Launching in beta on the Firefly app, the “Generate Sound Effects” tool can be used with recorded and generated footage, providing greater control over audio generation compared to Google’s Veo 3 video tool. The interface resembles a video editing timeline and allows users to match the effects they create with uploaded footage.
For example, users can play a video of a horse walking along a path and simultaneously record “clip-clop” sounds with its hoofsteps, alongside a text prompt saying “hooves on concrete.” The tool will then generate four sound effect options to choose from. This builds on the Project Super Sonic experiment Adobe showcased at Max in October. The tool does not work with speech, but supports generating impact sounds like breaking twigs, footsteps, and zipper effects, as well as ambient sounds like nature and city atmospheres.
Advanced video generation controls
New advanced controls are also coming to Firefly’s text-to-video generator. The “Composition Reference” feature lets users upload a video alongside their text prompt to mirror the composition of those shots in the generated video, making it easier to achieve specific results than repeatedly entering text descriptions. Keyframe cropping will allow users to crop and upload images for the first and last frames that Firefly can use to generate video in between.
Additionally, new style presets offer a range of visual styles users can quickly select, including anime, vector art, claymation, and more. These presets are only available for use with Adobe’s Firefly video model. The results might leave something to be desired if the live demo seen was any indication—where the “claymation” option looked more like early 2000s 3D animation.
Adobe’s strategy: The platform, not the model
Adobe continues to add support for competing AI models within its own tools. Alexandru Costin, vice president of generative AI at Adobe, stated that similar controls and presets could become available for use with third-party AI models in the future. This indicates that Adobe is competing to maintain its position at the top of the creative software chain as AI tools grow in popularity, even if it trails the likes of OpenAI and Google in the generative models themselves. Adobe’s strategy is clear: be the ultimate platform used by creators, regardless of the AI model powering it.
Conclusion: The future of multimodal creativity
Adobe’s launch of these new tools marks an exciting step toward a future where AI-assisted creativity is more accessible and interactive. By allowing users to use their own sounds and movements as inputs, Adobe breaks down traditional barriers between creator and tool. It is no longer just about telling the AI what to do, but showing it and interacting with it in more natural ways. This multimodal approach has the potential to unlock new levels of creativity and put powerful AI tools in the hands of a broader range of people.
