« LOFTYSKY » is an innovative project that generates real-time AI-driven visualizations in response to both human performance and contextual prompts. The core concept is to create dynamic visuals based on live human movements, where the AI offers real-time visual feedback. This enhances the performance by adding immersive visual elements that synchronize with the performer’s movements, creating an interactive and engaging experience for the audience.
In this example, a contemporary dance performer’s movements are captured and integrated with a prompt related to a forest theme. These two data inputs—movement capture and contextual prompt—combine to produce an AI-generated animation of a forest and environment that moves in harmony with the performer, maintaining the same color palette as the live scene. The result is a seamless fusion of performance and AI animation, offering a deeply immersive visual experience.
ROADMAP
We currently support video input from cameras, external video feeds, and NDI sources. We are also planning to integrate dynamic prompt sources for enhanced flexibility and interactivity.
STORY TELLING
An example of dynamic prompt source integration: A singer performing on stage, with lyrics being transcribed in real-time as they are sung. These transcribed lyrics are instantly converted into dynamic prompts and, along with the live video feed of the performer, are fed into the LOFTSKY. The system generates visuals based on the context of the lyrics, and as the singer performs new lines, the visuals adapt in real-time. This process creates a continuously evolving visual narrative that enhances and supports the performance, adding an immersive layer of storytelling.
PROTOTYPE
LOFTYSKY is currently in the prototype phase and, while there are limitations, its potential is boundless. Built on the TouchDesigner platform, an advanced audio/visual programming environment, LOFTYSKY offers the flexibility to integrate with cutting-edge technologies. With strong community support, the platform enables continuous innovation and growth, paving the way for an exciting future.
TECHNOLOGY
LOFTYSKY leverages Stream Diffusion technology, heavily reliant on CUDA drivers and NVIDIA GPUs for optimal performance. The integration of Stream Diffusion into TouchDesigner has been expertly facilitated by DotSimulate, enabling seamless real-time visual generation.
For this demonstration, LOFTSKY is running on an Alienware system powered by an NVIDIA RTX 3060 GPU, ensuring high-performance processing and rendering capabilities. Further information on both TouchDesigner and Stream Diffusion is available for those interested in exploring the underlying technologies.