FFMPEG Video Art, Generative Audio and Ken Perlin
#051 - Creative Coding / Generative Arts Weekly
Website | Instagram | Youtube | Behance | Twitter | BuyMeACoffee
Silence creates space for content to appear. -J.R. Rim
Video Art with FFMPEG
One of the many tools we use to describe motion is a sequence of images or videos. We love them and use them for our work as it adds yet another dimension to the photos. The following few selections all use FFMPEG, an open-source tool to convert video from different formats in the command line or through a programming language. But there are a few more creative uses of this tool, and Derrick Schultz has a tremendous four-week video course that he uses the tools in innovative ways and provides code artists with s another way to create some exciting work.
If you are looking for a quicker fix, this presentation by Matic Lubej on FFmpeg is an overview of how to use FFMPEG, specifically in Python creatively.
Historically, video art doesn’t have too many generative aspects, but I can see that there are a lot of possibilities.
Have a great week!
Chris Ried
🖌️ Unconventional Media
Matrix-Webcam
This package displays your webcam video feed in the console.
It’s a fun little tool to play around with; it’s open source, so the code…
📸 Generative Graphics
Ken Perlin - How to Build a Holodeck
In the age of COVID-19 it is more clear than ever that there is a compelling need for better remote collaboration. Fortunately a number of technologies are starting to converge which will allow us to take such collaborations to a whole new level. Imagine that when you join an on-line meeting you are present with your entire body, and that you can see and hear other people as though you are all in the same room. There are many challenges to realizing this vision properly. The NYU Future Reality Lab and its collaborators are working on many of them. This talk will give an overview of many of the key areas of research, including how to guarantee universal accessibility, user privacy and rights management, low latency networking, design and construction of shared virtual worlds, correct rendering of spatial audio, biometric sensing, and a radical rethinking of user interface design.
Color Palette from Image
The following Observable notebook has an example of using the Vibrant.js library. It’s an exciting way to help choose color palettes from a specific image.
🚤 AI / Deep Learning
IMAGEN VIDEO
We present Imagen Video, a text-conditional video generation system based on a cascade of video diffusion models. Given a text prompt, Imagen Video generates high definition videos using a base video generation model and a sequence of interleaved spatial and temporal video super-resolution models. We describe how we scale up the system as a high definition text-to-video model including design decisions such as the choice of fully-convolutional temporal and spatial super-resolution models at certain resolutions, and the choice of the v-parameterization of diffusion models. In addition, we confirm and transfer findings from previous work on diffusion-based image generation to the video generation setting. Finally, we apply progressive distillation to our video models with classifier-free guidance for fast, high quality sampling. We find Imagen Video not only capable of generating videos of high fidelity, but also having a high degree of controllability and world knowledge, including the ability to generate diverse videos and text animations in various artistic styles and with 3D object understanding.
We have seen a lot of innovation in the image space regarding generative deep-learning models. There is some exciting work in the audio generation/audio processing space. The following are a few more exciting work
Nutone
Neutone is a go-to platform for you to share real-time AI audio processing models with potential users in the audio production community.
Harmonia AI, Diffusion and Audio Generation Revolution
Listen to Harmonai core team's vision and how their Dance Diffusion model is the first step on the journey to high quality music and audio generation for everyone.
🔖 Articles and Tutorials
Joshua Davis - Dribble Conference 2019
Joshua Davis is a design polymath—designer, technologist, author & artist who has used creative coding to generate algorithmic images and animations since 1995.
Josh Davis is one of the most prolific artists/designers, and so much of his work provides a historical context on the progression of early generative works from 1995 to 2001.
📚Books
Real-Time 3D Graphics with WebGL2
This book presents a clear roadmap to learning real-time 3D computer graphics with WebGL 2. Each chapter starts with a summary of the learning goals for the chapter, followed by a detailed description of each topic. The book offers example-rich, up-to-date introductions to a wide range of essential 3D computer graphics topics, including rendering, colors, textures, transformations, framebuffers, lights, surfaces, blending, geometry construction, advanced techniques, and more. With each chapter, you will "level up" your 3D graphics programming skills. This book will become your trustworthy companion in developing highly interactive 3D web applications with WebGL and JavaScript.
Tools
ct.js: 2D Game Editor
ct.js makes learning programming fun and game development easy by its visual tools, good docs and flexible, modular library. It is free, open-source, and is loved by hobbyists, professionals, teachers, and their students.