Multi-pass shaders and persistent shader buffers
  • jasonbeyers
    super8 member
    Posts: 125
    Joined: Mon Oct 08, 2018 7:16 am

    Multi-pass shaders and persistent shader buffers

    by jasonbeyers » Sun Oct 11, 2020 5:10 am

    After 3 years of working with Madmapper, I continue to find cool new ways to do things with the tool. However, there is one broad category of media that has never been playable in Madmapper: shaders with multiple passes or persistent shader buffers.

    Madmapper is really good with generative content, thanks to the unique Material shader type (which provides smooth changes in speed) and the auto-generated sliders for shader inputs - including for ISFs. Madmapper's audio controls are also very impressive, and these complex audio controls are only possible when Madmapper is running the generative content.

    There is a whole world of multi-pass shaders on shadertoy that is untapped here. Things like reaction diffusion, motion detection, and trail effects are out of reach, requiring external tools with different control mechanisms. The MIDI-out and OSC-out modules help bridge this gap, to leverage MM's awesome cue system, but it would be better if MM could play this kind of content natively.

    With that said, *some* shaders with shader buffers miraculously do work within MM, but it is rare. I've sampled a few hundred of these kinds of shaders in MM and maybe 10% work as expected. I've also noticed that these buffered shaders (in MM) introduce GPU load even when they are applied to surfaces that are disabled (unlike regular shaders), and content can "bleed" between shaders if multiple ISFs use a buffer of the same name.

    And aside from buffers: shaders with multiple passes may display an output in MM today, but typically only from the first pass. And typically filter/FX shaders are very sensitive to the type of texture input -- syphon input sources don't seem to work, even for that first functional pass, within MM.

    Yes, it is possible to split an existing multi-pass shaders into multiple single-pass ones, and pass textures between them using syphon/spout. I've had some limited success with this, but I have not found such a workaround for persistent shader buffers.

    Some of the most impressive multi-pass or buffered shaders happen to be FX/filter types. Imagine the possibilities if we had support for these kinds of shaders, combined with custom FX support (https://forum.garagecube.com/viewtopic.php?f=29&t=35263). The possibilities are endless!

    Also, I recall that MM does not support these shaders due to the performance implications (ex. GPU) with many surfaces. All visual software has these considerations (ex. Touchdesigner, Resolume, etc), and it would be better if we at least had the option for these performance-heavy graphics. People could then determine what is too much for their projects & hardware before FPS drops too much.

    Additional notes:

    When using an eGPU on Mac, syphon doesn't always use the eGPU even if Madmapper does. This introduces an unexpected bottleneck when the internal GPU is underpowered. Multi-pass shaders would eliminate the need for syphon in some cases -- either because the external software (like TouchDesigner) OR internal syphon routing within MM would not be needed for those scenarios anymore. So this is an added opportunity for folks running on Mac with an eGPU.

    For reference, I've been using Magic Music Visuals and Touchdesigner for these kinds of shaders, where Magic is far simpler to use. But I sorely miss Madmapper's audio controls with attack-decay-release filters.

    Thanks for your consideration!
  • franz
    madMapper master
    Posts: 1032
    Joined: Fri Feb 18, 2005 7:05 pm
    Location: Paris, France
    Contact:

    Re: Multi-pass shaders and persistent shader buffers

    by franz » Wed Oct 14, 2020 2:44 pm

    +1
    I've always wanted multi pass materials, as you said, for these reaction diffusion processes..
  • looseopinions
    member
    Posts: 10
    Joined: Tue Oct 19, 2021 1:57 am

    Re: Multi-pass shaders and persistent shader buffers

    by looseopinions » Tue Mar 29, 2022 9:01 pm

    Do the new features of the 5.1 beta now make this possible?
    >IMPROVEMENTS
    * Materials can use last frame data for more complex creations

    If so, are there any examples of using last frame data to create Surface FX such as feedback delay?
  • franz
    madMapper master
    Posts: 1032
    Joined: Fri Feb 18, 2005 7:05 pm
    Location: Paris, France
    Contact:

    Re: Multi-pass shaders and persistent shader buffers

    by franz » Wed Mar 30, 2022 4:16 pm

    MadMapper as of version 5.1 now has access to the last frame rendered by a material - is set to render to texture -
    You'd have to add this to the JSON header:

    "RASTERISATION_SETTINGS": { "DEFAULT_RENDER_TO_TEXTURE": true, "DEFAULT_WIDTH": 512, "DEFAULT_HEIGHT": 512, "DEFAULT_PIXEL_FORMAT": "PF_U8_BGRA", "REQUIRES_LAST_FRAME": true
    },

    then within the shader you can access the last frame with this:
    IMG_NORM_PIXEL(mm_LastFrame,uv)

    where mm_LastFrame is the sampler2D with the last frame.

    Regarding multipass materials, we'd rather design some "Media FX", which would be stackable shader fx that you'd put on top of materials (or any media actually). That solution seems smoother and more versatile, as it would work with video files as well.
  • franz
    madMapper master
    Posts: 1032
    Joined: Fri Feb 18, 2005 7:05 pm
    Location: Paris, France
    Contact:

    Re: Multi-pass shaders and persistent shader buffers

    by franz » Wed Mar 30, 2022 4:34 pm

    "Feedback Grid" is an example material that shows how to use feedback
  • looseopinions
    member
    Posts: 10
    Joined: Tue Oct 19, 2021 1:57 am

    Re: Multi-pass shaders and persistent shader buffers

    by looseopinions » Fri Apr 01, 2022 7:30 am

    When I refer to Feedback Grid and try to do something like this, my mm_LastFrame returns a black and white checkerboard instead of the actual previous frame. So I tried copy/pasting the exact code from Feedback Grid into my own material and it returns a (red and black) checkerboard instead of the previous frame.

    What might I be doing wrong?


    also..
    Regarding multipass materials, we'd rather design some "Media FX", which would be stackable shader fx that you'd put on top of materials (or any media actually). That solution seems smoother and more versatile, as it would work with video files as well.
    That's what I really want to do. Thanks!
  • franz
    madMapper master
    Posts: 1032
    Joined: Fri Feb 18, 2005 7:05 pm
    Location: Paris, France
    Contact:

    Re: Multi-pass shaders and persistent shader buffers

    by franz » Fri Apr 01, 2022 11:46 am

    are you running 5.1 beta 2 ?
  • looseopinions
    member
    Posts: 10
    Joined: Tue Oct 19, 2021 1:57 am

    Re: Multi-pass shaders and persistent shader buffers

    by looseopinions » Sat Apr 02, 2022 12:07 am

    Yes. I just relaunched 5.1 beta 2 to double check, and the problem was gone! The copy/paste of Feedback Grid code from yesterday was working fine.

    So I repeated the process: Copy the code from Feedback Grid, create a new material, paste the code, and the problem is back (no feedback, instead I get a black/red grid). Save the project, reopen the project, my new material is still broken. Create a new project, go to Library > Mine, drag my new material to the output window... NOW it works.

    Is this a bug that last frame code doesn't work until you close the software and reload the material from the library?

Who is online

Users browsing this forum: No registered users and 6 guests