rendervulkan: Add Anime4K 2x CNN UL upscaling filter

[Et3rnos]

Overview

Added Anime4K 2x CNN UL (Ultra-Large) as a new upscaling filter option for gamescope.

• New CLI option: --filter anime4k
• 2x neural network upscaler optimized for anime/cartoon content, like visual novels and 2D RPG games

Performance

Benchmarked on RTX 3050 Ti (Laptop) against Lossless Scaling's Anime4K UL implementation:

Input Resolution	Gamescope	Lossless Scaling	Improvement
1280×720	6ms (~166 fps)	9ms (~111 fps)	50% faster
800×600	4ms (~250 fps)	Not tested	Not tested

Estimated VRAM usage:

Input Resolution	VRAM
1280×720	~191 MB
800×600	~100 MB

A consistent 50% performance gain achieved by merging 25 original shader passes into 9, reducing redundant texture fetches by writing to 3 output textures per dispatch.

Visual Quality

Output compared against Lossless Scaling (a perfect match). It is correctly implemented, and the quality matches the original Anime4K shader.

Comparison with FSR

The source game was run at 544x416, upscaled to 1920x1080:

FSR:

---

Anime4K:

Related Threads / Motivation

Those threads mentioned interest on Anime4K being added to Gamescope.

#262
#788

Notes

• This is only an implementation of the Ultra Large (UL) shader variant.
• This is a Vulkan compute shader implementation of the original Anime4K shaders (https://github.com/bloc97/Anime4K - MIT licensed)

[malucard]

To run this without crashing, I had to change 2 to 3 in rendervulkan.cpp:915

[Et3rnos]

To run this without crashing, I had to change 2 to 3 in rendervulkan.cpp:915

Thanks for giving it a try, I've now updated that line.

[JerwuQu]

Great work! I gave a shot at implementing this myself but didn't get far, so I'm really happy for this.

While a 2X scaling works well for a lot of users, I think having the possibility to have another up/downscaler after the 2X would be good for scaling 720p->1080p (2x + down) or 720->4k (2x + up) as it currently seems to get cropped otherwise. Tested again and it worked.

Also, my GPU usage jumped up from ~5% to ~75% by activating the scaling at 1280x720@60fps input. It would be nice to have some lighter models available so that it works on more constrained hardware or for more demanding games.

EDIT: There's also some flicker/jitter going on when scaling is active despite seemingly stable FPS. Does not happen with FSR.

Sidenote for others: I had to use --backend sdl or the game would be completely white when upscaling, though that's likely unrelated to this as FSR requires the same thing for me.

[Et3rnos]

Replying to #2072 (comment)

Thanks for taking a look!

Regarding your first point, it is supposed to handle those cases very well. It might be upscaling to twice the dimensions but after the anime4k passes, there is one last BLIT pass that resizes to the output dimensions (and also preserves the aspect ratio of the original frames). So I'm not exactly sure how to replicate your findings. I'm on a laptop with CachyOS (KDE) and a NVIDIA GPU and everything works perfectly fine for me.

For your second point, I do agree that porting at least the VL or the L models might be a good idea, I ported just the UL one since originally this was supposed to be something for personal use, but I totally see the benefit in supporting the more lightweight models for lower-end devices. I can't promise anything but I'll give it a try when I have some extra free time.

As for your edit and backend issues I'm afraid I never came across them... but if you find the causes for them please let me know!

[JerwuQu]

The resizing was my bad! I tested with some other applications and it behaves like it should. It was an issue of the nested application being windowed rather than fullscreen and simply going out of bounds. Nothing wrong with the upscaler.

Nice to hear about ideas for more ports. For the implementation I was working on I had the goal of converting existing CNN upscalers (mainly from Magpie) to Vulkan shaders, with a sidecar file that specified inputs/outputs for each pass. Kinda overkill for an Anime4K specific implementation, but I thought it may be relevant for future upscalers of similar multi-pass design. Any work here is appreciated on my end though, and I can help test!

Will test in more games to see if I can find a source of the stuttering.
Am running on latest mesa/amdgpu AMD drivers.