- Mar 23, 2021
- 127
- 111
This seems to have squashed the 'lost sorting' bug, hasn't recurred all weekend!
- Mar 15, 2020
- 1,843
- 1,801
Although F95Checker is fairly intuitive [kudos to WillyJL and his helpful associates, I agree it has evolved to be sufficiently complex to need SOMETHING.
As for myself, I don't use it except in the most basic manner to check all my games for updates. using it to launch my games would require rearranging my folder structure, which I am too lazy to do ATM.
Edit: Having said that, I will probably re-structure my folders anyway, just because I was already thinking about doing it.
I also don't use a lot of the other more advanced features mostly because I am unaware of them. I don't read this thread or the GitHub threads faithfully enough to be aware of all the advanced features which are currently the only places I know of that has any mention of them.
Last edited:
- Mar 7, 2019
- 1,318
- 1,121
starting to investigate this. picked 4 images from my images folder to test on. posting this as a reference for myself, to see what we can manage to achieve.
shows in order, file size on disk, texture byte size to send to gpu, and pixel resolution
(idk what happened with those floating brackets, powershell is tripping, they dont exist)
from what i found here
You must be registered to see the links
we might be able to achieve a 6:1 compression ratio with DXT1 (actually, maybe even more, that wiki mentions 6:1 from 24bit RGB to DXT1, but we're currently using RGBA which i assume is 32bit, so 32bit RGBA * 16pixel per chunk / 64bits per DXT1 chunk = 8:1 compression). but converting to that format might take time. will probably do something like, first time an image is loaded, if not correct format it gets converted then overwritten to disk. means all images will be .dds from now on, when this is all worked out. also, it mentions that DXT1 is an extension, not part of opengl spec, so might not work on all gpus?EDIT: yep, 800% compression ratio
thats the first image, the 18.2mb on gpu and 5.8mb on disk. crazy how dxt1 gets even better compression than the original png image (btw thats "race of life" banner)
now next problem is that i used this wand library, its basically image magick. ideal would be to have just dxt1 conversion in python without extra dependencies, but pillow doesnt do that from what i can see. could maybe include the magickwand .dll / .so
Last edited:
- Oct 1, 2020
- 1,303
- 885
Curious if DXT5 would do even better if sticking with RGBA. Don't waste time on DXT3, between the two of them DXT5 is the better choice anyways.
Although why RGBA..., yes it's 32bit but why waste 8bits that never get used? Transparency is just not a thing with image banners.
Less bits in is less bits out...
Last edited:
- Mar 7, 2019
- 1,318
- 1,121
thats what it was before, not what im making it do now. for our application here, alpha doesnt matter. but when i was implementing images with imgui and opengl long ago, it just did not like RGB without alpha for some reason. it would always end up with a few buggy images that were skewed or incorrect colors. so i just made it do RGBA.
- Mar 7, 2019
- 1,318
- 1,121
some preliminary vram results, with opening the tool to 12 collections gif banners:
background baseline usage before opening the tool: 1.6gb
no compression: 3.5gb (1.9gb used by the tool)
dxt1 with wand library: 2.1gb (0.5gb used by the tool)
dxt1 with opengl compression: 2.7gb (1.1gb used by the tool)
dxt1 with opengl compression and reapplying after compressing: 2.0gb (0.4gb used by the tool)
so yeah, 800% compression ratio between 1.9gb uncompressed and 0.4gb dxt1
what i gather from this is that dxt1 usage is same between these options, just with some caveats. opengl can compress in realtime, but seems like some of the uncompressed data is left behind. this is fixed by applying the texture uncompressed, letting opengl compress it, then reapplying the compressed texture. that gives 2.0gb, which id say is same usage as the wand library 2.1gb, probably just some background fluctuation on my pc in the meantime.
what im not super sure about is visual quality. dxt1 is lossy, i had not realized this when i started looking into this. it compresses in 4x4 pixel chunks, so after compression you can see some artifacts in some cases. its especially evident in gifs, where the different pixel chunks have slightly different color shades and are pretty visible. the original gifs already had some artifacts due to small color palette, but dxt1 makes it a bit more noticeable when zooming in. still tho, very much usable from what i can see. might make it only compress images above a certain resolution later on.
i cant really tell which would work best between opengl and wand, they both have some pros and drawbacks:
- opengl seems to cause more artifacts on smooth gradients, while wand seems to cause more artifacts around well defined shapes
- opengl compresses while applying the texture thus blocking the interface while compressing, wand is a separate step so interface is responsive during compression
- opengl is already here, wand would be a new dependency and a new dll to include and adds complexity
- opengl might compress differently between gpu vendors, wand would be same for everyone
overall the visual quality seems better with wand, and i think im leaning towards using that one. still tho, would be nice to get some feedback.
ive pushed build 1410 that includes some initial testing of these modes, you can switch between them at the bottom of the sidebar settings.
i am still trying to figure out how to bundle image magick, for now youll need to install it from
You must be registered to see the links
(any of the "dll" options) if you try this build.KEEP IN MIND: images will load slower with compression. for now, the files on disk are left untouched, so compression is done every time the image is loaded. the idea eventually is to compress once, then store on disk, which will make images loading much faster and occupy less disk space. but for now, i am looking for feedback on vram usage and on visual quality of compressed images. once that is figured out and compressed textures will start being saved to disk, we might even be able to make it unload images when offscreen, if loading textures will be fast enough for that. would mean constant vram usage for how many are on screen, not exponential with more games in library.
Dukez FaceCrap ascsd
- Oct 1, 2020
- 1,303
- 885
personally speaking, I wouldn't like the images getting compressed written to disk. Especially if the deterioration in quality is visible when zooming in (thinking of a future slideshow option), and even more so if the original image is already compressed with a lossy format like jpg.
And regarding your final comment, if that would become possible, then why even bother with the whole compression complexity and the need to install a 3rd party bundle. Since that alone would already drastically reduce vram usage, not?
On top of that, we're talking VRAM, not regular ram, if it's there, use it. It's not like you're going to be needing it for something else much other than the VN you're playing alongside. Don't understand the problem folks have with it getting used. I can understand not keeping things loaded if offscreen, but that's a different issue.
Let's have some examples where the use of VRAM causes issues for other software. (where not compressing to dds, coupled with offloading if not on-screen, would still be an issue)
Guess I'm glad F95 doesn't have these hangups about storage, otherwise we'd seen them already converting the lot to dds.
Last edited:
- Mar 7, 2019
- 1,318
- 1,121
The whole "unload images off screen" entirely depends on precisely the texture compression. Without it, images need to be decoded and converted for the gpu to understand. Writing compressed images to disk would mean no more conversion at all, just read from file and dump to gpu. Right now, it has to load the file, interpret it, convert it to RGBA if it's not, then make a huge pixel buffer for it, and dump that to gpu. Storing the converted RGBA pixel buffer to disk is also not an option, a 5mb image becomes a 18mb image on disk then, people already complain of how much space images take up, and I understand them, this is not doable. And even if, it would mean more data needs to be loaded from disk, which might even make slow hard drives even slower to load images as the bottleneck becomes disk io instead of cpu converting the images.
ive had issues myself with running out of vram, and I have 16gb of vram, with not even that many games. With the previews feature on the horizon, keeping EVERYTHING in vram is absolutely not possible at all. It may be fine if all are compressed since it takes 8 times less space, but even more so if this simplification in loading without conversion will mean it's fast enough to load each time the images become visible, then nothing would need to be in vram unless it's on screen. O( 1 ) instead of O( n ) space required then. Games libraries can keep growing, vram can't. And you're wrong, people can, already have, and will continue to be affected by all this vram being used. There's a whole background mode, what if you play a modern videogame? These games already struggle to work with todays gpus, if my tool is in the background hogging half the vram it's a lost battle, that game you want is never launching.
loading each time the image is shown and unloading when it goes off screen, without converting beforehand, is obviously going to cause issues. You already know how slow images can be to load. And some folks, me included, symlink their config colder to a spinning hard drive, which makes it even slower. Switching to animations tab I have takes for me close to 10 seconds to load all images since they're gifs. Now imagine that they always take that long to show up when you move around in the tool or scroll your list. That's a horrible user experience.
thats a disingenuous comment and you know it. We're talking about vram which is very limited, and about user's drive space, which both are already causing issues with how much this program hogs of them as banner images are stored in high resolution. Obviously f95zone server will have enough space to store the huge files, that's what they are designed for, and its browsers that show the images, which are state of the art in showing images efficiently. We don't have this luxury, we're dealing with the equivalent of game textures and need to convert images to such a format before being able to show them. Storing the RGBA pixels on disk would mean no more converting, but then a 5mb image takes 18mb on disk, and in slow hard drive cases would maybe take even longer to load as the bottleneck becomes disk I/O instead of cpu converting the textures, which again would make loading each time the image is show unfeasible. So the only solution here to speed up image loading and lessen disk usage and vram usage is to compress. Atleast for large images. As I said, it could be enabled only for images above a certain resolution, where the quality loss I can't even notice myself, and lower quality images remain as is, but that's for later. Now it would be useful if you tried it and gave some honest feedback about how it looks compressed and how much difference you get in vram usage, in stead of making disingenuous comparisons like a grumpy grandpa (I'm sorry but that's how you sound here, just skeptical of change in any way without even trying it from what I can tell. If that's not how you meant it I apologize). Nothing is set in stone yet, but this is the general direction that I will likely take. Getting some good feedback will help this not be such of an issue and become a net positive for everyone.
Last edited:
- Mar 7, 2019
- 1,318
- 1,121
im onto another lead. found out about ASTC texture compression, much newer standard and much more flexible.
with a 2000x2000 image:
RGBA is 15.3mb
DX1 is 1.9mb
ASTC 4x4 is 3.9mb
ASTC 5x5 is 2.5mb
ASTC 6x6 is 1.7mb
ASTC 8x8 is 0.98mb
ASTC 12x12 0.44mb
still lossy compression (there is no lossless compression with textures, needs to be linear when it reaches gpu and lossless compression is never linear), but a much newer format. ASTC 6x6 or 5x5 seems the best compromise, looks eons better than DXT1. astcenc has a quality parameter, 100 is very slow to compress but looks the best, 0 is instant but looks horrible, 80 is quite fast and looks the same as 100 to me.
for reference: astcenc-avx2.exe -cl in.png out.astc 6x6 80 -perceptual
then can view with "tacent view" which is a open source program for viewing textures (actually also uses imgui lol).
small problem: i cant seem to find how to load ASTC textures with opengl.
i think it just needs some parsing, cant find any good documentation for this but nothing some reverse engineering wont solve.
the real small problem: its a "new" format (2012), so some gpus might not support it. need to do more research on this.
Last edited:
in terms of img quality: I'm testing on maximized, 1440p screen, grid view w/ 1 column.
80% of the time I can't tell the difference unless I'm pixel peeping comparing uncompressed against compressed, and even then alot of them are not really noticable. That said, most of my banners seem to be HD, without much gradients, and no gifs..
If you are just scrolling through, you wouldn't notice that they are compressed.
As for the remaining 20%..
opengl seems to struggle with color banding on gradients the most, its quite noticeable. "Summer's Gone" is a good example.
Left to right: Original, Opengl, Wand
You don't have permission to view the spoiler content.
Log in or register now.
wand handles very dark gradients worse, but its generally less noticeable due to how dark the colors are.
wand also seems to handle text/edges worse at times, eg: "My best deal"
You don't have permission to view the spoiler content.
Log in or register now.
At times, both compressions caused the image to look noticeably blurrier (w/o pixel peeping/comparison) eg: "Lustyverse: passion on display"
You don't have permission to view the spoiler content.
Log in or register now.
Both do really bad with "A Man For All" & "Being A Dik"
You don't have permission to view the spoiler content.
Log in or register now.
You don't have permission to view the spoiler content.
Log in or register now.
Overall, i found myself using opengl to check which banners are problematic since its easier to tell, so thats saying something. Id say wand compression flies at the limit of whats acceptable, sometimes dipping below. Whereas opengl is often noticeably distracting in comparison.
Some cases are bad enough to warrant rethinking compression all together if another compression algo is not available..
So, I would be happy with wand as I'm impartial to slight compression artifacts, although people who care about their banners might complain... Although when its bad, even I'm like ok I have to actively ignore the quality of this banner.
tldr; 80% of the time compression is unnoticeable. when it is, wand is generally less noticeable than opengl. seems to be a viable solution. some cases are quite bad though, wouldn't hurt to check if better compression algos are supported.
Putting image quality aside, the UI hang with opengl makes it unusable
vram usage:
format, before, after loading all images
none, 0.45, 1.2gb (0.75gb)
opengl, 0.45, 0.6gb (0.15gb)
wand, 0.45, 0.6gb (0.15gb)
Last edited:
- Mar 7, 2019
- 1,318
- 1,121
yep, opengl is overall the worst one. i also tried dxt3 and dxt5, theyre slightly better with opengl than dxt1, but wand's dxt1 is still better than opengl dxt5. i havent managed to make wand create dxt3 and dxt5. if we go with dxt then it would have to be wand dxt1.
thanks for the detailed breakdown and previews, thats about the same conclusions as i had come to.
but looks like astc would be the best if its viable in terms of hardware support. ive managed to put it into the checker for testing, but havent gotten to supporting gif images yet (thats what im doing after posting this comment), heres a preview:
You don't have permission to view the spoiler content.
Log in or register now.
maybe im tripping, but to me ASTC here looks even better than uncompressed in some lines lmao
and it results in a smaller file than DXT1 too xD
EDIT: size comparison of a few still images (ASTC is 6x6):
You don't have permission to view the spoiler content.
Log in or register now.
EDIT: though, im seeing about the same vram usage as uncompressed with ASTC... i wonder if its because the gpu cannot draw ASTC and needs to decompress it in vram. if thats the case, astc would be viable only if it speeds up loading enough to the point where storing astc format images and unloading when off screen is viable. at that point yeah same vram usage but only for the images shown on screen, and also less disk usage
Last edited:
- Mar 7, 2019
- 1,318
- 1,121
ive pushed
You must be registered to see the links
which includes ASTC for png/jpeg. you will need astenc-avx2.exe in root of program folder to use it, can get it from here
You must be registered to see the links
. of course, just like wand, when this is all finalized and decided what to use, either astcenc or wand will be bundled with f95checker binaries.it also includes opengl dxt3 and dxt5 options which i was testing, but as i mentioned they looks worse than wand dxt1.
ill do some experiments with gif loading, and also see how it viable it is to unload when offscreen with astc saved to file
Last edited:
- Mar 23, 2021
- 127
- 111
- Mar 7, 2019
- 1,318
- 1,121
You selected DXT1 Wand compression, that one needs imagemagick installed. If you did this on accident and really do not want to install imagemagick you can edit the db settings and change wip_image_compress_mode to astc-6x6
EDIT: actually never mind, I realize that it still tries to import it, my bad. Just install it for now, can remove it later once compression is finalized as a feature (considering how well astc works I don't think dxt1 wand will be the final one)
- Mar 23, 2021
- 127
- 111
Just wanted to let you know. Now go enjoy your Christmas (unless of course you are already).
- Mar 7, 2019
- 1,318
- 1,121
merry christmas everyone!
indeed, after compressing with astc and saving the result to file, image loading is dramatically faster, to the point that unloading off-screen images is actually viable. and there is no perceptible visual quality loss at all.
from what i can tell, disk space is not that much better (i see 1gb before and 983mb astc compressed), and atleast on my system vram usage is about the same between uncompressed and astc, maybe it doesnt support astc and it's instead decompressing in some way. but regardless, the load speed increase is enough to the point that vram is solved by just unloading non-visible images.
- just astc compression: same vram usage as before, faster future load times after first shown and compressed
- just unload off-screen: constant lower vram usage,scrolling is a bit jittery, also very slow to load images
- astc compression and unload off-screen: constant lower vram usage,scrolling is a bit jittery (still need to investigate this), much faster loading
try these in
FaceCrap ascsd Dukez WhiteVanDaycare
whats missing for this is:
-fully unload even last shown images when minimized or in bg mode
- distribute astcenc with binaries for all platforms
-figure out whats causing the jitters while scrolling - this is the only real thing keeping me from enabling astc by default for all users (yeah also need to find out more about what hardware support is like, and disk space isnt that much better than i had hoped)
PS: i had to make my own .aastc file, .astc is only a single texture, but gifs needed multiple, so i made my own format which is loosely based on it but stored multiple frames. but this also means that enabling astc compression and replace means you will only have .aastc files in the f95checker images folder, and no program exists that can read these files aside from f95checker.
EDIT: i shouldve waited to type, i just fixed most of the scroll jitters. turns out, it was loading images so fast that they all applied during the same frame, which takes time to transfer data to gpu. fixed by transferring 1 image per frame. still incredibly fast loading, and no more stutters. also made it unload images when going into bg mode, if you enabled the option.
i have now mostly fully implemented astc compression, and dropped the dxt testing code. there are now 3 options: astc compression (in general enable compression and save results to file), astc replace (remove original images after compression), unload off-screen (unload images when not visible).
indeed, after compressing with astc and saving the result to file, image loading is dramatically faster, to the point that unloading off-screen images is actually viable. and there is no perceptible visual quality loss at all.
from what i can tell, disk space is not that much better (i see 1gb before and 983mb astc compressed), and atleast on my system vram usage is about the same between uncompressed and astc, maybe it doesnt support astc and it's instead decompressing in some way. but regardless, the load speed increase is enough to the point that vram is solved by just unloading non-visible images.
- just astc compression: same vram usage as before, faster future load times after first shown and compressed
- just unload off-screen: constant lower vram usage,
- astc compression and unload off-screen: constant lower vram usage,
try these in
You must be registered to see the links
and let me know how it goes!FaceCrap ascsd Dukez WhiteVanDaycare
whats missing for this is:
-
- distribute astcenc with binaries for all platforms
-
PS: i had to make my own .aastc file, .astc is only a single texture, but gifs needed multiple, so i made my own format which is loosely based on it but stored multiple frames. but this also means that enabling astc compression and replace means you will only have .aastc files in the f95checker images folder, and no program exists that can read these files aside from f95checker.
EDIT: i shouldve waited to type, i just fixed most of the scroll jitters. turns out, it was loading images so fast that they all applied during the same frame, which takes time to transfer data to gpu. fixed by transferring 1 image per frame. still incredibly fast loading, and no more stutters. also made it unload images when going into bg mode, if you enabled the option.
Last edited:
- Oct 1, 2020
- 1,303
- 885
Ok, sorry if I sounded like a grumpy grandad...
2 things,
1. I can't test either build since I don't have a GPU that utilizes these features. It's still an old Radeon so your efforts are wasted on me. Most modern games play like a slideshow for me if I try.
2. As for the disk space... I'm sorry but I have to suppress a little chuckle here. I've got almost a terabyte in 4K comics stored on one of my HD's, so the half a gigabyte used by the checker is a joke to me.
And that part was what my comment about F95's storage was about.
They may have servers, but if they're using a hosting service, diskspace comes with costs...
Also, another thing, if you're going to play a game that is highly dependent on every last bit of VRAM you can muster, you'd be stupid to leave anything running that competes with that...
Having said that, I ain't opposed to anything you do to lower vram usage, although I am not a fan of anything that reduces the image quality to the point it's noticeable. As for loading times, well, that's perhaps the result of the way I use the checker. I just close it down whenever I want to run something that requires as much system resources as it can get/use, same for anything else I would have running in the background and don't need to have running while playing. For taxing software that's just common sense.
When I'm not running such things, I could care less how fast or slow it loads images, I'm not looking at them constantly, neither do I spent more time in the checker than needed to pick a game to play or do the occasional manual refresh. And in normal use it's fast enough for me, even on the crappy box I have.
I'm sorry but that's just how it is for me. I don't have a fast system and certainly not a GPU you could brag about, so me testing things that are targeted to utilize modern GPU features is me wasting time.
Between running VSCode when I'm tinkering with the code and running the Checker source there's not much left to run anything else smoothly.
Anyways, to you and everyone else.
a Merry Xmas/Happy Hanuka or whatever else you celebrate!
EDIT: And to satisfy your curiosity I still tried to run it, but I get "Invalid image" across the board. (Didn't expect anything else, like I said, pre-historic GPU). So, if you're going to implement this, I hope it's optional.
EDIT2: As for DXT. 1410 just silently dies on me without any info in the console if I try OpenGL, it also is slow as molasses while it lasts, which isn't long. It dies before every image on a tab is loaded.
With Wand I don't notice a difference... images load just as slow as with no compression... and also see no difference in quality so I don't think it is
Last edited:
- Mar 11, 2017
- 63
- 35
Now I'm not smart enough about all the technical things at all, but after reading this I got this idea I wanted to throw out there because if it works like I would think (which doesn't say much). couldn't you compress/convert the images to a more or less lossless quality and store those instead of the original and have that "just read from file and dump to gpu." so that way you would make the whole "unload images from screen" thing work much faster without losing any noticible visual quality of the images?
As I said, was just an idea that came to mind, might be stupid
And Happy Holidays everyone
- Mar 7, 2019
- 1,318
- 1,121
I mentioned this in another comment, lossless compression doesn't exist on gpus. They're designed for textures and games and things like that, and textures need to be directly addressable. This means, if you only want to draw a part of the texture, you can go directly to that part of the buffer and draw it even if the whole texture is much larger. Lossless compression is not directly addressable, how it works is usually by Huffman codes and other similar techniques, which basically store a table of symbols at the beginning, then the rest of the content is built just by referencing one of the symbols at the beginning, like puzzle pieces that can be used repeatedly. Take "aaabbbbbbaaabbbb", a lossless compression may call 1:aaa 2:bb, then store the content as 1222122. What it means is that all of the content buffer is meant to be loaded sequentially, symbols have different sizes so you can't just "jump" to a specific pixel inside a lossless compressed image, you have no idea where that pixel info is located in the buffer until you decompress it. And decompressing each time you need to draw something is extremely expensive.
So, no lossless compression. If you mean converting to any format the gpu understands and storing that to file, then also not viable, as RGBA uncompressed is what it's been using so far and for example a 2000x2000 image that takes 300kb on disk is 15mb of RGBA, with DXT1 and ASTC 6x6 it's under 2mb.
How compression in gpu textures works is in blocks, you pick a block size of for example 6x6 pixels, and compress each of those independently. Now if you want to jump to a specific part of the texture and just draw that part you simply calculate in which block the pixel is located, which would roughly be a simple x/6 and y/6.
So yeah, all gpu textures compression is lossy, but some are better than others. DXT1 is a very old format and looks kinda bad, though it is the most efficient and fastest from what I can see. ASTC is a newer format from 2012 and it has some genius algorithms to calculate pixel weights and perceptual quality over what would be pure compression math accuracy. I personally cannot tell the difference between uncompressed and ASTC 6x6 80% quality. And that's what I've implemented finally now in build 1433
- Mar 7, 2019
- 1,318
- 1,121
Good thing it's an optional toggle then
Yeah I get what you're saying there, I think that people that complain about it is because of 2 things, 1 they don't see that much importance in banner image quality so they don't see why they would take "so much" space, which is actually a reasonable amount but much more than what they expected, and 2 because maybe they have a small system drive and a large hard drive, so storage in appdata is limited. Of course, for the latter we both know about symlinks to move those to the hard drive too, but most don't, so I can see how it would seem like a lot.
I agree, but also that's a mindset that many don't have, the average windows user and the average gamer just leave whatever in the background. They probably have Steam, epic, origin, zoom, Skype, iCloud, and god knows what else, always in the system tray and leave them open when doing whatever, including games, and those don't really use up much vram. This tool would likely end up with the same fate in the system of one such user, so an option to unload from vram is good imo. Then again, users like this are precisely the ones that would never know or look to enable such a setting, but can't enable it by default either as hardware support isn't there yet.
DXT1 was visibly worse then original even without zooming in, but with ASTC I'm zooming in to the actual pixels and I still cannot notice any compression artifacts or visual quality loss. But of course, it's optional.
Did you add the astcenc-avx2.exe to the program folder? Also, it might just be that, your cpu maybe doesn't have avx2 so maybe just the compressor fails to run but the gpu would understand ASTC just fine. Check if there's any .aastc files in the images folder, if so it means compression went fine and it's the gpu that can't show it. If there isn't any, please try to run the astcenc executable by itself and see what it says (astcenc-avx2.exe -cl in.png out.astc 6x6 80 -silent), it will be useful for proper error handling