Tool Others F95Checker [WillyJL]

5.00 star(s) 24 Votes

WillyJL

Veni, vidi, vici
Donor
Respected User
Mar 7, 2019
1,408
1,205
talked in dm but ill reply here too and expand too
Yeah, crappy box... "ERROR: Host does not support AVX2 ISA extension"
same error for astcenc-sse4.1.exe
great to know, ill match for such an error and make it use an alternative executable. ill bundle the avx2 for best performance, and sse2 for best compatibility (astcenc mentions sse2 version will work on all x84_64 machines).

but...
astcenc-sse2.exe ran fine with the same cmdline, and produced an .astc file, although I've got nothing on my system that will allow me to view it.
can show .astc textures. the checker cannot, as it uses .astc and astcenc only as an intermediary, the final files it uses are .aastc which is my own format loosely based on .astc but with multiple frames to support animated images.

1735133318434.png
1735133318434.png

Left PNG, Right ASTC. what surprised me though is when I changed 80 to 100 (quality setting?) resulting file size was the same, but it took ages longer to compress, setting it 98 gave about the same speed (maybe slightly longer but not noticeable)
yeah, being a block based compression, the size of the output depends only on size of blocks and how many blocks there are (which depends on image resolution). quality is just how much it tries to make each block look good after compression. ive found very diminishing results between 80 and 100, cant tell them apart. also, not entirely sure how the numbers work, it may be that the only accepted numbers are those of the presets and thus -thorough which is 98 quality is used even if specifying 80, ill have to check.

Another thing, it don't like lossles webp (what I convert everything to, including my comics... think what it would amount to if I didn't... roughly ~35% more space)
yeah astcenc just does png, jpeg and things like that. the checker will convert to png intermediary if astcenc cant understand it. so for example in your case it would be webp > png > astc > aastc, lmfao. but yeah same for gifs too, just many png and astc intermediates one for each frame, and one single aastc final output.
 

WillyJL

Veni, vidi, vici
Donor
Respected User
Mar 7, 2019
1,408
1,205
converting to astc spawned a terminal for every image, probably a windows thing that can be sorted out.
nice catch, i always run from vscode terminal so i dont notice these things.
EDIT: forgot about this while typing the reply below, fixed that problem below but not this one xD, this is up next.

whats alil weird about load times is if I scroll at a medium speed, it loads everything instantly.
whereas if I scroll fast, it takes a second to catch up to loading the newest images in view. looks like a bottleneck somewhere, maybe its not cancelling the render calls for images that are now out of view, or its set as FIFO?. but anyways that's a low priority optimisation given how fast things are already!
youre right, i see this too. and youre about right.
each frame when an image is shown that is not loaded, it gets put to the top of stack of images (LIFO) to load (thats why it loads from bottom right, because when theyre all shown at same time, its drawn top left to bottom right, so bottom right is the last image drawn, which goes to top of stack as first one to load).
then, images in the stack are processed one at a time. means that if a new image is added to top of stack, if theres one currently still loading, it has to wait for that one to finish.
for how its currently architected, cancelling loading an image is kinda complex, im not sure i trust myself to do it properly without race conditions.
anyway, after an image is loaded, it gets added to a queue to be applied (sending the loaded texture data to gpu). this currently is FIFO, but thats not the whole problem.
i had it limited to 1 texture apply per frame, otherwise it would apply all images in one frame and stutter really badly. so, its FIFO, and "ratelimited", so scrolling by super fast would load all the ones you go by instantly due to being ASTC, then they all get queued, and they take time to apply the ones off screen, just to unload them right after as they are offscreen.
the main issue i just fixed by removing images off-screen from the apply queue, as i mentioned i dont trust myself to interrupt the load process (which includes conversion if applicable), but removing the image from apply queue is simple and effective, and fixes the issue basically entirely.
as for without unloading offscreen images, FIFO still seems to work better for applying... logic tells me otherwise, but yeah... with LIFO applying, it takes a while tostart applying the images currently shown on screen, with FIFO theyre loaded immediately when shown (talking about ASTC without unload, without ASTC it would load slower but same effect).
anyway this issue is resolved in build 1437 :D

EDIT: alil rabbithole diving into astc, seems adoption is slow due to S3TC & BC7 being "good enough" compared to astc, albeit astc being slightly better. potential fallbacks if astc is not supported?
S3TC is the specification for DXT1/3/5, and i think BC7 is part of BPTC which i also tried. they all look like shit, so idk what these clowns are on about xD

looks like astc is the perfect candidate.
for hardware that doesnt support it, you can have a fallback to opengl at the cost of quality for people who prioritize lower vram use
i was moreso thinking of just letting people disable compression altogether. but i have one last hope that i need to check. if we are incredibly lucky, maybe opengl can convert ASTC to RGBA before uploading to gpu, if its not supported. if thats the case, we could benefit from disk space savings and *maybe* load speed too, at the cost of VRAM, but that is again offset by unloading off-screen images.
FaceCrap 's gpu seems to not have ASTC, but the error he encountered thus far was due to the encoder, not the gpu loading the texture. we'll see what his gpu says when presented with valid ASTC then decide what to do. if it cant decompress ASTC to RGBA live then yeah as you said, loading RGBA and letting opengl compress it to DXT1 would be a good enough alternative to allow lower VRAM (but of course wouldnt work well with unload off-screen images). also if we end up doing that, the ui freezing like before wouldnt happen anymore (atleast not to that degree) as only one image is applied per frame, so only 1 would need to be compressed by opengl per frame.
 

WillyJL

Veni, vidi, vici
Donor
Respected User
Mar 7, 2019
1,408
1,205
can u add login thing? when i delete this program i dont want to lose my game list
so you just casually come here out of nowhere with a "can u add login thing?" like its a simple thing.
yeah i will just casually build an account system, server infrastructure for it, syncing between clients, reinvent the wheel too while im at it.
no. just no. make a copy of your application data. or use the export links option.
 
  • Haha
Reactions: WhiteVanDaycare

FaceCrap

Ghost of torrents passed
Donor
Oct 1, 2020
1,523
1,050
can u add login thing? when i delete this program i dont want to lose my game list
I fail to see what benefit you'd get from login. besides, there is a login option, but it's just to access the forum from the Checker. Has zero influence on your data.

Your data is stored elsewhere ( )
So, delete the app whenever to reinstall it later, your data is safe.

Oh WillyJL quality -medium equals 60, so 80 would be ok, it doesn't get transposed to -thorough... which btw is noticably slower if you do batch conversion.

Another thing, I just found out the forum sometimes messes with you, I downloaded the banner image for threads which have an animated cover image. according to the css these were all jpg... but some are actually png... even found a gif posing as jpeg. And the converter don't check magick numbers so complains about invalid image formats. That might be something to be aware of.
 
Last edited:
  • Like
Reactions: WillyJL

simple_human

Newbie
Dec 2, 2018
86
3,712
Well, ever since the sorting issue was fixed, the program hasn’t crashed even once so far. I couldn’t even reproduce the crash the way I demonstrated in the video. So yeah, it seems the crashes were indeed related to that.
 

WillyJL

Veni, vidi, vici
Donor
Respected User
Mar 7, 2019
1,408
1,205
As far as I'm concerned the ASTC implementation is basically done. Now it bundles all the appropriate astcenc executables with binary releases, looks for cpu feature flags to decide which one to use, and looks in PATH when running from source. Also fixed the cmd popping up during compression. And finally, error handling should be a lot better, it will now tell you what went wrong like astcenc failing or pillow not recognizing the image or astcenc missing.

Seems to work fine on windows and Linux for me, can't test on macOS. It's build 1448.

Now for the interesting part, let's see if for FaceCrap it will correctly detect that cpu features are missing and so use the sse2 version, and most importantly if after compression it will manage to load the textures (and with what error if not so it can be accounted for)

Also ascsd Dukez WhiteVanDaycare give if a shot if you can :D
 
Last edited:
  • Heart
Reactions: ascsd

ascsd

Member
Jul 26, 2021
100
75
As far as I'm concerned the ASTC implementation is basically done. Now it bundles all the appropriate astcenc executables with binary releases, looks for cpu feature flags to decide which one to use, and looks in PATH when running from source. Also fixed the cmd popping up during compression. And finally, error handling should be a lot better, it will now tell you what went wrong like astcenc failing or pillow not recognizing the image or astcenc missing.

Seems to work fine on windows and Linux for me, can't test on macOS.

Now for the interesting part, let's see if for FaceCrap it will correctly detect that cpu features are missing and so use the sse2 version, and most importantly if after compression it will manage to load the textures (and with what error if not so it can be accounted for)

Also ascsd Dukez WhiteVanDaycare give if a shot if you can :D
just as im downloading i see Oops + Build show up :ROFLMAO:
 

WillyJL

Veni, vidi, vici
Donor
Respected User
Mar 7, 2019
1,408
1,205
just as im downloading i see Oops + Build show up :ROFLMAO:
Yeah I was up too late and couldn't for the life of me get the very basic download working for macOS, did it slightly wrong in every possible way the forgot to check if the last change worked xD
That's the only change tho
 
  • Haha
Reactions: ascsd

ascsd

Member
Jul 26, 2021
100
75
everything was running perfectly. speed is great.

I was in the 3rd view mode (forget its name, 3 vertical lines, "No Labels" at top) was dragging the scroll bar up and down alot of times, pausing in between just checking the speed of the image loading. And it just crashed, no log or anything.
Ran debug, kept scrolling up and down wildly, with random pauses, took a few minutes of this to crash again, no log.
Ran debug from terminal, couldnt get it to crash again, been trying for more than 5 mins..

EDIT: also debug shows:
"Unable to import OpenGL.arrays.numpymodule.NumpyHandler: No numpy module present: No module named 'numpy'"

otherwise, with normal use it seems to be great
 
Last edited:

ascsd

Member
Jul 26, 2021
100
75
i had it limited to 1 texture apply per frame,
is it viable to load more than 1 per frame eg 3? or not a good tradeoff?


S3TC is the specification for DXT1/3/5, and i think BC7 is part of BPTC which i also tried. they all look like shit, so idk what these clowns are on about xD
one game dev was pretty annoyed and passionate regarding the lack of support for astc, didnt realize these compression algos were the same as you mentioned before. now I share their frustration. nvidia atleast seems to have adopted it recently so thats good news for everyone
 
  • Like
Reactions: WillyJL

WillyJL

Veni, vidi, vici
Donor
Respected User
Mar 7, 2019
1,408
1,205
everything was running perfectly. speed is great.

I was in the 3rd view mode (forget its name, 3 vertical lines, "No Labels" at top) was dragging the scroll bar up and down alot of times, pausing in between just checking the speed of the image loading. And it just crashed, no log or anything.
Ran debug, kept scrolling up and down wildly, with random pauses, took a few minutes of this to crash again, no log.
Ran debug from terminal, couldnt get it to crash again, been trying for more than 5 mins..

EDIT: also debug shows:
"Unable to import OpenGL.arrays.numpymodule.NumpyHandler: No numpy module present: No module named 'numpy'"

otherwise, with normal use it seems to be great
the numpy thingy is normal, weird thing with how opengl is installed, doesnt affect anything.
i just got one crash myself, but not sure if its the same. i was adding a new game, and got RuntimeError: Set changed size during iteration
thats because during draw it iterates all image instances to check which ones need to be unloaded, and the new game and thus new then image was added at the same time. thats an easy enough fix.
were you adding/removing games when it crashed?
 

ascsd

Member
Jul 26, 2021
100
75
the numpy thingy is normal, weird thing with how opengl is installed, doesnt affect anything.
i just got one crash myself, but not sure if its the same. i was adding a new game, and got RuntimeError: Set changed size during iteration
thats because during draw it iterates all image instances to check which ones need to be unloaded, and the new game and thus new then image was added at the same time. thats an easy enough fix.
were you adding/removing games when it crashed?
Nope, just scrolling wildly in the 3rd view mode

Just tried adding a new game now while in list view. no issue
 

ascsd

Member
Jul 26, 2021
100
75
yeah its a race condition, i got it only once so far, but it is possible. as for what you had, still havent replicated it
yea it wasnt easy to replicate, almost gave up before the second time happened. havent been able to replicate again either..
i was wondering if it was also a race condition that caused it. maybe a load and unload being called at the same time. I did notice one thing, it started to become a touch stuttery/sluggish just before it crashed but not sure if thats from the issue that caused the crash, or a symptom of it just about to crash

think it safe to ignore until it happens again with normal use
 

FaceCrap

Ghost of torrents passed
Donor
Oct 1, 2020
1,523
1,050
Related to hunting an issue in the regular version.

Try resetting the banner on a game that uses an animated gif (e.g. Motherless) then do a full recheck on just that game.

As for trying the build. Have to wait until I'm home.
 

WillyJL

Veni, vidi, vici
Donor
Respected User
Mar 7, 2019
1,408
1,205
Try resetting the banner on a game that uses an animated gif (e.g. Motherless) then do a full recheck on just that game.
does it keep saying image is missing? because i had removed the image.resolve() from game.refresh_image() as i thought just image.loaded = False would be enough, since when drawing the image it checks for .loaded and then does .resolve() internally when loading, but doing it that way the .missing and .invalid remained set even if the image is fine.
i fixed it in feat/tex-compress branch in (specifically the missing and invalid attributes being properties which also check .loaded, if not loaded it will fake being valid so it will do .reload() when shown and trigger proper file lookup and checking/conversion/loading). doing it this way means its also lazyloaded, previously at startup it would resolve the path on disk for all game images, now it does it lazily when they are shown so it also starts up significantly faster.
with how different the image loader is on that branch, i wont look at bug reports for it on main for now until its merged, if its a bug on that branch too then ill take a look at it
 

FaceCrap

Ghost of torrents passed
Donor
Oct 1, 2020
1,523
1,050
does it keep saying image is missing?
YES!!!!
The latest regular build gotten through update works, but has as issue that it becomes unresponsive, and CPU goes mile high, this settles down after a couple of minutes and it becomes responsive again, but the latest source from main does exactly that... keeps saying image is missing.

i had removed the image.resolve() from game.refresh_image()
Saves me a lot of trial and error, fixed.

EDIT: Doesn't occur in the feat/tex-compress branch.
 
Last edited:

FaceCrap

Ghost of torrents passed
Donor
Oct 1, 2020
1,523
1,050
WillyJL
First, I made a filter on Score without changing the value, this has as result that if you filter all tabs, you get all your games in a single list.
At least this allowed me to check which images it had converted. (see *ISSUE)

I only checked ASTC compression and left Unload off-screen clear (found out that wasn't the smartest thing to...).

1st attempt it encoded only 356 of the 534 games. Since 313 of those were png converted to webp, I converted those back to png to see if that would change things, but while comparing image names, I saw it had done webp's too before I reconverted those back to png's.

2nd attempt it encoded only 135 images, and no matter what I did, it didn't do anymore. But while creating this post I checked back and somehow it actually had come alive again as the count had risen. But stopped again at 319... (see *NOTE)

Now as I said, I found an easy way to check which it had done. The bad news is none of the encoded image's show. All I get to see is a black area. And it's that black area which revealed which it had encoded.

I discovered that if I opened the More Info popup and scrolled through the list, for every image it had encoded, the black area had the aspect ratio of the original image. For those images that still weren't encoded, the black area was a square (verified by checking ID's against encoded images)..

#NOTE: Then I noticed that whenever I scrolled through images showing a black square, the encoded count went up again. Contrary to this, when I switched to grid view (middle button) scrolling through the list had no effect on the encoded count.

*ISSUE: Doing so, I also ran into an issue. At first, I used the mouse to click the right-arrow button. This resulted in an annoying flashing because the popup gets closed on the current game and recreated/opened on the next which also causes the game list getting undimmed and dimmed again. Surprisingly when I switched to using the keyboard right-arrow (or left-arrow) keys, the popup stayed open, and the flashing was gone.


So, after 45 minutes of scrolling and waiting till it had done all, the total count became 532 out of 534 entries. Turns out I got two tools in my list that don't have a banner.

Conclusion. apart from uncovering a bug, ASTC is not something I can use. Encoding works, as I said, but displaying the result is a total failure. (*Tacent however does show them, so it does something other than directly dumping them to the GPU)

Final note... If I turn off ASTC Compression, it does not load the originals, not even after removing all .aastc files. Here yet another suprise surfaced... it actually did load some and it turned out these were all animated gif's. All the rest just stays a solid black area.
Ah, Now this is a situation where you DO need to have Unload off-screen checked. Doing so immediately made it reload the original images.

EDIT: Almost forgot the most important thing.
No errors showed, neither in the Checker nor in the debug console
 
Last edited:

WillyJL

Veni, vidi, vici
Donor
Respected User
Mar 7, 2019
1,408
1,205
1st attempt it encoded only 356 of the 534 games. Since 313 of those were png converted to webp, I converted those back to png to see if that would change things, but while comparing image names, I saw it had done webp's too before I reconverted those back to png's.

2nd attempt it encoded only 135 images, and no matter what I did, it didn't do anymore. But while creating this post I checked back and somehow it actually had come alive again as the count had risen. But stopped again at 319... (see *NOTE)

Now as I said, I found an easy way to check which it had done. The bad news is none of the encoded image's show. All I get to see is a black area. And it's that black area which revealed which it had encoded.

I discovered that if I opened the More Info popup and scrolled through the list, for every image it had encoded, the black area had the aspect ratio of the original image. For those images that still weren't encoded, the black area was a square (verified by checking ID's against encoded images)..

#NOTE: Then I noticed that whenever I scrolled through images showing a black square, the encoded count went up again. Contrary to this, when I switched to grid view (middle button) scrolling through the list had no effect on the encoded count.
this whole things sounds like trying to link your actions to some meaning where there isnt. whats most likely happening when it appears to be stuck is it is compressing a gif. each frame needs to be converted to png first then compressed to astc, so gifs take a few orders or magnitude longer to compress than normal images. when you said it was stuck at 313, then when it was at 135 and stopped then came back alive, and then when you were scrolling through the more info popup, i believe all of these were cases where it was compressing a large gif image (which using sse2 and the relatively older cpu made even slower than one would expect) but you interpreted it as being stuck, then something else you did coincided with it finishing encoding that large gif and thus moving onto the next image.

Since 313 of those were png converted to webp, I converted those back to png
webp is not supported by astcenc but it should be by pillow, the checker converts to png temporarily using pillow as an intermediary for astcenc

The bad news is none of the encoded image's show. All I get to see is a black area. And it's that black area which revealed which it had encoded.

I discovered that if I opened the More Info popup and scrolled through the list, for every image it had encoded, the black area had the aspect ratio of the original image. For those images that still weren't encoded, the black area was a square (verified by checking ID's against encoded images)..
this makes sense, after compressing and loading from .aastc file it knows the size of the image so it draws the right shape, but then looks like opengl didnt like the texture format. the black square is just a placeholder 1x1 texture (resized to fit the more info popup) while images load (same one that was always there, just that first compression takes longer than loading from file so you noticed it now). i feel like changing it to text that says "Loading..." or a loading spinner is unnecessary and distracting, if youre scrolling by with images slowly loading all that text / spinners doesnt bring much info, and is just distracting, so simply showing a brack image when it is still loading but not yet failed is the best solution. as for why even after loading the compressed image it shows black for you is a separate topic, will need to dig deeper to find a way to fix it or detect it to show an appropriate error

So, after 45 minutes of scrolling and waiting till it had done all, the total count became 532 out of 534 entries. Turns out I got two tools in my list that don't have a banner.
well thats atleast some "good" news, compression worked fine, even if slow. that is of course offset by it being a one-time thing, and from now on it will be 1 or 2 images getting compressed at a time when they get updated. also i suppose it would be wise to add a status message to bottom right like "Compressing X textures..." (where each frame in a gif counts as a texture, so the counter is proportionate to processing time), would also help the uncertainty of what is happening that i explained in the first paragraph.

Conclusion. apart from uncovering a bug, ASTC is not something I can use. Encoding works, as I said, but displaying the result is a total failure. (*Tacent however does show them, so it does something other than directly dumping them to the GPU)
ill have to look at how tacentview does it then

Final note... If I turn off ASTC Compression, it does not load the originals, not even after removing all .aastc files. Here yet another suprise surfaced... it actually did load some and it turned out these were all animated gif's. All the rest just stays a solid black area.
Ah, Now this is a situation where you DO need to have Unload off-screen checked. Doing so immediately made it reload the original images.
toggling the astc compression checkbox, or the astc replace checkbox, will unload all images and reload them from file. i supposed i forgot to add a check to make it prefer the original file instead of the .aastc file when the toggle is off, i will fix that.
what happened here as far as i can tell is:
- you disabled the toggle
- it reloaded the images, but it still preferred the .aastc ones (this is a bug, will fix it)
- you deleted the .aastc files, but did not trigger the program to reload the images (as far as its concerned, they are already loaded, just your gpu cant show them)
- unloading off screen triggered the reload. its not at all required to disable astc compression, i simply forgot to make it avoid .aastc files when the astc compression toggle is off. with that bug fixed, then simply toggling off astc compression will reload all images and use the original files instead of .aastc even if present

EDIT: Almost forgot the most important thing.
No errors showed, neither in the Checker nor in the debug console
yeah hard to say why, i wouldve imagined the gpu wouldve just errored saying astc is not supported and the opengl error wouldve crashed the main loop... but then again tacent view loaded it... will need to look more into it.

EDIT: fixed the priority, now .aastc is avoided if the settings is disabled, prefers .gif first then any other format that is not .aastc. but will use .aastc if no other format is present.
also added a counter for image compression, but i realize that with how its currently architected, it can only process one image at a time, so it cant know how many frames the next images will need to compress (they might not have anything to compress even), so it would always show "Compressing 1 texture" when actually theres many images that are queued to compress. only gifs would show a varying number, as they have multiple frames.
what i think ill do is show "Compressing images..." generically when the counter is exactly one, since thats the case for static images and you can actually see that message blink when it finishes one image and moves to the next. for gifs ill make it "Compressing x frames..."

EDIT: did those, also fixed the flicker when clicking the arrows (i had noticed it long ago and never cared enough, thought it was imgui closing the popup, but it was actually due to how i was telling it to do so). and also made it always show a scrollbar in more info popup so width doesnt keep shiftin when changing the tabs or cycling through games
so of all things reported here, only the fact that your gpu doesnt like the astc textures is missing hmmm
 
Last edited:
5.00 star(s) 24 Votes