Tool Others F95Checker [WillyJL]

5.00 star(s) 25 Votes

ascsd

Member
Jul 26, 2021
102
83
As far as I'm concerned the ASTC implementation is basically done. Now it bundles all the appropriate astcenc executables with binary releases, looks for cpu feature flags to decide which one to use, and looks in PATH when running from source. Also fixed the cmd popping up during compression. And finally, error handling should be a lot better, it will now tell you what went wrong like astcenc failing or pillow not recognizing the image or astcenc missing.

Seems to work fine on windows and Linux for me, can't test on macOS.

Now for the interesting part, let's see if for FaceCrap it will correctly detect that cpu features are missing and so use the sse2 version, and most importantly if after compression it will manage to load the textures (and with what error if not so it can be accounted for)

Also ascsd Dukez WhiteVanDaycare give if a shot if you can :D
just as im downloading i see Oops + Build show up :ROFLMAO:
 

WillyJL

Veni, vidi, vici
Donor
Respected User
Mar 7, 2019
1,470
1,287
just as im downloading i see Oops + Build show up :ROFLMAO:
Yeah I was up too late and couldn't for the life of me get the very basic download working for macOS, did it slightly wrong in every possible way the forgot to check if the last change worked xD
That's the only change tho
 
  • Haha
Reactions: ascsd

ascsd

Member
Jul 26, 2021
102
83
everything was running perfectly. speed is great.

I was in the 3rd view mode (forget its name, 3 vertical lines, "No Labels" at top) was dragging the scroll bar up and down alot of times, pausing in between just checking the speed of the image loading. And it just crashed, no log or anything.
Ran debug, kept scrolling up and down wildly, with random pauses, took a few minutes of this to crash again, no log.
Ran debug from terminal, couldnt get it to crash again, been trying for more than 5 mins..

EDIT: also debug shows:
"Unable to import OpenGL.arrays.numpymodule.NumpyHandler: No numpy module present: No module named 'numpy'"

otherwise, with normal use it seems to be great
 
Last edited:

ascsd

Member
Jul 26, 2021
102
83
i had it limited to 1 texture apply per frame,
is it viable to load more than 1 per frame eg 3? or not a good tradeoff?


S3TC is the specification for DXT1/3/5, and i think BC7 is part of BPTC which i also tried. they all look like shit, so idk what these clowns are on about xD
one game dev was pretty annoyed and passionate regarding the lack of support for astc, didnt realize these compression algos were the same as you mentioned before. now I share their frustration. nvidia atleast seems to have adopted it recently so thats good news for everyone
 
  • Like
Reactions: WillyJL

WillyJL

Veni, vidi, vici
Donor
Respected User
Mar 7, 2019
1,470
1,287
everything was running perfectly. speed is great.

I was in the 3rd view mode (forget its name, 3 vertical lines, "No Labels" at top) was dragging the scroll bar up and down alot of times, pausing in between just checking the speed of the image loading. And it just crashed, no log or anything.
Ran debug, kept scrolling up and down wildly, with random pauses, took a few minutes of this to crash again, no log.
Ran debug from terminal, couldnt get it to crash again, been trying for more than 5 mins..

EDIT: also debug shows:
"Unable to import OpenGL.arrays.numpymodule.NumpyHandler: No numpy module present: No module named 'numpy'"

otherwise, with normal use it seems to be great
the numpy thingy is normal, weird thing with how opengl is installed, doesnt affect anything.
i just got one crash myself, but not sure if its the same. i was adding a new game, and got RuntimeError: Set changed size during iteration
thats because during draw it iterates all image instances to check which ones need to be unloaded, and the new game and thus new then image was added at the same time. thats an easy enough fix.
were you adding/removing games when it crashed?
 

ascsd

Member
Jul 26, 2021
102
83
the numpy thingy is normal, weird thing with how opengl is installed, doesnt affect anything.
i just got one crash myself, but not sure if its the same. i was adding a new game, and got RuntimeError: Set changed size during iteration
thats because during draw it iterates all image instances to check which ones need to be unloaded, and the new game and thus new then image was added at the same time. thats an easy enough fix.
were you adding/removing games when it crashed?
Nope, just scrolling wildly in the 3rd view mode

Just tried adding a new game now while in list view. no issue
 

ascsd

Member
Jul 26, 2021
102
83
yeah its a race condition, i got it only once so far, but it is possible. as for what you had, still havent replicated it
yea it wasnt easy to replicate, almost gave up before the second time happened. havent been able to replicate again either..
i was wondering if it was also a race condition that caused it. maybe a load and unload being called at the same time. I did notice one thing, it started to become a touch stuttery/sluggish just before it crashed but not sure if thats from the issue that caused the crash, or a symptom of it just about to crash

think it safe to ignore until it happens again with normal use
 

FaceCrap

Ghost of torrents passed
Donor
Oct 1, 2020
1,539
1,071
Related to hunting an issue in the regular version.

Try resetting the banner on a game that uses an animated gif (e.g. Motherless) then do a full recheck on just that game.

As for trying the build. Have to wait until I'm home.
 

WillyJL

Veni, vidi, vici
Donor
Respected User
Mar 7, 2019
1,470
1,287
Try resetting the banner on a game that uses an animated gif (e.g. Motherless) then do a full recheck on just that game.
does it keep saying image is missing? because i had removed the image.resolve() from game.refresh_image() as i thought just image.loaded = False would be enough, since when drawing the image it checks for .loaded and then does .resolve() internally when loading, but doing it that way the .missing and .invalid remained set even if the image is fine.
i fixed it in feat/tex-compress branch in (specifically the missing and invalid attributes being properties which also check .loaded, if not loaded it will fake being valid so it will do .reload() when shown and trigger proper file lookup and checking/conversion/loading). doing it this way means its also lazyloaded, previously at startup it would resolve the path on disk for all game images, now it does it lazily when they are shown so it also starts up significantly faster.
with how different the image loader is on that branch, i wont look at bug reports for it on main for now until its merged, if its a bug on that branch too then ill take a look at it
 

FaceCrap

Ghost of torrents passed
Donor
Oct 1, 2020
1,539
1,071
does it keep saying image is missing?
YES!!!!
The latest regular build gotten through update works, but has as issue that it becomes unresponsive, and CPU goes mile high, this settles down after a couple of minutes and it becomes responsive again, but the latest source from main does exactly that... keeps saying image is missing.

i had removed the image.resolve() from game.refresh_image()
Saves me a lot of trial and error, fixed.

EDIT: Doesn't occur in the feat/tex-compress branch.
 
Last edited:

FaceCrap

Ghost of torrents passed
Donor
Oct 1, 2020
1,539
1,071
WillyJL
First, I made a filter on Score without changing the value, this has as result that if you filter all tabs, you get all your games in a single list.
At least this allowed me to check which images it had converted. (see *ISSUE)

I only checked ASTC compression and left Unload off-screen clear (found out that wasn't the smartest thing to...).

1st attempt it encoded only 356 of the 534 games. Since 313 of those were png converted to webp, I converted those back to png to see if that would change things, but while comparing image names, I saw it had done webp's too before I reconverted those back to png's.

2nd attempt it encoded only 135 images, and no matter what I did, it didn't do anymore. But while creating this post I checked back and somehow it actually had come alive again as the count had risen. But stopped again at 319... (see *NOTE)

Now as I said, I found an easy way to check which it had done. The bad news is none of the encoded image's show. All I get to see is a black area. And it's that black area which revealed which it had encoded.

I discovered that if I opened the More Info popup and scrolled through the list, for every image it had encoded, the black area had the aspect ratio of the original image. For those images that still weren't encoded, the black area was a square (verified by checking ID's against encoded images)..

#NOTE: Then I noticed that whenever I scrolled through images showing a black square, the encoded count went up again. Contrary to this, when I switched to grid view (middle button) scrolling through the list had no effect on the encoded count.

*ISSUE: Doing so, I also ran into an issue. At first, I used the mouse to click the right-arrow button. This resulted in an annoying flashing because the popup gets closed on the current game and recreated/opened on the next which also causes the game list getting undimmed and dimmed again. Surprisingly when I switched to using the keyboard right-arrow (or left-arrow) keys, the popup stayed open, and the flashing was gone.


So, after 45 minutes of scrolling and waiting till it had done all, the total count became 532 out of 534 entries. Turns out I got two tools in my list that don't have a banner.

Conclusion. apart from uncovering a bug, ASTC is not something I can use. Encoding works, as I said, but displaying the result is a total failure. (*Tacent however does show them, so it does something other than directly dumping them to the GPU)

Final note... If I turn off ASTC Compression, it does not load the originals, not even after removing all .aastc files. Here yet another suprise surfaced... it actually did load some and it turned out these were all animated gif's. All the rest just stays a solid black area.
Ah, Now this is a situation where you DO need to have Unload off-screen checked. Doing so immediately made it reload the original images.

EDIT: Almost forgot the most important thing.
No errors showed, neither in the Checker nor in the debug console
 
Last edited:

WillyJL

Veni, vidi, vici
Donor
Respected User
Mar 7, 2019
1,470
1,287
1st attempt it encoded only 356 of the 534 games. Since 313 of those were png converted to webp, I converted those back to png to see if that would change things, but while comparing image names, I saw it had done webp's too before I reconverted those back to png's.

2nd attempt it encoded only 135 images, and no matter what I did, it didn't do anymore. But while creating this post I checked back and somehow it actually had come alive again as the count had risen. But stopped again at 319... (see *NOTE)

Now as I said, I found an easy way to check which it had done. The bad news is none of the encoded image's show. All I get to see is a black area. And it's that black area which revealed which it had encoded.

I discovered that if I opened the More Info popup and scrolled through the list, for every image it had encoded, the black area had the aspect ratio of the original image. For those images that still weren't encoded, the black area was a square (verified by checking ID's against encoded images)..

#NOTE: Then I noticed that whenever I scrolled through images showing a black square, the encoded count went up again. Contrary to this, when I switched to grid view (middle button) scrolling through the list had no effect on the encoded count.
this whole things sounds like trying to link your actions to some meaning where there isnt. whats most likely happening when it appears to be stuck is it is compressing a gif. each frame needs to be converted to png first then compressed to astc, so gifs take a few orders or magnitude longer to compress than normal images. when you said it was stuck at 313, then when it was at 135 and stopped then came back alive, and then when you were scrolling through the more info popup, i believe all of these were cases where it was compressing a large gif image (which using sse2 and the relatively older cpu made even slower than one would expect) but you interpreted it as being stuck, then something else you did coincided with it finishing encoding that large gif and thus moving onto the next image.

Since 313 of those were png converted to webp, I converted those back to png
webp is not supported by astcenc but it should be by pillow, the checker converts to png temporarily using pillow as an intermediary for astcenc

The bad news is none of the encoded image's show. All I get to see is a black area. And it's that black area which revealed which it had encoded.

I discovered that if I opened the More Info popup and scrolled through the list, for every image it had encoded, the black area had the aspect ratio of the original image. For those images that still weren't encoded, the black area was a square (verified by checking ID's against encoded images)..
this makes sense, after compressing and loading from .aastc file it knows the size of the image so it draws the right shape, but then looks like opengl didnt like the texture format. the black square is just a placeholder 1x1 texture (resized to fit the more info popup) while images load (same one that was always there, just that first compression takes longer than loading from file so you noticed it now). i feel like changing it to text that says "Loading..." or a loading spinner is unnecessary and distracting, if youre scrolling by with images slowly loading all that text / spinners doesnt bring much info, and is just distracting, so simply showing a brack image when it is still loading but not yet failed is the best solution. as for why even after loading the compressed image it shows black for you is a separate topic, will need to dig deeper to find a way to fix it or detect it to show an appropriate error

So, after 45 minutes of scrolling and waiting till it had done all, the total count became 532 out of 534 entries. Turns out I got two tools in my list that don't have a banner.
well thats atleast some "good" news, compression worked fine, even if slow. that is of course offset by it being a one-time thing, and from now on it will be 1 or 2 images getting compressed at a time when they get updated. also i suppose it would be wise to add a status message to bottom right like "Compressing X textures..." (where each frame in a gif counts as a texture, so the counter is proportionate to processing time), would also help the uncertainty of what is happening that i explained in the first paragraph.

Conclusion. apart from uncovering a bug, ASTC is not something I can use. Encoding works, as I said, but displaying the result is a total failure. (*Tacent however does show them, so it does something other than directly dumping them to the GPU)
ill have to look at how tacentview does it then

Final note... If I turn off ASTC Compression, it does not load the originals, not even after removing all .aastc files. Here yet another suprise surfaced... it actually did load some and it turned out these were all animated gif's. All the rest just stays a solid black area.
Ah, Now this is a situation where you DO need to have Unload off-screen checked. Doing so immediately made it reload the original images.
toggling the astc compression checkbox, or the astc replace checkbox, will unload all images and reload them from file. i supposed i forgot to add a check to make it prefer the original file instead of the .aastc file when the toggle is off, i will fix that.
what happened here as far as i can tell is:
- you disabled the toggle
- it reloaded the images, but it still preferred the .aastc ones (this is a bug, will fix it)
- you deleted the .aastc files, but did not trigger the program to reload the images (as far as its concerned, they are already loaded, just your gpu cant show them)
- unloading off screen triggered the reload. its not at all required to disable astc compression, i simply forgot to make it avoid .aastc files when the astc compression toggle is off. with that bug fixed, then simply toggling off astc compression will reload all images and use the original files instead of .aastc even if present

EDIT: Almost forgot the most important thing.
No errors showed, neither in the Checker nor in the debug console
yeah hard to say why, i wouldve imagined the gpu wouldve just errored saying astc is not supported and the opengl error wouldve crashed the main loop... but then again tacent view loaded it... will need to look more into it.

EDIT: fixed the priority, now .aastc is avoided if the settings is disabled, prefers .gif first then any other format that is not .aastc. but will use .aastc if no other format is present.
also added a counter for image compression, but i realize that with how its currently architected, it can only process one image at a time, so it cant know how many frames the next images will need to compress (they might not have anything to compress even), so it would always show "Compressing 1 texture" when actually theres many images that are queued to compress. only gifs would show a varying number, as they have multiple frames.
what i think ill do is show "Compressing images..." generically when the counter is exactly one, since thats the case for static images and you can actually see that message blink when it finishes one image and moves to the next. for gifs ill make it "Compressing x frames..."

EDIT: did those, also fixed the flicker when clicking the arrows (i had noticed it long ago and never cared enough, thought it was imgui closing the popup, but it was actually due to how i was telling it to do so). and also made it always show a scrollbar in more info popup so width doesnt keep shiftin when changing the tabs or cycling through games
so of all things reported here, only the fact that your gpu doesnt like the astc textures is missing hmmm
 
Last edited:

Dukez

Member
Dec 19, 2020
445
1,565
i have now mostly fully implemented astc compression, and dropped the dxt testing code. there are now 3 options: astc compression (in general enable compression and save results to file), astc replace (remove original images after compression), unload off-screen (unload images when not visible).
Haven't really read all the previous posts and I ended up going back to an older drive to solve my issue for the time being. Does this use a different way of rendering the software now or something? (As in not opengl or w/e it was) If so, it wouldn't be hard for me to go back to the newer drivers and test it but if it's like something you have to enable/opt into afterwards it would have the same initial problem lol.
 

WillyJL

Veni, vidi, vici
Donor
Respected User
Mar 7, 2019
1,470
1,287
Haven't really read all the previous posts and I ended up going back to an older drive to solve my issue for the time being. Does this use a different way of rendering the software now or something? (As in not opengl or w/e it was) If so, it wouldn't be hard for me to go back to the newer drivers and test it but if it's like something you have to enable/opt into afterwards it would have the same initial problem lol.
sorry, i got you mixed up with Danv . you had the opengl driver goofy shit, he had concerns about vram. when i was pinging you regarding image compression stuff, i meant to ping them instead.
i havent done anything about underlying opengl stuff, i still believe it was some issue with your newer drivers.
then Danv please try the latest beta (macos builds are failing but windows and linux ones work fine) and test out astc compression + unload off-screen
 

Danv

Well-Known Member
Aug 21, 2020
1,443
2,002
then Danv please try the latest beta (macos builds are failing but windows and linux ones work fine) and test out astc compression + unload off-screen
Win10, i7-7700, gtx1070 8gb, 16gb RAM

testing on tab with 182 animation collection threads
version 11.0.1 - (160 gif + 22 png, total 800mb of HDD) - eats all available VRAM (around 7gb) and around 1.5gb of extra RAM (drops to 1gb after few minutes)
build 1458 with ASTC - (all aastc, total 867mb of HDD) - eats all available VRAM (around 7gb) and around 2gb of extra RAM (drops to 1.5gb later), hard freezes while trying to load many images at once, hard freezes with already loaded images too, took quite a few tries to catch middle of the list - it was lag-scrolling from top straight to bottom and back to top again

yeah, i don't know... it was using VRAM much more efficently and it took much longer to fill up, but in the end result is pretty much the same, if not worse
unload off-screen on the other hand is awesome, works without noticeable problems and keeps VRAM under control, i'm totally fine with image load time (it's around a second with gifs for me, longer with aastc and damn those freezes), so this thing just by itself is already great for me
 

WillyJL

Veni, vidi, vici
Donor
Respected User
Mar 7, 2019
1,470
1,287
Win10, i7-7700, gtx1070 8gb, 16gb RAM

testing on tab with 182 animation collection threads
version 11.0.1 - (160 gif + 22 png, total 800mb of HDD) - eats all available VRAM (around 7gb) and around 1.5gb of extra RAM (drops to 1gb after few minutes)
build 1458 with ASTC - (all aastc, total 867mb of HDD) - eats all available VRAM (around 7gb) and around 2gb of extra RAM (drops to 1.5gb later), hard freezes while trying to load many images at once, hard freezes with already loaded images too, took quite a few tries to catch middle of the list - it was lag-scrolling from top straight to bottom and back to top again

yeah, i don't know... it was using VRAM much more efficently and it took much longer to fill up, but in the end result is pretty much the same, if not worse
unload off-screen on the other hand is awesome, works without noticeable problems and keeps VRAM under control, i'm totally fine with image load time (it's around a second with gifs for me, longer with aastc and damn those freezes), so this thing just by itself is already great for me
i see, then id say its safe to assume those stutters are because your gpu doesnt support astc so it has to fallback convert to rgba, meaning the load times wont be improved by astc. well cant win them all i guess, but good to have as an option because if the hardware supports it its heaps better. glad that atleast unloading offscreen images is useful, even without astc )
from what i read astc was invented by either an arm or an amd engineer, and is maintained by both to a certain degree, so makes sense that amd gpus are further ahead with astc support. i have an rx 7800xt. though ascsd has a recent nvidia card and he has it working with astc.
i tried on qubes os (main os i use, but not for f95 stuff) and yeah i got the same stutters loading astc, in qubes everything is virtualized so there is no hardware gpu, and so it makes sense that there is no hardware acceleration for astc so it needs to fallback to rgba and stutter while doing so
 
Last edited:

KaosKally

Member
Dec 21, 2019
287
325
I did notice now site was a minute down and got the error in the screen when it happened. But the issue is that even the site is back online, the error persists...
Screenshot_1.png

Edit: Now seems all is fine.
 
Last edited:
5.00 star(s) 25 Votes