Tool Others F95Checker [WillyJL]

[WillyJL]

I suppose it may be counter intuitive in the sense of reducing load on the forums but is this something that would we ever be able to run locally rather than relying on another party for?

This is precisely how it was until last month. This program singlehandedly caused multiple downtimes for f95zone. It's basically a DDOS. Now it isn't anymore.

If you mean self hosting your own cache api, technically nothing is stopping you, it's open source and even in the same repository. Only dependency is redis and the xf_user cookie in the env file. But of course this defeats the whole purpose, as we go back to multiple people fetching data from f95zone with the same objective and in the same way for no reason. Also, Sam helped me out by disabling some rate limits on latest updates and the version check api, so you wouldn't be able to run the indexer api as it is (mostly just changing the version check api limit from 1000 to 100), and would encounter some rate limits when getting precise data from latest updates if more than a certain amount of requests in an hour.
The idea is that Sam has some more lenient rate limits for my cache api and that it's the only one, so on its own it's more stressful on the server than 1 single f95checker user in the past, but means that it's only this one thing, not thousands of users all doing it at the same time.

Secondly, if we're receieving cached data from your cache server and comparing it with ours, why does the program require signing in? This seems redundant, I personally don't care about the notifications from this program, telling me someone replied to a thread, etc. That could be done away with, it has no impact on the purpose of this software for me which is keeping track of games and that's it.

So I guess my question is if we're receiving cached data from your server, is there any plans to be able to remove our account from this program and keep it focused on it's main task which from what I can see, requires no account at all.

As stated in the changelog and readme, this is no longer the case. It can be used fully without an account, and it is so by default as notification checking is disabled for new users from now on.
The limitation is mostly download links. To prevent links being taken down, I obviously cannot include naked download links in the cache api. It only includes an indication of where to find the download link in the thread, which then the checker on your machine will retrieve from the f95zone thread, which of course requires an account. This is true for regular, masked, and donor ddl links. Rpdl torrents require their own account, but not an f95zone account.

Worst case scenario would be we're passing our account info onto your cached servers and you have a copy of it... obviously this could be checked by going through the source code. It wouldn't be the first time someone has done something malicious despite it being available in open source code, after all someone has to go looking for it in order to find it. I'm not saying you're doing that, I'm just highlighting an example for others.

You can see in modules/api.py that both fast_check() and full_check(), which are the only methods connecting to my api, use cookies=False.

 

[WillyJL]

app need auto memory cleaning or something
started app (GPU memory around 10% used) -> hover over list of video collection threads (their banners almost always animated) -> GPU memory skyrockets with every hovered thread (with 100+ threads i have in list it reaches 92% used out of 8gigs) -> it stays like this till app is closed, holding all this memory for itself, even in BG mode

Unfortunately that's how it is, banner images are very high quality and the program allows you to zoom a lot into them. With the imgui paradigm this means sending the full resolution texture to the gpu. You can compress the images on disk for yourself to lower both disk and gpu vram usage, and I plan to implement something like this in the tool eventually but it's low priority. Also I hope it's just vram you mean, it used to handle images a bit weirdly in the past, it would keep the texture data in ram if images were shown for a moment then hidden, now it properly passes to gpu.

but yeah keeping all images in vram is a bit of a problem. Not sure what is the correct way to handle it. Loading them is slow, and just having them unload when they go off screen seems like it will just make it horrible to use as it keeps loading images all the time.
As always cache invalidation is the unsolved computer science problem

 

[Danv]

but yeah keeping all images in vram is a bit of a problem. Not sure what is the correct way to handle it. Loading them is slow, and just having them unload when they go off screen seems like it will just make it horrible to use as it keeps loading images all the time.
As always cache invalidation is the unsolved computer science problem

hm, maybe some sort of compromise?
- unload images as tab is switched, so only pics on currently active tab ended up cached (especially helps if one have VNs/animation separated, plus switching tabs can be used to free memory)
- checkbox to disable *display* of banners completely, even better - this checkbox in tab's context menu, so it can be done per tab instead of globally
- straight up button (next to "BG mode" for example) to purge memory without restarting app

considering how aggressive this thing eats memory (with one accidental mouse swoop above the list during scrolling costing few gigs of vram in the instant), at least some sort of a solution still would be nice

 

[Dukez]

As stated in the changelog and readme, this is no longer the case. It can be used fully without an account, and it is so by default as notification checking is disabled for new users from now on.

Ahh that's good to know, perhaps I should be reading the changelogs and stuff lol. I'll have to figure out how to unconnect the account then since I don't need that aspect of it.

The cache api thing makes sense too, wouldn't want to overload the forums and cause issues but it's good to know that it's possible in case the need ever arises for whatever reason.

 

[WillyJL]

Ahh that's good to know, perhaps I should be reading the changelogs and stuff lol. I'll have to figure out how to unconnect the account then since I don't need that aspect of it.

manage > clear > all cookies. though it will make downloads stop working, as those need login as i mentioned. to be honest i could rename that to "F95zone Session" to mimic the "RPDL Session"

 

[FaceCrap]

So I guess my question is if we're receiving cached data from your server, is there any plans to be able to remove our account from this program and keep it focused on it's main task which from what I can see, requires no account at all.

On the one hand I get what you're saying, and by now you have been told how to 'disconnect' your account, so no, there aren't any plans to add this, it's been an option since day one.

But on the other hand.... You've used the checker for how long? And only now you're worried about your login data?

 

[Dukez]

On the one hand I get what you're saying, and by now you have been told how to 'disconnect' your account, so no, there aren't any plans to add this, it's been an option since day one.

But on the other hand.... You've used the checker for how long? And only now you're worried about your login data?

I feel like I might've bought it up a long time ago but this time was because how the program works changed recently with it running through an external server so I was curious on exactly what details get passed to that which is what made me wonder about the whole account thing.

manage > clear > all cookies. though it will make downloads stop working, as those need login as i mentioned. to be honest i could rename that to "F95zone Session" to mimic the "RPDL Session"

Good to know but honestly I've never even used it lol. I guess the Donor DDL thing would automate downloading if I downloaded to the website right? It's something I'll have to think about but for me, I usually install mods in most games so I can get a walkthrough or other patches right so it would still be a manual process anyway.

Perhaps something could be done to automate that too, I know some mods have their own thread altogether but others (e.g the game 'No More Money') have their mods posted under the game as a regular post which probably makes it harder to identify and figure out. If that could all be automated in some way and queued up (important because you need the base game there before adding a mod since they usually overwrite) then it would be pretty neat. It would probably require these mods to be attached to the game in the program somehow though in order to put them all in the same folder in the case of it being like a separate thread altogether?

 

[FaceCrap]

running through an external server so I was curious on exactly what details get passed

Nothing, at least not to the caching server. Your F95zone login cookie is only used and sent to F95zone when accessing F95zone directly.
e.g. using a download link or checking for alerts.

I guess the Donor DDL thing would automate downloading if I downloaded to the website right?

Nope, you'd still need to click a button to start the download. Just like with the regular ones taken from the thread.
The only automatic thing is that it can extract the downloaded archive for you. But since the DDL option downloads to the default C:\Users\<you>\Downloads folder, you also will need to move it yourself to wherever you keep your games.

Perhaps something could be done to automate that too,

Regardless if a mod has its own thread or not... you would still need to download it separately, and manually. It was brought up before but given that the OP of a thread is the only one following a strict template, it's undoable to hunt for mod posts within the thread.
Since these don't follow a strict template it's near impossible to parse them for a download. Some will have them attached, others offering them via filesharing hosts, some even use patreon.
Having the downloads section at least parsed in full makes it easier to get them without having to visit the forum.

 

[ascsd]

Unfortunately that's how it is, banner images are very high quality and the program allows you to zoom a lot into them. With the imgui paradigm this means sending the full resolution texture to the gpu. You can compress the images on disk for yourself to lower both disk and gpu vram usage, and I plan to implement something like this in the tool eventually but it's low priority. Also I hope it's just vram you mean, it used to handle images a bit weirdly in the past, it would keep the texture data in ram if images were shown for a moment then hidden, now it properly passes to gpu.

but yeah keeping all images in vram is a bit of a problem. Not sure what is the correct way to handle it. Loading them is slow, and just having them unload when they go off screen seems like it will just make it horrible to use as it keeps loading images all the time.
As always cache invalidation is the unsolved computer science problem

cant remember if we talked about this suggestion before.. saving a thumbnail sized jpg alongside the original. if gifs are an issue, maybe even choose the middle frame to save as jpg?

cost of jpg storage size should be a small price to pay compared to limited vram.
I have 100 games, none animated, uses 1gb of vram so not directly affected by this, but i imagine the people with 1k+ games would suffer even from stills, let alone gifs.

use thumb for refresh button and grid view, which stays loaded in vram. while full res is loaded when game is opened, and unloaded when closed (if thats too slow, if loading can be done async/threaded you could possibly preload the full size on hover and unload on unhover if that isnt too demanding. possible optimisations are debounce or load queue of size 1 that gets flushed out on selection change so only last hover gets preloaded)

updated games could also be preloaded for the update popup

 

[Versace217]

WillyJL This is just me throwing an idea out; would it be possible for the F95checker itself to add the update date whenever a new version gets released? I saw 6 games versions get updated but only the dates of 3 games were entered correctly.

Having the system itself enter the date on the day a new version gets released would remove any human error from F95's mod team.

 

[WillyJL]

WillyJL This is just me throwing an idea out; would it be possible for the F95checker itself to add the update date whenever a new version gets released? I saw 6 games versions get updated but only the dates of 3 games were entered correctly.

Having the system itself enter the date on the day a new version gets released would remove any human error from F95's mod team.

not viable. neither the checker nor the api fetches info for threads when they are updated. they are fetched when someone refreshes their list and they have it in their list. so if an update happened last year but no f95checker user had it in their library, the date would be today, even though the update was last year. please drop this matter. it is an unsolvable computer science dilemma. i am fucking tired of all this crying about some stupid date. use a more reliable way of sorting, like the checkboxes, to sort by whether you have played the update, not when the update was released.

 

[WillyJL]

cant remember if we talked about this suggestion before.. saving a thumbnail sized jpg alongside the original. if gifs are an issue, maybe even choose the middle frame to save as jpg?

cost of jpg storage size should be a small price to pay compared to limited vram.
I have 100 games, none animated, uses 1gb of vram so not directly affected by this, but i imagine the people with 1k+ games would suffer even from stills, let alone gifs.

use thumb for refresh button and grid view, which stays loaded in vram. while full res is loaded when game is opened, and unloaded when closed (if thats too slow, if loading can be done async/threaded you could possibly preload the full size on hover and unload on unhover if that isnt too demanding. possible optimisations are debounce or load queue of size 1 that gets flushed out on selection change so only last hover gets preloaded)

updated games could also be preloaded for the update popup

i dont see how this would help. gpu does not speak jpeg. it speaks pixels. any image file is decoded and rendered to a pixel bitmap and transferred to gpu. no matter what file fortmat or compression you use for the image file, what the gpu has to store will always be the same size for the same width and height in pixels. only thing that will change with image format and compression is disk space, and ram usage while its between cpu and gpu, but once its on gpu vram is removed from ram.
also, if you mean using lower resolution images, then no grid would not be able to use them, as you could keep a low amount of grid columns and keep the window fullscreen, now the grid images are huge and would look horrible at a lower resolution. only place where lower resolution would help are refresh button on hover, and updates popup. the amount of complexity it introduces is not worth the savings.
what im considering to investigate is if theres some way to optimize how theyre actually stored in vram (consider game textures, theres 4k texture mods and they have so many textures, yet they work with today's gpus), and to see if convertin to bitmap on disk would help load times in exchange for disk space. if so, could possibly just store bitmaps and unload as soon as images are not on screen

 

[Dukez]

i dont see how this would help. gpu does not speak jpeg. it speaks pixels. any image file is decoded and rendered to a pixel bitmap and transferred to gpu.

Speaking of the GPU since I have a unique problem that results in your program turning into a white screen when using newer AMD drivers in a virtual machine setup (that I'm once again trying to figure out a way around without reverting drivers lol), I was searching through the thread here and saw there is apparently a way to disable F95Checker from using the GPU but it was in Linux using some chromium command.

However I use Windows and not linux so is there a way to do this same thing, launch the program with the --disable-gpu line?

I can disable the GPU entirely in Windows but then the program doesn't open at all and using the debug launcher shows the error "Could not initialize window" which makes me assume it needs a GPU and doesn't have like a software rendering fallback?

My driver issue extends further than just you're program though, renpy games get the same thing but if you hold shift while opening the game you can switch the render. So just for informative purposes, renpy games work fine for me on this driver if I'm using the ANGLE2 renderer but not the default GL2Renderer - that gets the white screen too.

Obviously this is an AMD issue first and foremost, I'm just trying to figure out a way to get things working since it's a very driver issue that only affects your program and renpy, one of which I can work around easily lol. It made me test out an older version of your program and I got a slightly different error with this one. "GLFWError: (65542) b'WGL: The driver does not appear to support OpenGL'". I guess this is more to the point and I'll have to research it, but is there a way to change how the program renders itself built in or not, probably not I'm guessing? I am still curious if the disable-gpu thing would work since the post here mentioned it did work for them but that was under linux.

 

[estrada777]

the tool is made for users managing games theyre playing and keep track of what they have installed and played and so on. i understand that this isnt exactly what you use it for, but youre the minority here then and i cant change the behavior yet again just for you. if you explain better what exactly is your workflow i can suggest a way to use the checkboxes in a way that makes sense for you.

from what i understand you do game ports and modding right? i would imagine you need to download games to do these things. here is what i imagine could work:
- download an update, then mark as installed (checkbox fully set), this will make the finished checkbox half set
- port and mod the version you just downloaded, then mark as finished (checkbox fully set), now youre done with this game until next update
- when an update happens, the installed checkbox will be half set, as you do not have the latest one installed
- again from the start: update it, then mark as installed
- now finished checkbox is half set, because you havent made ports or mods or whatever for this new version you just downloaded
- port and mod this update, then mark as finished, done until next update
you know which games have been updated by the half-set installed checkbox, and you know what you still need to port/mod by the half-set finished checkbox. you cant port/mod the games without downloading, so first step is downloading the update.
if you dont usually keep games installed after youre done modding them, then just delete the files after marking as finished. the tool will remember you marked it as installed and finished for that version, regardless of what files you have downloaded, thus it will still tell you about an update with the installed checkbox being half-set.

if this is not the kind of workflow you need then ill need more information to understand how to help you use it effectively

I think I know what is happening that I didn't fully understand. First, I use filters. I filter out the games that I have marked Finished. When a new update comes in, sometimes the Finished box is checked and sometimes it is not, so my filter is causing it to disappear from my list even though I haven't done anything with it yet.

Is the Finished filter aware of the three different states or is it only a two state filter?

 

[WillyJL]

Speaking of the GPU since I have a unique problem that results in your program turning into a white screen when using newer AMD drivers in a virtual machine setup (that I'm once again trying to figure out a way around without reverting drivers lol), I was searching through the thread here and saw there is apparently a way to disable F95Checker from using the GPU but it was in Linux using some chromium command.

However I use Windows and not linux so is there a way to do this same thing, launch the program with the --disable-gpu line?

that was when talking about the integrated browser, its an argument passed to it. its equivalent to passing this argument to chrome.exe. doesnt apply to f95checker itself

I can disable the GPU entirely in Windows but then the program doesn't open at all and using the debug launcher shows the error "Could not initialize window" which makes me assume it needs a GPU and doesn't have like a software rendering fallback?

as far as i know, "software rendering" just means that its not using a gpu but the cpu as a gpu. its still using a "gpu", just changes whether its the cpu in a gpu coat, or an actual gpu. what youre doing is disabling the gpu entirely, so there is no gpu

My driver issue extends further than just you're program though, renpy games get the same thing but if you hold shift while opening the game you can switch the render. So just for informative purposes, renpy games work fine for me on this driver if I'm using the ANGLE2 renderer but not the default GL2Renderer - that gets the white screen too.

this tells me that its not an issue with the gpu driver in general, and youre not making renpy games disable gpu rendering i would assume. to me it sounds more like youre making the gpu use a different renderer, but still using a gpu. i would imagine that disabling the gpu like you tried for f95checker would make renpy games fail to work aswell

Obviously this is an AMD issue first and foremost, I'm just trying to figure out a way to get things working since it's a very driver issue that only affects your program and renpy, one of which I can work around easily lol. It made me test out an older version of your program and I got a slightly different error with this one. "GLFWError: (65542) b'WGL: The driver does not appear to support OpenGL'".

to me it sounds like these new amd drivers have broken opengl. opengl is not the only language gpus speak. but it is the only one f95checker speaks. implementing more graphical backends is for all intents and purposes out of the question, opengl is the most basic and accepted standard for these simple graphics. implementing more is way too complicated and would invalidate a bunch of code we already have (for example, everything to do with images is opengl specific, and the window itself is glfw which is opengl only)

PS: i looked at renpy source code, theyre using sdl via pygame, not glfw. angle is a translation layer from opengl es to native operating system calls, sdl supports this, glfw doesnt, let alone pyopengl or pyglfw. also there is no option to enable software rendering in these libraries.

i have spent far more time investigating this than i wanted to, and got no further than i knew previously really. rewriting everything to another graphics library is out of the question. basically im telling you youre on your own on this issue. wait for amd to fix it or whatever

 

[WillyJL]

Is the Finished filter aware of the three different states or is it only a two state filter?

 

[estrada777]

View attachment 4365809

Ah, I wondered what outdated meant.

So a game that I previously marked as Finished, gets an update and is supposed to get the Finished half check. But what I've been seeing is that sometimes these games are getting the full check, and in my case get hidden by the filter.

 

[ascsd]

i dont see how this would help. gpu does not speak jpeg. it speaks pixels. any image file is decoded and rendered to a pixel bitmap and transferred to gpu. no matter what file fortmat or compression you use for the image file, what the gpu has to store will always be the same size for the same width and height in pixels. only thing that will change with image format and compression is disk space, and ram usage while its between cpu and gpu, but once its on gpu vram is removed from ram.
also, if you mean using lower resolution images, then no grid would not be able to use them, as you could keep a low amount of grid columns and keep the window fullscreen, now the grid images are huge and would look horrible at a lower resolution. only place where lower resolution would help are refresh button on hover, and updates popup. the amount of complexity it introduces is not worth the savings.
what im considering to investigate is if theres some way to optimize how theyre actually stored in vram (consider game textures, theres 4k texture mods and they have so many textures, yet they work with today's gpus), and to see if convertin to bitmap on disk would help load times in exchange for disk space. if so, could possibly just store bitmaps and unload as soon as images are not on screen

jpg was for storing the thumbnails to reduce size since u'd be adding another image alongside the original image. just so this change doesn't have a big impact.

I meant lower res, to reduce vram usage. my grid is 3 columns and forgot its customizable so I thought you could get away with smaller..
Id say only low column grids need the larger images, so you could just treat it as an outlier, or even have a system that checks image size and choses the low vs high res image, that way they are not hindering optimisation for the small images.
In saying that, im giving all these suggestions in a optimistic vacuum, i know you might not have the time

as for storing as bitmap, storage size might balloon real quick, but im sure you'd already test that beforehand

onto a more promising path:
did a quick search, opengl supports texture compression which seems to be alot better than simple lower res resize.

You must be registered to see the links

DXT1 seems to be popular, not sure if there is a newer compression format thats taken over nowadays

You must be registered to see the links

glfw should support glCompressedTexImage2D? havent checked, but more importantly idk if its exposed to imgui

 

[WillyJL]

So a game that I previously marked as Finished, gets an update and is supposed to get the Finished half check.

for the 4th time now, no, this is not what is intended to happen at all. the *installed* checkbox is half-set after an update. as you do not have the latest version installed. the finished checkbox indicates if what is installed is the same version that you finished. when theres an update, it doesnt magically install a new version, therefore the installed version is different than latest version, causing a half-set *installed* checkbox, but since its still the same installed version that you had marked as finished, finished is still fully set, as you did not install any new content to be marked as finished. youve already finished what you have installed. only after changing the installed checkbox from half set to fully set, the installed version will differ from the finished version, and thus the finished checkbox will be half set.
the filter you have was suitable for the old behavior, not the new. for 4 times now ive been asking you what the workflow was and repeating the same explanation that frankly it seems you never even bothered to read, but finally we know the sacred workflow you use, that my previous explanations wouldve explained to actually do the opposite of what you wanted if only they had been read.

snarky comments aside, i take it that you want to only see games you need to do whatever with, then when youre done with them hide them until the next update.
this is not really possible with the current filter system if you use both checkboxes (after an update installed is half set, then after installing the update the finished checkbox is half set) as filters do not have any logical "and/or" nor any grouping at this time.
i plan to have these in the future as part of the filter rework, but that is months if not even years away.
what you can do to preserve this workflow is change to using only 1 checkbox.
if the game is not marked installed (checkbox fully empty), the finished checkbox will relate to the latest version instead of the installed version (as there is none). this means you can use it as a simple marker for "i am done with this game" that gets half-set right after a game gets an update (again stress on the "if the game is not marked installed").
if this is what you want, you can proceed to:
- ctrl+a to select all games (repeat for all tabs if you have more than one
- right click > installed > click the X, this marks all games as not installed
- filter by finished, disable "include outdated", invert the filter (this will hide games that have finished fully checked)
now only games with finished half checked, or not checked, will show. and since all the installed checkboxes are empty, game updates will cause the finished checkbox to be half-set instead of the installed checkbox

 

[estrada777]

for the 4th time now, no, this is not what is intended to happen at all. the *installed* checkbox is half-set after an update. as you do not have the latest version installed. the finished checkbox indicates if what is installed is the same version that you finished. when theres an update, it doesnt magically install a new version, therefore the installed version is different than latest version, causing a half-set *installed* checkbox, but since its still the same installed version that you had marked as finished, finished is still fully set, as you did not install any new content to be marked as finished. youve already finished what you have installed. only after changing the installed checkbox from half set to fully set, the installed version will differ from the finished version, and thus the finished checkbox will be half set.
the filter you have was suitable for the old behavior, not the new. for 4 times now ive been asking you what the workflow was and repeating the same explanation that frankly it seems you never even bothered to read, but finally we know the sacred workflow you use, that my previous explanations wouldve explained to actually do the opposite of what you wanted if only they had been read.

snarky comments aside, i take it that you want to only see games you need to do whatever with, then when youre done with them hide them until the next update.
this is not really possible with the current filter system if you use both checkboxes (after an update installed is half set, then after installing the update the finished checkbox is half set) as filters do not have any logical "and/or" nor any grouping at this time.
i plan to have these in the future as part of the filter rework, but that is months if not even years away.
what you can do to preserve this workflow is change to using only 1 checkbox.
if the game is not marked installed (checkbox fully empty), the finished checkbox will relate to the latest version instead of the installed version (as there is none). this means you can use it as a simple marker for "i am done with this game" that gets half-set right after a game gets an update (again stress on the "if the game is not marked installed").
if this is what you want, you can proceed to:
- ctrl+a to select all games (repeat for all tabs if you have more than one
- right click > installed > click the X, this marks all games as not installed
- filter by finished, disable "include outdated", invert the filter (this will hide games that have finished fully checked)
now only games with finished half checked, or not checked, will show. and since all the installed checkboxes are empty, game updates will cause the finished checkbox to be half-set instead of the installed checkbox

I'll go with the one box solution.
Sorry for bothering you.