Tool Others F95Checker [WillyJL]

[blackop]

Made any updates on your system between builds?

Spoiler: glibc 2.38 not found

You don't have permission to view the spoiler content. Log in or register now.

Seems to be a case of "everything is faster than a debian-update, even glaciers"

Kinda. Yesterday build was done on ubuntu 22.04 with python 3.11.x, today on 24.04 with python 3.12.3. But it worked on my host manjaro system so I assumed everything is fine. Give me 10-15 mins and I'll make one on yesterday VM.

Upd:

Spoiler: Host glibc version

You don't have permission to view the spoiler content. Log in or register now.

Spoiler: VM Ubuntu 22.04 glibc version

You don't have permission to view the spoiler content. Log in or register now.

LucasG uploaded new build to shared folder, renamed old one and specified glibc version from build env for both.

Upd 2:
Added build made on dockerized ubuntu 20.04 with glibc 2.31 and Python 3.12.7

 

[blackop]

My antivirus is flagging it and showing this:

Trojan/Wacatac.B!ml.

Not sure how I can help with it. Here is results of

You must be registered to see the links

for win11 archive. Also there is a link to

You must be registered to see the links

, you can build it yourself from source code for any platform, just remove .dev69 from asynciolimiter version in requirements.txt as mentioned in original post.

 

[LucasG]

The new glibc 2.35 build works fine.
Just a very small thing: while the debug cmd says n game(s) left, the GUI window states Running n-1 full rechecks...
But it's no actual error, just me finding tiny things

EDIT: Recheck with 453 games needed 15 minutes and took 10 breaks for 429's

 

[Danv]

View attachment 4209109

probably stupid question, but where to find log like this?
on my system Checker can't download game pictures (dunno why), i have to update them manually every time existing pic get changed, but checking through three hundred games to find which one got new pic and throws error is a fucking pain, having actual log would be a blessing

 

[Jbudbill]

Probably a dumb question but how do I increase the text size? Everything is very small and hard to read and understand.

 

[WhiteVanDaycare]

Probably a dumb question but how do I increase the text size? Everything is very small and hard to read and understand.

Try this setting

EDIT: you have to press enter after you change it

 

[blackop]

probably stupid question, but where to find log like this?
on my system Checker can't download game pictures (dunno why), i have to update them manually every time existing pic get changed, but checking through three hundred games to find which one got new pic and throws error is a fucking pain, having actual log would be a blessing

That was something I temporarily added to show progress and "proof" of application waiting on 429 error without doing anything. And it's not a real "update", just two rows I wrapped making actual HTTP request in like "we're sending request to check this game -> we're received response on that request".

I had same issue and it was some connectivity issue to attachments.f95zone.to - reason why I implemented proxy support, I was tired to turn on and off VPN on every new game or random picture update. If you have access to some VPN you can try to add new game and start full recheck for it:

Spoiler: Steps

You don't have permission to view the spoiler content. Log in or register now.

If you get error try "Full Recheck" again but with VPN connected (better other region/country with less censorship). If you succeed - that's it, just find proxy server and put it's setting in F95Checker. But login window only works with HTTP/HTTPS without authorization (I haven't implemented it), with SOCKS5 proxy without authorization (chromium doesn't support authorization for this type) or without proxy. Update process works with any type with/without authorization.

The new glibc 2.35 build works fine.
Just a very small thing: while the debug cmd says n game(s) left, the GUI window states Running n-1 full rechecks...
But it's no actual error, just me finding tiny things

EDIT: Recheck with 453 games needed 15 minutes and took 10 breaks for 429's

Probably it's because of this. Not sure I want to do anything with it because logs are static and I don't want to implement dynamic console output.

Also cought situation I mentioned above - one of games issued request before 429 (on another game) and got response after 429.
Changes already included in MEGA builds in original post.

I checked it again and tested during development and counter matched logs most/all the times.

Spoiler: Screens

You don't have permission to view the spoiler content. Log in or register now.

 

[GrammerCop]

With this issue, several things come to mind.

1. Will the auto-full-recheck lock the app or prevent you from doing other things in the app while it is working?
2. Will there be lengthy issues with 1000+ games? In other words, should I plan on starting the app and ignore it for a while if there are a lot of rechecks?
3. Should the app be changed to make a list of the games to auto-recheck and work off that list in the background in a TSR process (which does not stop running when the app is closed)?
4. A while ago I saw a post (or maybe even posted it myself) requesting an option to skip recheck for individual games. Considering the current issues with auto-recheck, should this option be added to the app to reduce the number of games being rechecked?

 

[evilution382]

With this issue, several things come to mind.

1. Will the auto-full-recheck lock the app or prevent you from doing other things in the app while it is working?
2. Will there be lengthy issues with 1000+ games? In other words, should I plan on starting the app and ignore it for a while if there are a lot of rechecks?
3. Should the app be changed to make a list of the games to auto-recheck and work off that list in the background in a TSR process (which does not stop running when the app is closed)?
4. A while ago I saw a post (or maybe even posted it myself) requesting an option to skip recheck for individual games. Considering the current issues with auto-recheck, should this option be added to the app to reduce the number of games being rechecked?

Just did a full recheck 1600ish games, because why the hell not
took... 40 minutes, just about, seemed to have full functionality of the app during the progress

 

[Jbudbill]

Try this setting View attachment 4210133

EDIT: you have to press enter after you change it

I tried that before but was trying to set it too high. Thanks for the info got it fixed!!

 

[GrammerCop]

Just did a full recheck 1600ish games, because why the hell not
took... 40 minutes, just about, seemed to have full functionality of the app during the progress

Nice. But as far as I know, you also had an issue with this. What did you do to have success?

 

[evilution382]

Nice. But as far as I know, you also had an issue with this. What did you do to have success?

Used the modded version from Blackop

Here is my custom build for linux and windows:

You must be registered to see the links

Include (all are mine but 429 status fix isn't uploaded to repo):

• You must be registered to see the links
• You must be registered to see the links
• rate limiter: "API RPM" under "Refresh" settings section - you can try your luck and use it instead of next one, otherwise leave it at 0 (unlimited)
• retries for 429 response: "Retry on 429" under "Refresh" settings section, right under previous one, so far it worked for me

I suggest to make a copy/backup of your current f95checker settings directory since my build will modify settings table in sqlite file.

- WIN: C:\Users\Username\AppData\Roaming\f95checker
- Linux (at least on my manjaro): /home/username/.config/f95checker/

Upd: Uploaded changes for last two -

You must be registered to see the links

. Version of asynciolimiter used in requirements.txt also slightly modified.
You should remove .dev69 from the end if you want to build it by yourself, but expect AssertionError from time to time due to

You must be registered to see the links

.
I replaced those lines with plain print to stderr (which is probably not working for some reason since I never saw it in console, but at least it's not blocking update process anymore).

Upd. 2: This workaround is only applied to full rechecks, fast one still may fail, but it's easy to restart it by yourself.

Upd. 3: Updated archives in MEGA. Added this, fixed bug in my changes in asynciolimiter, slightly rewrote debug related logic - now message about pause will be printed to console if running F95Checker-Debug binary from terminal/cmd/etc.

 

[GrammerCop]

Used the modded version from Blackop

Thanks. I saw. I tried. It worked well.
You are correct that it seems to look like it is locked up. Perhaps adding a small message "...pausing".

I know what it is doing so that is not an issue for me. It's just the developer in me jumping out.

 

[blackop]

Thanks. I saw. I tried. It worked well.
You are correct that it seems to look like it is locked up. Perhaps adding a small message "...pausing".

I know what it is doing so that is not an issue for me. It's just the developer in me jumping out.

Done, new builds uploaded to mega.

Spoiler

You don't have permission to view the spoiler content. Log in or register now.

 

[GrammerCop]

Done, new builds uploaded to mega.

Spoiler

You don't have permission to view the spoiler content. Log in or register now.

Great. I think I read you say that it skips archived entries. Does this still hold true?

 

[blackop]

Great. I think I read you say that it skips archived entries. Does this still hold true?

Yes, if you uncheck this box

 

[GrammerCop]

Yes, if you uncheck this box
View attachment 4212946

Great. This allows me to skip the games I don't want to be refreshed. Thanks a lot. I hope your mod will be incorporated into the base version soon. Great job.

 

[Josus Joestar]

DO 430 tracking watch threads considered too many that it can proc 429 error? Is there any solution to fix this?

 

[GrammerCop]

DO 430 tracking watch threads considered too many that it can proc 429 error? Is there any solution to fix this?

Here is my custom build for linux and windows:

You must be registered to see the links

Include (all are mine but 429 status fix isn't uploaded to repo):

• You must be registered to see the links
• You must be registered to see the links
• rate limiter: "API RPM" under "Refresh" settings section - you can try your luck and use it instead of next one, otherwise leave it at 0 (unlimited)
• retries for 429 response: "Retry on 429" under "Refresh" settings section, right under previous one, so far it worked for me

I suggest to make a copy/backup of your current f95checker settings directory since my build will modify settings table in sqlite file.

- WIN: C:\Users\Username\AppData\Roaming\f95checker
- Linux (at least on my manjaro): /home/username/.config/f95checker/

Upd: Uploaded changes for last two -

You must be registered to see the links

. Version of asynciolimiter used in requirements.txt also slightly modified.
You should remove .dev69 from the end if you want to build it by yourself, but expect AssertionError from time to time due to

You must be registered to see the links

.
I replaced those lines with plain print to stderr (which is probably not working for some reason since I never saw it in console, but at least it's not blocking update process anymore).

Upd. 2: This workaround is only applied to full rechecks, fast one still may fail, but it's easy to restart it by yourself.

Upd. 3: Updated archives in MEGA. Added this, fixed bug in my changes in asynciolimiter, slightly rewrote debug related logic - now message about pause will be printed to console if running F95Checker-Debug binary from terminal/cmd/etc.

 

[WillyJL]

seems like a band-aid fix, and from what i understand any single thread request is quite heavy on your backend.
might be time to make my own heavily cached api that does full checks to f95zone slowly every now and then and keeps results to serve quickly to everyone.
would you be ok with something like this?

general idea im thinking of is:
- keep a server of mine monitoring the latest updates page
- when an update is up then full check it and save parsed data
- when a user asks for a thread info serve the cached info
- if the cached info is like more than 6 or 12 hours old, then recheck to f95zone
this way, each thread will be fetched from f95zone at most once per 6 hours, across whole f95checker userbase, since users dont reach out to f95zone anymore.
would make the dedicated checker api you made useless.
only thing that would reach to f95zone directly from users would be notification checking basically

for the record i have no clue when i will have enough time to implement this and let alone make it stable, but atleast havign a plan is a start

i decided to go ahead with this idea. im making a very simple wrapper api as described there, it will parse the same way using the same code as f95checker currently does. basically just a single endpoint that does what all the checkers already do but only does it once to avoid strain on the server.
i talked with sam and he said its up to me whether we go with a delay after 429 or if i make my own cached api. he also said they still plan to have an api of their own but no eta yet so if i wanna make my own thing it would be duplicated work. but considering that what i plan to do is much more primitive than a solution that would serve the whole website, i think its fine.
i already pushed a branch with the core structure of the indexer im making with redis backend and fastapi. core logic works well, now just need to make the part that actually scrapes f95zone and saves data into redis. and decide how to monitor the latest updates page to grab new data earlier as it becomes available. still, totally doable, might even be done within a week.