Tool Others F95Checker [WillyJL]

[batblue]

My server uses Cloudflare self signed certs, so no one would be able to securely connect to it except Cloudflare. And everyone else connects to Cloudflare, and I doubt they have invalid certs being used. So yeah something is probably going wrong either with your system's root CA or python's bundled ones (not sure if it does bundle them tho).
While working on all this I noticed that I was using ssl=False in aiohttp, probably from a time where younger and more naive me thought this was an acceptable fix for the issue. Unacceptable to use insecure ssl, so I re-enabled it, so it might be that you had this issue the whole time but aiohttp simply wasn't checking CA validity

Okay, makes sense. If I clone the code from Github and run it from the indexer branch, everything seems to work okay, so I suspect it's something in the libs that cx_Freeze includes in the app bundle. This is way outside my area of expertise, but I found

You must be registered to see the links

that might help?

 

[WillyJL]

How to find the latest beta version -

1. Go to
   You must be registered to see the links
   and select Actions
   View attachment 4244202
2. Select the most recent workflow (1229 in this case)
   View attachment 4244204
3. Scroll down and select the Artifact that matches your OS to download the program
   View attachment 4244205
4. You will find a zip file within the zip file, extract the files to where you want to run the checker from. On first run it will create your datastore, etc. If upgrading it is a good idea to create a backup copy of your datastore first (the location is listed in the OP of this thread).

linked in OP, will be on github readme too, thanks!

 

[WillyJL]

Updated to 1229 and i assume this is still the latest one, so far no issues. Now if i think about it, I might have had internet down over night for a couple minutes and app probably did a check at that time. I live in Romania and we have nothing blocked, not on ISP or otherwise, well except the usual region error on YT or other sites.

followup on this, i fixed the second part of the error, where res was not defined. but that was an error in how the original error was getting handled. this fix wouldnt have prevented the core issue, that it could not connect to github for whatever reason, and looks like we have the answer to why that was.

During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/Users/runner/work/F95Checker/F95Checker/modules/api.py", line 574, in fast_check UnboundLocalError: cannot access local variable 'res' where it is not associated with a value

same goes for this

During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/Users/runner/work/F95Checker/F95Checker/modules/utils.py", line 87, in coro_wrapper File "/Users/runner/work/F95Checker/F95Checker/modules/api.py", line 1130, in refresh File "/Users/runner/work/F95Checker/F95Checker/modules/api.py", line 590, in fast_check UnboundLocalError: cannot access local variable 'last_changes' where it is not associated with a value

but this was a bit different, while doing things way too late and tired, i had changed part of the function without adapting the rest of it, so it had ended up running when it shouldnt have (was trying to continue with fast_check behavior during error handling), fixed now

ill be looking into that SSL cert issue you got now

 

[WillyJL]

Okay, makes sense. If I clone the code from Github and run it from the indexer branch, everything seems to work okay, so I suspect it's something in the libs that cx_Freeze includes in the app bundle. This is way outside my area of expertise, but I found

You must be registered to see the links

that might help?

im actually getting the same myself on linux, when i use packed binaries. definitely looks like somethign going wrong with cx_Freeze

i dont understand why it would bundle its own certs however... all OSs should have their own system certificate bundles... needing to update dependencies to get new certs sounds very inefficient

 

[batblue]

i dont understand why it would bundle its own certs however... all OSs should have their own system certificate bundles... needing to update dependencies to get new certs sounds very inefficient

Makes no sense to me, either. Homebrew on Mac has a package that installs the cacert.pem from

You must be registered to see the links

for use with open source stuff, and it's a dependency of most other packages. Apparently the newer versions os macOS don't include it at all. No idea why.

 

[WillyJL]

Makes no sense to me, either. Homebrew on Mac has a package that installs the cacert.pem from

You must be registered to see the links

for use with open source stuff, and it's a dependency of most other packages. Apparently the newer versions os macOS don't include it at all. No idea why.

this is looking more and more stupid by the minute. the pypi package "certifi" includes a "cacert.pem", which is what requests uses. turns out, distributions have a modified certifi package in their repos, which points to the system certs. but this doesnt answer everything, because i dont see anywhere that aiohttp would use certifi, it seems to only rely on ssl.create_default_context(), which does some special loading for windows environments then falls back to SSLContext.set_default_verify_paths(), which is not in python code and isnt clear what it does at all.
if all else fails, we should be able to manually determine where certificates live, and provide a custom SSLContext with them hardcoded for aiohttp

 

[batblue]

this is looking more and more stupid by the minute. the pypi package "certifi" includes a "cacert.pem", which is what requests uses. turns out, distributions have a modified certifi package in their repos, which points to the system certs. but this doesnt answer everything, because i dont see anywhere that aiohttp would use certifi, it seems to only rely on ssl.create_default_context(), which does some special loading for windows environments then falls back to SSLContext.set_default_verify_paths(), which is not in python code and isnt clear what it does at all.
if all else fails, we should be able to manually determine where certificates live, and provide a custom SSLContext with them hardcoded for aiohttp

Wow, what a pain. If nothing else works out, I did a bunch of googling on all this, and it turns out that there's a cert file at /etc/ssl/cert.pem that's part of a standard macOS install. (The one used by the OS is stored in the Apple keychain, which is kind of useless.) People seem to think it's outdated, but maybe it'll still work for Cloudflare?

 

[WillyJL]

Wow, what a pain. If nothing else works out, I did a bunch of googling on all this, and it turns out that there's a cert file at /etc/ssl/cert.pem that's part of a standard macOS install. (The one used by the OS is stored in the Apple keychain, which is kind of useless.) People seem to think it's outdated, but maybe it'll still work for Cloudflare?

funny, same path as on some linux distros. though the standard seems to vary between distros, some have a symlink in that path.
what im doing is i have a list of known locations for all OSs, even not from a standard install (eg, homebrew curl on macos will be checked too), use the first one found, if none of them exist then use certifi as a fallback.
so knowing this path exists helps a lot, thanks!

EDIT: done and pushed, hopefully should work now.

 

[blackop]

Looks like I'll end up with a mixed solution then, use aiolimiter but retry indefinitely with some delay if a rate limited response is received. Do you have your rate limited fork code anywhere?

Yes, in this

You must be registered to see the links

mostly. But as I said few times before, I made changes in asynciolimiter library for preventing exception in

You must be registered to see the links

 

[batblue]

what im doing is i have a list of known locations for all OSs, even not from a standard install (eg, homebrew curl on macos will be checked too), use the first one found, if none of them exist then use certifi as a fallback.
so knowing this path exists helps a lot, thanks!

EDIT: done and pushed, hopefully should work now.

Great idea, being able to load the cert file from homebrew as well as from the system should help a lot.

I just tried build 1232 on my Mac and it connects with no problem. Everything seems to work fine so far, will report any other bumpiness. Thanks for the quick fix!

 

[BrockLanders]

Holy cow is it fast now! I have 400+ pages to check between novels, mods, and utilities, and it cached and completed on Windows 11 fast as fuck boi

We appreciate your hard work! Thanks WillyJL and the team!

 

[WhiteVanDaycare]

Build 1229 gave this strange error:

 

[WillyJL]

Build 1229 gave this strange error:
View attachment 4247059

How long was the tool open for?

EDIT: for context, I recently changed how the login check is done, instead of using /account/ it uses /sam/latest_alpha/ which also requires login, but is less heavy on f95zone's backend. However from /account/ I also used to grab the xfToken which is needed for notification checking and quick search. As a temporary solution I made it so the first login check uses /account/ and grabs the xfToken, then subsequent login checks use /sam/latest_alpha/ so it's less heavy and doesn't rate limit you. But my guess is, xfToken needs to be changed every now and then, and probably more often than I had hoped. Might need to go back to using /account/ with rate limiting, or find another way to sniff xfToken (maybe a cookie, or maybe it is also in /sam/latest_alpha/ returned html, or maybe it can be generated in some other way)

 

[WhiteVanDaycare]

How long was the tool open for?

LOL, constantly since updating - say over a day.

EDIT: also, while a low priority, the tool occasionally forgets the sorting and sorts by title z-a (not sure what triggers it though).

EDIT2: I restarted the app and it is no longer getting the error.

 

[EvilChef]

Just ran the tool on a 2080 games library and it went flawlessly. I used the 1232 beta build.

 

[WillyJL]

How long was the tool open for?

EDIT: for context, I recently changed how the login check is done, instead of using /account/ it uses /sam/latest_alpha/ which also requires login, but is less heavy on f95zone's backend. However from /account/ I also used to grab the xfToken which is needed for notification checking and quick search. As a temporary solution I made it so the first login check uses /account/ and grabs the xfToken, then subsequent login checks use /sam/latest_alpha/ so it's less heavy and doesn't rate limit you. But my guess is, xfToken needs to be changed every now and then, and probably more often than I had hoped. Might need to go back to using /account/ with rate limiting, or find another way to sniff xfToken (maybe a cookie, or maybe it is also in /sam/latest_alpha/ returned html, or maybe it can be generated in some other way)

looks like xfToken and xf_csrf are generated on serverside and only when visiting full xenforo pages, so checking /account/ is probably still the safest bet. /sam/latest_alpha/ does not generate these cookies/tokens (rightfully so)

 

[WillyJL]

im hoping to get 11.0 out within the next few days.

we're missing:
- linux wayland fixes
- if doable, macos blank webview fix
- merging a few more PRs, probably the smaller ones, others can wait for next and smaller release
- updating readme, OP, and other tooltips around the tool, to explain the new cache api system instead of previous thread parsing method
- writing the huge ass changelog

if you have any last minute urgent problems with latest beta builds, now is the time to say it.

 

[rentalunshipped982]

im hoping to get 11.0 out within the next few days.

we're missing:
- linux wayland fixes
- if doable, macos blank webview fix
- merging a few more PRs, probably the smaller ones, others can wait for next and smaller release
- writing the huge ass changelog

if you have any last minute urgent problems with latest beta builds, now is the time to say it.

Main issue I've been noticing is the sorting messing up sometimes as mentioned by others. Not urgent necessarily, but just something that happens relatively frequently.

 

[WillyJL]

ive noticed the sorting myself too, like 1 or 2 times total. i just cant find a consistent way to reproduce it so its hard to see whats happening. if you have a consistent way to reproduce it ill look into it

 

[rentalunshipped982]

ive noticed the sorting myself too, like 1 or 2 times total. i just cant find a consistent way to reproduce it so its hard to see whats happening. if you have a consistent way to reproduce it ill look into it

It mostly happens when I'm marking updated games as installed. Might have been some other occurences too, but not significant enough for me to remember.