Tool Others F95Checker [WillyJL]

[WillyJL]

I have around 400 games and I'm also having trouble with the app being very slow and getting random errors. If I set threads to 80 will it still check every game ?

of course
ill give you an example
you have 5 students and each has done their exam and they need to be checked
if you have 5 people correcting the exams, then each person will check one exam and they all work at the same time, so it takes less
if instead you have 1 person correcting, then he will have to check one at a time and then move to the next and so on, so it takes longer
in your case you have 400 exams to correct, but having 400 professors in the same class correcting the exams is a bit hard, so they struggle
same concept here, you have 400 games to check, if you have more threads to check them they work more at the same time, but if you have too many threads your computer struggles

in my situation i started having issues past 100 games, and to be honest i set my threads to 20 and it works flawlessly. it might be my shitty internet connection, but 20 threads for me works at the same speed as 60 or 08 and so on... you just need to try it and see what works best for you

i wil fix the behavior of the threads setting in the future when i get to rewriting this tool, how it works right now is completely stupid. i will probably put 20 threads by default, but still allow it to be manually edited

 

[WillyJL]

Weird bug, for some reason i'm getting a few games with the wrong game_id:

normally it's just the thread number, but some are "to/threads/#####", which means the image isn't found.

Just editing the .json file to remove the to/threads/ part fixes it.

E: also if you use a random page to add the game it's not stripped out of the thread url, so you end up with threads/blurb.####/page-123

E2: it's a trivial fix for both of them. I don't have a github account that i'm willing to link to my f95 name with but I'll give you the diff here

diff --git a/modules/callbacks.py b/modules/callbacks.py
index 36731da..00327cb 100644
--- a/modules/callbacks.py
+++ b/modules/callbacks.py
@@ -79,7 +79,15 @@ async def add_game(*kw):
             globals.gui.add_input.setFocus()
             return

+    # if we are given a link to a page deeper in the thread trim it off
+    trim = link.rfind('/page-')
+    if trim != -1:
+        link = link[:trim+1]
     game_id = link[link.rfind('.')+1:link.rfind('/')]
+    # if the URL is in the form f95zone.to/threads/#####/
+    # we will get the wrong game_id, so clean it up.  No need to test,
+    # if the text isn't found nothing will change.
+    game_id = game_id.replace('to/threads/','')

     # Config
     if game_id in globals.config["games"]:

huh, i never encountered this issue... well when i get to rewriting the tool i will implement a better behavior

 

[WillyJL]

Confirmed. Seems like the forum's new DDOS-GUARD is to blame.

A related-ish project Yet Another Manager rolled it's own

You must be registered to see the links

library and is looking to bypass this as well. A contributor to their repo found

You must be registered to see the links

which seems promising. Maybe something to investigate?

Another possible solution would be to bundle the b binaries for

You must be registered to see the links

(a bypass proxy) which supports Clioudflare and DDOS-GUARD as of v2.2.0. I am testing FlareSolverr in a VM and will report back.

Thanks for all of your work to make F95Checker an amazing app!

we'll have to see... in this tool's case actually i think there isnt even the need to bypass it, i think i kinda wrote retarded code for the auth... i will get around to fix everything eventually

 

[HelpfulSunglasses]

huh, i never encountered this issue... well when i get to rewriting the tool i will implement a better behavior

well, in the meantime the patch cleans up the URLs and avoids some weird errors with missing images or not finding the changelog (because it's not looking at the first page of the thread).

The URL format https://f95zone.to/threads/##### comes if you add a link directly from the latest updates page, they all link to just the threadid without the slug. When I was looking up my games to add the fastest search was the title filter there, which is why I ended up with a bunch of broken game_id fields.

 

[forgetmaster]

Is there a way to see where are the names of all the games I added are saved?

 

[unroot]

Is there a way to see where are the names of all the games I added are saved?

these names are stored in the f95checker.json file inside a location of your user-folder

windows
C;/Users/YOURUSERNAME/AppData/Roaming/f95checker
(Would have been correct as of Win7)

linux
~/.f95checker

macos
~/Library/Application Support/f95checker
(Can't verify, never had a Mac)

# All locations are taken from the f95checker.py-file

 

[WillyJL]

A bit of an update on things:
TL;DR:
Much improvement, such innovation.

Technical details:
I’m looking into switching from Qt to imgui for the interface, cos while Qt has all the bells and whistles and runs on all platforms, I just don’t like it’s code structure (I mean how you are supposed to write interfaces for it). ImGui (immediate mode gui) is much more intuitive and easier to work with in my opinions, even though it has less features. The main difference is that Qt is imperative, meaning you create all the widgets once at the beginning and then update them later, while imgui is immediate, meaning that it runs in a loop and renders the interface each frame, running all the code for the interface on each frame. Might seem resource intensive but not really, and it’s much easier to work with for this project’s design (each row with a game is a custom Qt widget, if I were to add custom columns support, I’d have to edit all the widgets one by one to enable the column, while with imgui it runs each frame and it’s a simple if check to decide if it’s shown). Also drag and drop looks much easier with imgui, I’m planning to have normal sort options, and then a custom sort option where you can drag and drop them around, and it remembers if you switch to another sort in the meantime. However imgui is a sync library, while this project makes heavy use of async structure (aiohttp is much better than requests, I’m not giving that up). For this problem I discovered that it is possible to run a main sync render loop for imgui on the main thread and an asyncio loop on a second thread, and share variables and async tasks effortlessly between them. Also the config file is a nightmare in the current state, I’m considering (and very leaning towards) storing game data in a SQLite database, while keeping other common settings in json format, but we’ll have to see, might even switch completely to database. Finally this major rewrite will of course implement most, if not all, of the planned features on the OP and will also try to fix the login madness, I’ll just login with a browser window like now but then store all cookies from it, and purge all cookies if the login expires.

 

[batblue]

I’m looking into switching from Qt to imgui for the interface

Nice! This looks super promising for the Mac, since parts of Qt5 don't compile on the new M1 processors, while imgui supports both M1 and Intel. I'm looking forward to playing with this in the Mac port if/when you integrate it.

 

[WillyJL]

Nice! This looks super promising for the Mac, since parts of Qt5 don't compile on the new M1 processors, while imgui supports both M1 and Intel. I'm looking forward to playing with this in the Mac port if/when you integrate it.

now that i think more about it i might need to still bundle some parts of qt, or find some alternatives, because while imgui is very nice for the main gui, it does not feature anything else, unlike qt which features a systemtray icon module and a webview engine... for tray i of course found pystray, but im having a very hard time to get it to work on linux, it starts just fine without errors but the icon is nowhere to be found... i am now attempting with wxPython, and that should also feature a webview widget of some kind, but we'll have to see

 

[oldshoe]

I just wanted to say a big thanks. I know nothing about code or anything tech but this program is a serious lifesaver to me, I would be seriously scared to lose it as I use it at least once a day.

 

[batblue]

now that i think more about it i might need to still bundle some parts of qt, or find some alternatives, because while imgui is very nice for the main gui, it does not feature anything else, unlike qt which features a systemtray icon module and a webview engine...

Well, the main issue for M1 Macs is QtWebKit. Nobody seems to have got it working on M1, although the WebKit in Qt6 is supposed to work. If there's a way to render web pages without using QtWebKit, or using Qt6, that would go a long way for Mac compatibility.

Of course, you can run F95Checker on M1 Macs using the i386 emulation layer, but it's a huge pain in the ass.

 

[WillyJL]

Still a lot to go but it's starting to take shape

I'm loving ImGui. This update will bring togglable columns, a fuckton more columns, more sort options and new filtering options, collapsible things, and a grid view similar to the latest updates page, among many other fixes and enhancements. Also the code will be actually maintainable from now on. Hopefully the config issues will be fixed for good since I'm moving everything to SQLite and it seems to be working so far. If anyone wants to check out the progress I am working on the

You must be registered to see the links

, but BEWARE it is not functional yet. At the moment it is simply a few pieces of the puzzle held together with duct tape, it is not yet a usable program!

Edit: of course that is just the default style of ImGui. I will try my best to give it the 8.x F95Checker look, and I know it's possible since ImGui is so flexible

Edit 2: also regarding the tray icon and the webview widget: the best solution would have been wxPython, but alas it looks like they too render using GLFW and it would crash my program when I initialized GLFW for ImGui. In the end I went with PyQt6 for the tray icon for now, and probably I'll also use that for the webview login widget when I get to it. Turns out it was quite easy to integrate into the ImGUi render loop, simply adding a qt_loop.processEvents() call on every iteration did the trick

 

[WillyJL]

Progress coming in steadily
again it still looks ugly, this is just to showcase some upcoming functionality
View attachment Record_select-area_20220410233244.mp4

 

[Dukez]

Is this new program which I assume is a remake, going to carry over our lists or do we need to re-add them all when it's done

 

[HelpfulSunglasses]

Is the new "last played" field a date or a version? Because it'd be nice to keep track of the last played version alongside the current. Maybe I don't care about a minor patch or I'd like to check the changelogs to see what all has been added since then.

Interface looking good.

 

[WillyJL]

Is this new program which I assume is a remake, going to carry over our lists or do we need to re-add them all when it's done

F95Checker is by no means a popular program, but the userbase is large enough that discarding old data would really hurt it and discourage many people from migrating. Also it's not how I do things, just consider that I'm still keeping it backwards compatible all the way to v6.0 (might be even earlier than that, it's hard to keep track of when I rewrote the config from the changelog notes...)! Of course all the missing new data like engine, developer, description (crazy to think we didn't have this before huh?) and tags will need to be fetched from the website, but that will happen automatically on the first refresh!

Also there is currently an annoying behavior caused by the design of the refresh process: refreshing only checks if the thread URL's changed, so that will cover title changes most of the time, but it won't cover the status changes. Consider a game that has no status, but then one day it is marked as abandoned... the version didn't change, and the title too didn't change... so the URL too is the same and F95Checker will not fetch all the game details, therefore still thinking it has no status, while in reality is has been abandoned. To combat this the new update will periodically do a "full refresh" of games that haven't been "fully refreshed" in a while. It will probably be a editable option how often this happens, and by default it might be maybe 2 days? Also please someone help me come up with a better name, because I feel like distinguishing between "refresh" and "full refresh" might be a hard task to explain...

Is the new "last played" field a date or a version? Because it'd be nice to keep track of the last played version alongside the current. Maybe I don't care about a minor patch or I'd like to check the changelogs to see what all has been added since then.

Last played is a date, just like last updated and added on. The main idea was to use it as a sort option so you can go back to games you haven't played in a while (someone suggested that some time ago and I perfectly agree).
However what you are suggesting will be a feature in the next update, just not for played. Consider this:

# initial state
"Game v1.0                             Played: yes  Installed: yes"

# update to v1.1
"Game Installed: v1.0 Latest: v1.1     Played: no   Installed: no"

# this is an irrelevant update, you mark as played
"Game Installed: v1.0 Latest: v1.1     Played: yes  Installed: no"

# update to v2.0
"Game Installed: v1.0 Latest: v2.0     Played: no   Installed: no"

# this is relevant, so you install
"Game v2.0                             Played: no   Installed: yes"

# you finish playing the update
"Game v2.0                             Played: yes  Installed: yes"

 

[WillyJL]

game info popups and settings section
View attachment Record_select-area_20220414040152.mp4

images load kinda slow because imgui doesnt like them very much, i need to convert the image data to an array of rgba pixels for it to work and the bigger the image the longer it takes...
i divided the settings into sections for tidiness and added help markers for not so intuitive settings or features. do let me know how you feel about this new layout!

 

[Dukez]

game info popups and settings section
View attachment 1755678

images load kinda slow because imgui doesnt like them very much, i need to convert the image data to an array of rgba pixels for it to work and the bigger the image the longer it takes...
i divided the settings into sections for tidiness and added help markers for not so intuitive settings or features. do let me know how you feel about this new layout!

Not sure if easier or harder, but instead of a popup what about a drop down? Like you click the row and that info from the pop up gets shown in a submenu below in the same list. Perhaps changelog etc would be made into submenus inside that, but would keep it all within the one window basically.

 

[WillyJL]

Not sure if easier or harder, but instead of a popup what about a drop down? Like you click the row and that info from the pop up gets shown in a submenu below in the same list. Perhaps changelog etc would be made into submenus inside that, but would keep it all within the one window basically.

Well imgui is not really a traditional GUI toolkit by any means. How it works is that you make your GUI statements and they get rendered into a single image, and that image gets applied to a regular window. Almost like a game would do. In fact imgui runs every frame and it has a framerate. So that popup is not a separate window, it’s inside the same window, but it appears as an overlay. I get your proposal but I feel like that it is too much information to fit into the main list, and if we cut out changelog there is very few information left that is not visible in the main view... also personally I feel like a popup works better, like some sort of character sheet but for games. And finally when I also implement the grid view I’m gonna need a popup anyway since your proposal wouldn’t really work in that situation and would create confusion IMO

 

[ascsd]

Thats a really long time to render an image of that size...
For a quick hack, you could emulate what interlaced png does and load a small thumbnail while the original image is being converted. Having a 2 images shouldnt be too much of a problem since the size would be small.
I'm hoping this hack would bring back the image on the refresh button on hover because I sometimes can't remember the name of a the games for the life of me and clicking through each would be a nightmare

edit:
just installed the latest dev build to try to profile the code and see the bottleneck.. and the images are loading instantly on windows