Tool Others F95Checker [WillyJL]

[Jehutiy]

This is a different error than others are having when refreshing, but it may still be due to the API changes. I am not able to help with this one, maybe WillyJL would be able to tell if this is caused by the changes or something new. What version are you running?

Thanks for looking. You might be right. Didn't realize there were API changes.

I downloaded what I think is the latest from the OP. It should be version 10.2

 

[BobbyBuilder]

WillyJL I am sure you know all of this way better than the rest of us and I will admit, I only read a few pages back, but why cache the whole record? Why not cache just threadId and timestamp (plus any other needed fields). On a Refresh, you then pull just the timestamps from your API and compare the timestamps to build a list of threads to check on the forum. You probably won't have to change all that much of the structure if you reuse the current check code and feed it the built list rather than the full list. In the worst case a user might still run into the same issue if they don't run a check in a long time, but you can catch the 429s and print out an error message saying to wait a minute and try again. I am probably missing some nuance here, but I thought I would throw out the suggestion as the more complicated your backend, the more possible maintenance burden you would have to carry.

 

[ascsd]

i was talking about data transferred between client and cache api per each refresh, if i dropped the fast checks, and was talking about for my case with just less than 300 games. for that situation, the estimate holds up, but yeah would not scale.


still not sure when if youre talking about storage or data transfer, but lets make some estimates
for you, 500 games, 4.3mb db, for me, 280 games, 3.6mb db. each game looks to be about 10kb on average then. lets consider worst case scenario, make it 30kb of data per game.
currently, latest thread IDs are in the 200.000s, that is over operating since 2016 (or atleast thats what i suspect given that thread id 1 was posted in 2016), so 8 years of *all* threads, not just games, is 200 hundred thousand threads. worst case scenario, and also being optimistic that both the forum and this checker will live long enough to reach that point, lets consider a maximum of 1 million threads that the forum will ever contain. (again keep in mind that of those only a tiny fraction are games, and many more are also deleted or privated).
30kb per thread, maximum count of threads that will ever exist 1 million, thats 30 gigabytes of thread data. again, considering that a very small fraction of that are games and some are deleted, lets say maybe 5 gigabytes will ever need to be stored on disk by my api. base plan with my hosting provider is 70gb nvme last i checked, with the 5$ a month vps. so storage is not an issue.
now for requesting all the threads, currently 200k total threads, of which 20k are in games forum (just noticed while typing that you can see this on the homepage). again, 20k over 8 years of this forum running, so worst case scenario the most it will reach is 100k game threads. from what i tested, the rate limit seems to be about 1-2 requests per second, not exactly clear, but i tried fetching thread ids 1-200 using a ratelimiter for asyncio and it could handle it with 2 requests per second sustainedly, with up to 6 requests back to back when starting to scrape. but lets assume worst case and say 0.5 requests per second.
so considering max 100k game threads, and 0.5 requests per second with no breaks, that would be 2.3 days to scrape more game threads than the forum will ever have. way more than we will ever get as traffic, and with slower rate limiting than what the forum allows.
lets consider a more realistic scenario. theres 20k game threads, lets say that between the whole f95checker userbase theres atleast 1 user for each thread (which there probably isnt). with 1 request per second, that can all be scraped within 5.5 hours.
*but* the whole point is making this a cached api, so we dont constantly reach out to the forum. originally i had mentioned having a cache timeout of 6 or 12 hours, and in hindsight that would be too low. but lets consider the same timeout as the checker, 7 days.
when a user refreshes the games, the cache api will check when each thread id was last cached. if it was last cached less than 7 days ago, it returns the cached data, nothing hits f95zone. and its gonna be quite unlikely that everyone stops refreshing all the games for 7 days, then everyone refreshes at the same time, and the api has to reach out to f95zone for 20k games at the same time, thus getting in queue for 5 hours. what is more likely to happen is that most requests will be cached, the most popular games will be requested from the cache periodically, and every once in a while it will expire the 7 days of cache and reach out to f95zone. with 20k games and 7 days cache timeout, that means an average of 2 requests per minute to f95zone.
and again, thats assuming that there even is one person using f95checker for each of the 20k game threads that exist, and that they are all refreshed within the same 7 day window.

as for my server being able to keep up, that could be a concern.
worst case lets say the average f95checker user had 1000 games, again worst case 30kb per game, thats 30 megabytes for a full check. now, since the api is handling the periodic full checks, i would remove the option to run full rechecks from the checker interface. instead, refreshing would do a fast check, but to another api on my server, which just returns the last time that each thread id has changed data, and only if the checker sees that there is new data the full info is fetched. lets say this happens for 100 of the 1000 games each week (superficial detail changes like rating count and similar will be ignored as a "change"), means 100 (still worst case estimate) full checks to the cached api per user per week. there are 16k downloads on f95checker 10.2, 34k downloads all time on all releases. worse case there are 50k users later down the line, 50k users doing 100 full checks of 30kb each to my cache api per week, thats 600 gigabytes of traffic per month for the full checks. my hosting provider gives max 40 terabytes per month with the base 5$ vps.
and thats before cloudflare. if done right, all of this is 100% aggressively cachable with cloudflare, thats what i do for another project of mine and that also gets roughly 600 gigabytes of traffic per month, of which only 22 gigabytes reach my server.

again, all with worst cases. i feel like its gonna be fine. the only concern there could be is whether my server is able to keep up cpu wise, but with cloudflare caching on top, and considering this is just serving data as is and not generating a webpage like f95zone has to do, should be fine.


the issue here is xenforo, which is a huge piece of software that manages lots of things. for each page request it probably has to handle all your profile, settings, customizations, notifications, recommendations, any custom plugins they have, and so on. as sam said, the issue is the requests that reach xenforo (like the page we are on right now). things like the latest updates page is not part of xenforo i dont think. and the checker api sam made for f95checker also isnt under xenforo. actually, that checker api he made is similar to what im doing now. he has a php script call out to their redis instance, and redis is designed for fast cached data access, so it just grabs the version for each thread id you want and returns it. the concept i have is similar, just that i store all the parsed game data too inside redis, and i have to periodically get new data. but point is, comparing to xenforo performance issues doesnt mean much. we cant know until i have it setup and we try it, and i feel like its gonna be fine, especially with cloudflare aggressively caching it on top.

thinking about optimizations..

batch user's check requests? maybe as chunks of 50-100, as that might send back too much data in 1 request.
update request could just be a list of [ {game_id: xxx, local_updated_timestamp: xxx}, ... ]

if the game needs to be updated on cache server and its gonna take time, maybe just send back an error (or a "no changes" response) to user for that game which doesnt change the local last checked or last updated timestamps (or has a recheck after 6 hrs timestamp), so that checker will check again next time. hopefully cache is updated by then.

queue/db table? to handle multiple requests asking for a game that needs to be updated on cache server to avoid rechecking the same game multiple times while its still in progress

do not check games that last check < 1 week. so refresh button does not check server if all games checked. since check is going through the cache server, theres no point in checking again before the cache server is due to update. reduces the load on fast checking server, not sure if its worth it tho..
another methodology is server returns when next cache update will happen for each requested game. that way theres no desync between the local and cache server, and no need to even hit the cache server to ask if theres an update when previous cache hasnt expired

 

[KaosKally]

If you back read over the last week you will see this is a change from F95 that Willy is working on.

Ah thank you, so is an API change on the website that messed up how the checker is handling the account and refresh requests? At least this is what I understood the issue is. Either way I went with the b-mod version posted by another member above on Mega and seems is working fine so far, tho more extensive testing is needed to fully make sure

 

[JJHey]

Hey, the pop up told me to report the bug, am I doing it right?

 

[WhiteVanDaycare]

Ah thank you, so is an API change on the website that messed up how the checker is handling the account and refresh requests? At least this is what I understood the issue is. Either way I went with the b-mod version posted by another member above on Mega and seems is working fine so far, tho more extensive testing is needed to fully make sure

That's actually a pretty good laymen's summary.

Technically it is a change to the F95 web site itself that is throttling to prevent overload, we are seeing it with the app as it hits the site so quickly (whole point of the app). The workaround version still gets the error, it was just modified to ignore it and retry (I think after a pause) - it is the best option until Willy works out how to balance the app needs and the web site health.

 

[WillyJL]

blackop for what its worth, looks like the easiest and most proper way to ratelimit and not overwhelm f95zone's xenforo backend is this:

You must be registered to see the links

i have implemented this in my indexer branch, since many operations still use f95zone (notifications, adding watched threads...)
however, going forward this is not a good idea both for the f95zone servers and for users getting a slower refresh, so what im working on was needed anyway eventually.

im happy to announce i have finished the minimum viable implementation of the f95checker indexer/cache api (aka f95indexer) (it doesnt have a real name, it just exists lol) at api.f95checker.dev
the logic is similar to what youre all used to, with "fast" and "full" checks, but their semantics changed a bit.
f95checker will make a "fast" check to this cached api, which triggers the indexer to check if the cache for these games needs to be refreshed.
so the "fast" checks now are more of a "check in" with the f95checker api, to see if there are new updates available.
this "fast" api returns just a list of timestamps for up to 10 games requested at a time, which indicate when each game has last "changed" aka received new details (like status changing, new download links) or a new update.
f95checker will compare this timestamp with the last recoded one for each game, and if the last changed time is newer than what it had recorded, it will then fetch the new data with a "full" check.
"full" checks are *heavily* cached on cloudflare to avoid pointless traffic from hitting my server, thats also why the timestamps, so it can retrieve the correct cached data.
i decided to keep the "fast" step because that means less traffic from the end user's machine to cloudflare/me per each refresh (having only "full" checks would mean an average of 20kb per game to download, with 1000 games thats already 20 megabytes, its not manageable with just "full" checks)

what does this mean for you?
refreshing will look very different. first of all, it hopefully will be much faster, and leave f95zone alone from the shitstorm of requests we were causing before.
next, this means that now "fast" checks will check if cache needs to be udpated, and update it too. so, if the cache for a game needs to be updated, it could end up being quite slow. the idea is that over time everything gets cached, and on average everything is faster.
to illustrate when f95checker is waiting for f95indexer to cache a thread, it will show this in the bottom right corner.

there is a build available on github actions for you to try, here's hoping my server doesnt crap out.
MAKE SURE TO BACKUP YOUR db.sqlite3 DATABASE IN F59CHECKER CONFIG FOLDER

 

[WillyJL]

This is a different error than others are having when refreshing, but it may still be due to the API changes. I am not able to help with this one, maybe WillyJL would be able to tell if this is caused by the changes or something new. What version are you running?

its all related, the program is not getting data how it expects to it triggers random errors

 

[WillyJL]

WillyJL I am sure you know all of this way better than the rest of us and I will admit, I only read a few pages back, but why cache the whole record? Why not cache just threadId and timestamp (plus any other needed fields). On a Refresh, you then pull just the timestamps from your API and compare the timestamps to build a list of threads to check on the forum. You probably won't have to change all that much of the structure if you reuse the current check code and feed it the built list rather than the full list. In the worst case a user might still run into the same issue if they don't run a check in a long time, but you can catch the 429s and print out an error message saying to wait a minute and try again. I am probably missing some nuance here, but I thought I would throw out the suggestion as the more complicated your backend, the more possible maintenance burden you would have to carry.

youre mistaken. thats what it already does basically. the problem is that getting the information for any thread from the forum is now very slow. the hope here is that if only my api does it, none of the users, it will cause less issues to the forum, and be faster for users. also, im not caching "the whole record", im jus tcaching the data that the checker would have otherwise needed to connect to the forum for. and im not sure your proposed solution would work either, based on what would the timestamps on my server update? my server would still need to check in with the forum... so might aswell cache the whole thing, 2 birds with 1 stone

 

[WillyJL]

thinking about optimizations..

batch user's check requests? maybe as chunks of 50-100, as that might send back too much data in 1 request.
update request could just be a list of [ {game_id: xxx, local_updated_timestamp: xxx}, ... ]

if the game needs to be updated on cache server and its gonna take time, maybe just send back an error (or a "no changes" response) to user for that game which doesnt change the local last checked or last updated timestamps (or has a recheck after 6 hrs timestamp), so that checker will check again next time. hopefully cache is updated by then.

queue/db table? to handle multiple requests asking for a game that needs to be updated on cache server to avoid rechecking the same game multiple times while its still in progress

do not check games that last check < 1 week. so refresh button does not check server if all games checked. since check is going through the cache server, theres no point in checking again before the cache server is due to update. reduces the load on fast checking server, not sure if its worth it tho..
another methodology is server returns when next cache update will happen for each requested game. that way theres no desync between the local and cache server, and no need to even hit the cache server to ask if theres an update when previous cache hasnt expired

im having trouble following what youre saying, im too tired xD but from what it sounds like, its similar to what i actually did implement. read above ^^

 

[effninefivehuman]

Hello, I am still seeing the rate limiting message when I am importing new watched threads

I am using the latest beta 1221 version

I have reduced the number of workers to 1, retries to 1, timeout is 30s

The async task count changes from 1 to 2 when the import is running

 

[WillyJL]

Hello, I am still seeing the rate limiting message when I am importing new watched threads

I am using the latest beta 1221 version

I have reduced the number of workers to 1, retries to 1, timeout is 30s

The async task count changes from 1 to 2 when the import is running

View attachment 4237751

View attachment 4237754

Hm, annoying, I'll look into it soon

 

[WillyJL]

WhiteVanDaycare is that you refreshing on the indexer beta? I've been seeing requests coming through on the server non stop for a solid 20 minutes or something lol
If it is you, would be cool to know an estimate of how long it took you overall when it's done, just to have an estimate for having all cache misses

 

[Danv]

will be proxy pull added to the main at some point?
kinda damn crucial, the only reason i have to use (and will keep using) modded version, cuz otherwise checker half working for me, with whatever broken in attachment handling (it worked just fine on version 8.5 without need for any proxy)

 

[WillyJL]

will be proxy pull added to the main at some point?
kinda damn crucial, the only reason i have to use (and will keep using) modded version, cuz otherwise checker half working for me, with whatever broken in attachment handling (it worked just fine on version 8.5 without need for any proxy)

Literally read my messages above yours

 

[Danv]

Literally read my messages above yours

i'm not seeing anything related to what i've asked? unless your new cache thingy will also store and handle game pics?

 

[WillyJL]

i'm not seeing anything related to what i've asked? unless your new cache thingy will also store and handle game pics?

I'm sorry, between modded version and basically broken it sounded like the same broken record of the last few days. I have no clue what issue with images you mean, need more context. As for proxy if you mean the pr that is open, I plan to get to it once the actual serious problem we have right now is dealt with fully.

 

[Danv]

I'm sorry, between modded version and basically broken it sounded like the same broken record of the last few days. I have no clue what issue with images you mean, need more context. As for proxy if you mean the pr that is open, I plan to get to it once the actual serious problem we have right now is dealt with fully.

well, long story short - checker of version 10+ was never able to download game pics for me, throwing generic connection error

Spoiler

You don't have permission to view the spoiler content. Log in or register now.

so i have to download and update pics manually (every time new game is added or existing had pic change)
version 8.5 was working fine, it wasn't change of my ISP or paranoid AV (don't have any anymore), but upgrade to new version 10+
using proxy in modded version actually fixes this, while using global proxy in OS - doesn't
as of note - i'm using f95 directly without any VPNs or proxy, so not sure why accessing attachments from checker (of new versions) suddenly require one
so yeah, not even asking to look into it - kinda gave up already, will just wait for proxy merge, just wanted to check if it happening at all

 

[Sam]

well, long story short - checker of version 10+ was never able to download game pics for me, throwing generic connection error

Spoiler

You don't have permission to view the spoiler content. Log in or register now.

so i have to download and update pics manually (every time new game is added or existing had pic change)
version 8.5 was working fine, it wasn't change of my ISP or paranoid AV (don't have any anymore), but upgrade to new version 10+
using proxy in modded version actually fixes this, while using global proxy in OS - doesn't
as of note - i'm using f95 directly without any VPNs or proxy, so not sure why accessing attachments from checker (of new versions) suddenly require one
so yeah, not even asking to look into it - kinda gave up already, will just wait for proxy merge, just wanted to check if it happening at all

Your ISP is blocking the attachments subdomain. If you use a properly configured VPN that tunnels all traffic, the application should work.

 

[Danv]

Your ISP is blocking the attachments subdomain.

then wouldn't attachments be broken for me period, not just in this app, but in general?