- Mar 7, 2019
- 1,318
- 1,121
Correct. Only reason to login will be to check notifications and to import watched threads and such. Because of this I plan to make notifications checking disabled by default, so the tool can work without any setup
- Oct 1, 2017
- 201
- 148
I systematically click check thread on everything im following, just to check the image previews of the games.
Is there a plan to add the previews available to view through the app instead of having to fully open each thread individually?
i get constantly an error:
thread parsing error
failed to parse necessary sections in thread response, the html file has been saved to........
thread parsing error
failed to parse necessary sections in thread response, the html file has been saved to........
- Dec 19, 2020
- 440
- 1,549
Not sure about anyone else but when I get these errors that's apparently related to the API or something since it pops up immediately after hitting refresh, I just hit ok and hit refresh again and it seems to work every time I do that, lol. Perhaps there's other issues on top if people can't refresh at all?
i did the same but after a few times i couldn´t refresh anymore and only got this failure message
- Mar 7, 2019
- 1,318
- 1,121
thats what a 429 too many requests error is. you are going too fast and need to slow down. between it erroring and you clicking refresh again, the timer has expired.
- Oct 1, 2020
- 1,303
- 885
Aren't those already included in the fast_check? i.e. the data returned by the existing api?
Last edited:
- Mar 7, 2019
- 1,318
- 1,121
im talking about monitoring it on my indexer api. so that when an update happens, it fetches it and updates the cache.
also, i might just get rid of fast checks completely. the whole point was to avoid stress on the f95zone servers, but if im making a dedicated api to cache this stuff, we can just get all the game details directly from the f95indexer api i am making. ill have to check how much data that would mean gets transferred each time tho. just for a rough idea, i have ~280 entries in my checker, and the sqlite databse is 3.6 megabytes, which includes 6000+ timeline entries. i'd imagine the size of the actual data for all of those games' data from the api would be something like 2-3 megabytes. not too much tbh. but also not super efficient.
maybe ill end up making 2 endpoints on this api, and one will act kinda like the current checker api sam setup but instead of giving version numbers it would give the last time a thread's data was changed (not just checked, but actually changed), so actually the checker wouldnt even need to do periodic full checks, thats handled by the api, and the checker can just check which threads have new data.
- Oct 1, 2020
- 1,303
- 885
to give you an idea...
~500 games
~17.500 timeline entries
db size 4.3 MB
since there's a shit ton of games on F95, RenPy alone roughly estimated 7500 (actual threadcount probably somewhat lower accounting for collection threads) and rough guestimate ~10000 combined in the other engines...
And that's not even counting the mods for those games, which some also will have added to their checker.
I fear your 2-3 MB is a way too low guestimate... even if only storing the most basic version/last modified data.
Actual counts will probably come around to half of that if accounting for completed/abandoned stuf, which doesn't need checking that often, but some might still get updated. So, they'll probably need get checked too on a lower frequency
You sure you're that's manageable? What you're talking about comes down to collecting version/last updated data for the whole of the games and mods forums. That's an awful lot of data. And with the current rate limit, and all of us checking against the result, you'll be looking at it accessing the forum almost non-stop from a single connection...
Another thing, if the forum already crumbles under the load of us doing full checks... are you sure you'll be able to handle the same type of load? Keeping in mind that it will only get higher the more users start using the checker and the more games need to get checked. Because what it effectively comes down to is just moving the load from one server to another.
It would probably be a lot more manageable if F95 would have the ability to do an automatic push every time the OP on a thread gets updated. This way you don't even need to query the forum but get the data delivered to you.
Last edited:
- Mar 7, 2019
- 1,318
- 1,121
i was talking about data transferred between client and cache api per each refresh, if i dropped the fast checks, and was talking about for my case with just less than 300 games. for that situation, the estimate holds up, but yeah would not scale.
still not sure when if youre talking about storage or data transfer, but lets make some estimates
for you, 500 games, 4.3mb db, for me, 280 games, 3.6mb db. each game looks to be about 10kb on average then. lets consider worst case scenario, make it 30kb of data per game.
currently, latest thread IDs are in the 200.000s, that is over operating since 2016 (or atleast thats what i suspect given that thread id 1 was posted in 2016), so 8 years of *all* threads, not just games, is 200 hundred thousand threads. worst case scenario, and also being optimistic that both the forum and this checker will live long enough to reach that point, lets consider a maximum of 1 million threads that the forum will ever contain. (again keep in mind that of those only a tiny fraction are games, and many more are also deleted or privated).
30kb per thread, maximum count of threads that will ever exist 1 million, thats 30 gigabytes of thread data. again, considering that a very small fraction of that are games and some are deleted, lets say maybe 5 gigabytes will ever need to be stored on disk by my api. base plan with my hosting provider is 70gb nvme last i checked, with the 5$ a month vps. so storage is not an issue.
now for requesting all the threads, currently 200k total threads, of which 20k are in games forum (just noticed while typing that you can see this on the homepage). again, 20k over 8 years of this forum running, so worst case scenario the most it will reach is 100k game threads. from what i tested, the rate limit seems to be about 1-2 requests per second, not exactly clear, but i tried fetching thread ids 1-200 using a ratelimiter for asyncio and it could handle it with 2 requests per second sustainedly, with up to 6 requests back to back when starting to scrape. but lets assume worst case and say 0.5 requests per second.
so considering max 100k game threads, and 0.5 requests per second with no breaks, that would be 2.3 days to scrape more game threads than the forum will ever have. way more than we will ever get as traffic, and with slower rate limiting than what the forum allows.
lets consider a more realistic scenario. theres 20k game threads, lets say that between the whole f95checker userbase theres atleast 1 user for each thread (which there probably isnt). with 1 request per second, that can all be scraped within 5.5 hours.
*but* the whole point is making this a cached api, so we dont constantly reach out to the forum. originally i had mentioned having a cache timeout of 6 or 12 hours, and in hindsight that would be too low. but lets consider the same timeout as the checker, 7 days.
when a user refreshes the games, the cache api will check when each thread id was last cached. if it was last cached less than 7 days ago, it returns the cached data, nothing hits f95zone. and its gonna be quite unlikely that everyone stops refreshing all the games for 7 days, then everyone refreshes at the same time, and the api has to reach out to f95zone for 20k games at the same time, thus getting in queue for 5 hours. what is more likely to happen is that most requests will be cached, the most popular games will be requested from the cache periodically, and every once in a while it will expire the 7 days of cache and reach out to f95zone. with 20k games and 7 days cache timeout, that means an average of 2 requests per minute to f95zone.
and again, thats assuming that there even is one person using f95checker for each of the 20k game threads that exist, and that they are all refreshed within the same 7 day window.
as for my server being able to keep up, that could be a concern.
worst case lets say the average f95checker user had 1000 games, again worst case 30kb per game, thats 30 megabytes for a full check. now, since the api is handling the periodic full checks, i would remove the option to run full rechecks from the checker interface. instead, refreshing would do a fast check, but to another api on my server, which just returns the last time that each thread id has changed data, and only if the checker sees that there is new data the full info is fetched. lets say this happens for 100 of the 1000 games each week (superficial detail changes like rating count and similar will be ignored as a "change"), means 100 (still worst case estimate) full checks to the cached api per user per week. there are 16k downloads on f95checker 10.2, 34k downloads all time on all releases. worse case there are 50k users later down the line, 50k users doing 100 full checks of 30kb each to my cache api per week, thats 600 gigabytes of traffic per month for the full checks. my hosting provider gives max 40 terabytes per month with the base 5$ vps.
and thats before cloudflare. if done right, all of this is 100% aggressively cachable with cloudflare, thats what i do for another project of mine and that also gets roughly 600 gigabytes of traffic per month, of which only 22 gigabytes reach my server.
again, all with worst cases. i feel like its gonna be fine. the only concern there could be is whether my server is able to keep up cpu wise, but with cloudflare caching on top, and considering this is just serving data as is and not generating a webpage like f95zone has to do, should be fine.
the issue here is xenforo, which is a huge piece of software that manages lots of things. for each page request it probably has to handle all your profile, settings, customizations, notifications, recommendations, any custom plugins they have, and so on. as sam said, the issue is the requests that reach xenforo (like the page we are on right now). things like the latest updates page is not part of xenforo i dont think. and the checker api sam made for f95checker also isnt under xenforo. actually, that checker api he made is similar to what im doing now. he has a php script call out to their redis instance, and redis is designed for fast cached data access, so it just grabs the version for each thread id you want and returns it. the concept i have is similar, just that i store all the parsed game data too inside redis, and i have to periodically get new data. but point is, comparing to xenforo performance issues doesnt mean much. we cant know until i have it setup and we try it, and i feel like its gonna be fine, especially with cloudflare aggressively caching it on top.
- Mar 15, 2020
- 1,843
- 1,801
Take a breath.
Sounds pretty good. And that's a lot of work. Can't wait for the implementation. (I mean I can).
- Mar 7, 2019
- 1,318
- 1,121
to be fair, im glad FaceCrap got my attention on this, i hadnt quite realized the scale i'm dealing with here until doing the math for that reply xD but atleast from the numbers it sounds plausible, so thats a reliefTake a breath.
Sounds pretty good. And that's a lot of work. Can't wait for the implementation. (I mean I can).
- Oct 1, 2020
- 1,303
- 885
WillyJL Apart from the refresh checks, I assume the 'search' from the bottom bar for any threads/games not yet in the checker will then also be handled by your api?
EDIT: Since I saw a mention about not needing to be logged in anymore, that would mean any forum related activity wouldn't be possible... that I would have a bit of an issue with, it would block checks for alerts, searches for games from same dev (F95 has a specific link for that)
EDIT: Since I saw a mention about not needing to be logged in anymore, that would mean any forum related activity wouldn't be possible... that I would have a bit of an issue with, it would block checks for alerts, searches for games from same dev (F95 has a specific link for that)
Last edited:
- Mar 7, 2019
- 1,318
- 1,121
No, quick search uses a different endpoint. Will still require login and reach out to the forum. Same for alerts. It's not a white or black situation, there's many shades of gray in between, in this case only refreshing your threads will use the cached api, since that's what causes issues. Everything else stays as is
- Dec 21, 2019
- 163
- 175
I got something similar but regarding my account for some reason... Either way, I cleared cache and relogin, using mouse to login, not enter and seems it works somewhat.... And i spoke too soon.... error again...What's going on?I need a hero!
- Mar 23, 2021
- 127
- 111
What's going on?I need a hero!
If you back read over the last week you will see this is a change from F95 that Willy is working on.
- Mar 23, 2021
- 127
- 111
I have found that resetting my IP (I use a VPN so it is literally just a single click for me) and restarting the full check gets me by. It runs in the background all the time on my computer so this keeps it fairly current (and I track over 2500 games). Just an option for those that don't want to install the fork by blackop that suppresses the errors.
Sorry if this has been brought up before. I tried to search the error but it seems like other people had something slightly different.
This is my first time using the tool and I tried to import using the "watched threads" since I have quite a few games watched but it keeps throwing this error when I refresh.
Anyone have any ideas on what to try?
This is my first time using the tool and I tried to import using the "watched threads" since I have quite a few games watched but it keeps throwing this error when I refresh.
Anyone have any ideas on what to try?
- Mar 23, 2021
- 127
- 111
This is a different error than others are having when refreshing, but it may still be due to the API changes. I am not able to help with this one, maybe WillyJL would be able to tell if this is caused by the changes or something new. What version are you running?