- Oct 1, 2020
- 1,523
- 1,050
Oooohh, finally one I got in my library too.
Yup, and the thread is still there too...
Yup, and the thread is still there too...
- Mar 7, 2019
- 1,408
- 1,205
does this happen with all games? when selecting a new exe, it will look inside of your default exe dirs to try to match the game for which you are adding the exe, it looks for type, developer and name. so if you have a full structure it would show exedir/category/developer/game/ (eg exedir/RenPy/Inceton/Lust Theory), but this is all modular, if it doesnt find something that looks like that structure it matches less like just exedir/category/game or exedir/developer/game or just exedir/game .... it works by fuzzy matching and only accepts with more than 85% match, or if the word it is looking for is exactly in the name and no other folder has it in its name.
need more context to understand whats happening, like what games you tried this with, what their types and developers are, what the exact folder path was (eg if you have other stuff in folder name) and so on
- Sep 3, 2018
- 300
- 595
- Mar 23, 2021
- 159
- 139
Yeah, it seems to be a run away condition (walking pace though), another just came up https://f95zone.to/threads/detective-mikami-final-asobi.240115/
- Mar 7, 2019
- 1,408
- 1,205
- Oct 1, 2020
- 1,523
- 1,050
Quick check, what does it say at the bottom of the tooltip if you hover the questionmark behind
Set exe dir: in the Manage section of the sidebar?You don't need to post the path, just if it is the same as the mentioned base directory?
- Mar 7, 2019
- 1,408
- 1,205
so, i checked logs, there were some errors:
- some generic connection errors i hadnt accounted for, they retried after 6 hours now i made them retry after 15 mins like they should
- ddos-guard being a bitch 2 times in total, added its new message to list of temporary errors that retry after 15 mins
- 1 php error on latest updates, told sam about it as it seems worth taking a look at just to be sure
- 1 unexpected error fetching a thread, added to list of errors to retry after 15 mins
but, all of those were caught even if not handled optimally (they will be now if they happen again), and none of those are what caused the games that "arent refreshing".
the games that "arent refreshing" dont have anything weird in logs (if there was an error, it wouldve told end users too actually). how it detects these is if the thread page returns a 403 or a 404, so it actually did receive such a response for those games.
i checked with the user account the api uses to fetch threads, they are all visible to it, so yeah it was something temporary.
i have 2 theories:
- mods privated the thread temporarily, it was refreshed at that time, then opened it back up
- maybe ddos-guard sends 403 when it wants to be a bitch?
the only other 2 instances of ddos-guard *that i see in logs* are on 2024-12-29 which i feel is about when y'all started reporting the missing threads, and it was in the latest updates watcher that it was caught, which doesnt have any 403/404 logic. when fetching game threads, it first checks for known error strings (ratelimit, f95zone unavailable, login required...) and then checks for 403/404, so it never got to actually try to use the received data to parse the thread and just assumed it was gone.
i have now added the new ddos-guard message string to the prior check, so it should be able to detect it and just retry in 15 minutes instead of marking it as missing for 14 days.
also changed how it detects missing threads:
- checks known error strings
- tries to parse the thread
- if cant find OP in response, checks if response is from f95zone (looks for
- if cant find OP, can find f95zone logo, and status is 403 or 404, marks as missing with no log messages and retries in 14 days
- if cant find OP and cant find f95zone logo, gives a parsing error (gets logged on my end and sent to users too so someone notices) and retries in 6 hours
in the case of these missing threads, i assume it would now fall at the first step checking for known error strings, and if in the future ddos-guard decides to change the error strings again it will fall into the parsing error situation instead of silently marking as missing.
still, i find it weird that ddos-guard would return 403...
if it indeed is ddos-guard (as odd as that would be to give a 403) then would also be useful to see if sam can add an exception for my server's ip, but before that i want to see if these changes catch the issue now
EDIT: actually,
- some generic connection errors i hadnt accounted for, they retried after 6 hours now i made them retry after 15 mins like they should
- ddos-guard being a bitch 2 times in total, added its new message to list of temporary errors that retry after 15 mins
- 1 php error on latest updates, told sam about it as it seems worth taking a look at just to be sure
- 1 unexpected error fetching a thread, added to list of errors to retry after 15 mins
but, all of those were caught even if not handled optimally (they will be now if they happen again), and none of those are what caused the games that "arent refreshing".
the games that "arent refreshing" dont have anything weird in logs (if there was an error, it wouldve told end users too actually). how it detects these is if the thread page returns a 403 or a 404, so it actually did receive such a response for those games.
i checked with the user account the api uses to fetch threads, they are all visible to it, so yeah it was something temporary.
i have 2 theories:
- mods privated the thread temporarily, it was refreshed at that time, then opened it back up
- maybe ddos-guard sends 403 when it wants to be a bitch?
the only other 2 instances of ddos-guard *that i see in logs* are on 2024-12-29 which i feel is about when y'all started reporting the missing threads, and it was in the latest updates watcher that it was caught, which doesnt have any 403/404 logic. when fetching game threads, it first checks for known error strings (ratelimit, f95zone unavailable, login required...) and then checks for 403/404, so it never got to actually try to use the received data to parse the thread and just assumed it was gone.
i have now added the new ddos-guard message string to the prior check, so it should be able to detect it and just retry in 15 minutes instead of marking it as missing for 14 days.
also changed how it detects missing threads:
- checks known error strings
- tries to parse the thread
- if cant find OP in response, checks if response is from f95zone (looks for
.p-header-logo img[alt="F95zone"])- if cant find OP, can find f95zone logo, and status is 403 or 404, marks as missing with no log messages and retries in 14 days
- if cant find OP and cant find f95zone logo, gives a parsing error (gets logged on my end and sent to users too so someone notices) and retries in 6 hours
in the case of these missing threads, i assume it would now fall at the first step checking for known error strings, and if in the future ddos-guard decides to change the error strings again it will fall into the parsing error situation instead of silently marking as missing.
still, i find it weird that ddos-guard would return 403...
You must be registered to see the links
this states that 403 is only returned for block lists, tor nodes and geoblocking, and i doubt any of these is the case... if it were, no games at all would be refreshing...? so weird.if it indeed is ddos-guard (as odd as that would be to give a 403) then would also be useful to see if sam can add an exception for my server's ip, but before that i want to see if these changes catch the issue now
EDIT: actually,
You must be registered to see the links
this here shows a ddos-guard browser verification that returned 403, so it really does look like its whats happening. the 2 ddos-guard errors that i did see in my logs had a different text content but seems reasonable that different ddos-guard pages with similar purpose would give the same response code. also, that issue 5 years old so also possible they change the look of the page (would also explain why the previous error string i had for ddos-guard didnt match, it had changed slightly with different capitalization)
Last edited:
- Oct 1, 2020
- 1,523
- 1,050
Assuming that whatever you did would remove the issue, I just tried refreshing Welcome to Free Will again. It still complains so hopefully it turns up something in your logs.
- Mar 7, 2019
- 1,408
- 1,205
i hadnt invalidated the cache yet. i updated the code so no new instances should appear.
i have now invalidated cache for threads 1190 240115 85755 30658 236807 184252 16887 228420 85156 81854. i dont have an easy way to check which ones were affected besides user input, as it wasnt detected as an error like i explained. let me know if theres more that were affected. but it shouldnt happen to *new* instances again, only talking about removing from those that were affected previously.
- Oct 1, 2020
- 1,523
- 1,050
I can only comment for 85755 (Welcome to Free Will), which of course now refreshes fine.
I tried routing the traffic through Fiddler to see if the headers would give any indication that would make the checker conclude the game couldn't be found, but it didn't like the certificate Fiddler uses to decrypt https connections. Also it would probably not reveal anything since it most likely happens during the connection between your api server and the forum.
- Mar 7, 2019
- 1,408
- 1,205
no need to use an external program to do this, you can just use chrome/firefox devtools (ctrl+shift+i), in nextwork tab refresh the page and you can see all requests, their headers, and payload
and yeah i doubt there would be something like that anyway. it not finding the OP but detecting that it is an f95zone webpage, and having 403 or 404 response code should be good enough
- Oct 1, 2020
- 1,523
- 1,050
That's the weird part. When the WtFW thread turned up 'missing' in the checker during a refresh, it loaded just fine when using the thread button.
Also, that's apples and oranges... refresh queries the api server, the thread button goes directly to f95.
Well, it did, didn't it? It told us the thread was 'missing'...
It would be more useful to monitor the requests between the api server and F95. Routed through Fiddler (=debugging proxy) you would be able to see directly what headers get returned. And if that doesn't reveal anything, using the decryption option would let you view the payload. Plus using its scripting option, you could let it filter out anything that doesn't return a 200 result.
But, like I said, that would only reveal something if used as a MITM between the api server and F95zone. It's that my collection is too limited, otherwise I'd be inclined to setup a local cache to see what happens, but the chance of hitting a thread that goes 'missing' is way too small on my side. WtFW was the first and only one so far. Aside from that, I don't think Sam would be too pleased if another party started scraping the games forum for its entirety.
Actually, using Fiddler isn't even necessary, you've got the code that makes up the cache, so it could be enhanced do the same thing. Coz somewhere, somehow it got convinced those threads went 'missing', all that's needed is to find out why.
Last edited:
- Oct 1, 2020
- 1,523
- 1,050
WillyJL
Something else you might want to take a look at. Looks like download mirrors don't get synced with the actual thread.
Example: 16 Years Later!
Forum:
F95Checker:
When the first update notification came through, the thread only had a Mega link for the Windows version, and the DDL didn't even have a Windows version. Had to report it to get it added to the DDL and to have them add more Windows mirrors.
But the Download tab didn't sync the WIndows mirrors after doing a recheck. It even had those listed when the thread still only had the Windows Mega link.
Something else you might want to take a look at. Looks like download mirrors don't get synced with the actual thread.
Example: 16 Years Later!
Forum:
F95Checker:
When the first update notification came through, the thread only had a Mega link for the Windows version, and the DDL didn't even have a Windows version. Had to report it to get it added to the DDL and to have them add more Windows mirrors.
But the Download tab didn't sync the WIndows mirrors after doing a recheck. It even had those listed when the thread still only had the Windows Mega link.
i didnt find this error when i searched... but this popped up this morning:
{
"INDEX_ERROR": "PARSING_FAILED"
}
{
"INDEX_ERROR": "PARSING_FAILED"
}
- Aug 28, 2023
- 7
- 12
Hello
same here : when doing full refresh :
{
"INDEX_ERROR": "PARSING_FAILED"
}
when manually updating one game no issue
Thank
same here : when doing full refresh :
{
"INDEX_ERROR": "PARSING_FAILED"
}
when manually updating one game no issue
Thank
- Mar 7, 2019
- 1,408
- 1,205
That was not in logs. And that was not something that I could use to gauge which ones are affected. It was the same effective result as any thread that actually doesn't exist, of which there are 800 thousand (I reject requests for ids above 1 million, currently largest thread Id is under 250k). I didn't look as again it would be futile, but I'm sure there's a decent chunk of non-existent thread ids cached that never ever existed, just because when an api is exposed to the internet people will fuck with it, and there's bound to be someone that has used the api to check bogus thread ids that never even existed on the forum. So just invalidating cache for threads that my cache api had cached as missing would do it to a lot more threads than just those affected.
which I already did, so idk why we're still on this lol
There's no need for fiddler here, and headers would be a less reliable way to detect whatever is happening. It was a logic issue on my end, now the logic issue is fixed, and if the actual underlying issue persists (as we see from people below your post) now I will see the logs for it
- Mar 7, 2019
- 1,408
- 1,205
As always, I can't fetch all threads always from f95zone. I don't have any way to know if download links changed besides version number changes and latest updates posts changing (though they do not detect download link changes, only information changes). If downloads links change and nothing else, I have no way to detect it. And as we already know, I can't make it fetch all threads from f95 when any user refreshes it because that would defeat the whole point of a cache api. So, it will refreshed next in 1 week when cache is invalidated (or 2 days if the game version is not tracked by latest updates so it invalidates cache more often).
DDL goes directly to f95zone, it's not cached on my end. If the issue also happened with DDL, this sounds to me like something actually went wrong with that thread on f95zone end itself.