Tool Others F95Checker [WillyJL]

[WillyJL]

Assuming that whatever you did would remove the issue, I just tried refreshing

Welcome to Free Will again. It still complains.

i hadnt invalidated the cache yet. i updated the code so no new instances should appear.

i have now invalidated cache for threads 1190 240115 85755 30658 236807 184252 16887 228420 85156 81854. i dont have an easy way to check which ones were affected besides user input, as it wasnt detected as an error like i explained. let me know if theres more that were affected. but it shouldnt happen to *new* instances again, only talking about removing from those that were affected previously.

 

[FaceCrap]

i have now invalidated cache for threads 1190 240115 85755 30658 236807 184252 16887 228420 85156 81854.

I can only comment for 85755 (Welcome to Free Will), which of course now refreshes fine.

I tried routing the traffic through Fiddler to see if the headers would give any indication that would make the checker conclude the game couldn't be found, but it didn't like the certificate Fiddler uses to decrypt https connections. Also it would probably not reveal anything since it most likely happens during the connection between your api server and the forum.

 

[WillyJL]

I can only comment for 85755 (Welcome to Free Will), which of course now refreshes fine.

I tried routing the traffic through Fiddler to see if the headers would give any indication that would make the checker conclude the game couldn't be found, but it didn't like the certificate Fiddler uses to decrypt https connections.

no need to use an external program to do this, you can just use chrome/firefox devtools (ctrl+shift+i), in nextwork tab refresh the page and you can see all requests, their headers, and payload
and yeah i doubt there would be something like that anyway. it not finding the OP but detecting that it is an f95zone webpage, and having 403 or 404 response code should be good enough

 

[FaceCrap]

no need to use an external program to do this, you can just use chrome/firefox devtools (ctrl+shift+i), in nextwork tab refresh the page and you can see all requests, their headers, and payload
and yeah i doubt there would be something like that anyway. it not finding the OP but detecting that it is an f95zone webpage, and having 403 or 404 response code should be good enough

That's the weird part. When the WtFW thread turned up 'missing' in the checker during a refresh, it loaded just fine when using the thread button.
Also, that's apples and oranges... refresh queries the api server, the thread button goes directly to f95.

the games that "arent refreshing" dont have anything weird in logs (if there was an error, it wouldve told end users too actually)

Well, it did, didn't it? It told us the thread was 'missing'...

It would be more useful to monitor the requests between the api server and F95. Routed through Fiddler (=debugging proxy) you would be able to see directly what headers get returned. And if that doesn't reveal anything, using the decryption option would let you view the payload. Plus using its scripting option, you could let it filter out anything that doesn't return a 200 result.

But, like I said, that would only reveal something if used as a MITM between the api server and F95zone. It's that my collection is too limited, otherwise I'd be inclined to setup a local cache to see what happens, but the chance of hitting a thread that goes 'missing' is way too small on my side. WtFW was the first and only one so far. Aside from that, I don't think Sam would be too pleased if another party started scraping the games forum for its entirety.

Actually, using Fiddler isn't even necessary, you've got the code that makes up the cache, so it could be enhanced do the same thing. Coz somewhere, somehow it got convinced those threads went 'missing', all that's needed is to find out why.

 

[FaceCrap]

WillyJL
Something else you might want to take a look at. Looks like download mirrors don't get synced with the actual thread.
Example: 16 Years Later!
Forum:


F95Checker:


When the first update notification came through, the thread only had a Mega link for the Windows version, and the DDL didn't even have a Windows version. Had to report it to get it added to the DDL and to have them add more Windows mirrors.
But the Download tab didn't sync the WIndows mirrors after doing a recheck. It even had those listed when the thread still only had the Windows Mega link.

 

[doctorx6969]

i didnt find this error when i searched... but this popped up this morning:
{
"INDEX_ERROR": "PARSING_FAILED"
}

 

[ipaqi]

WillyJL Any chance the server is down?

 

[simi00]

i didnt find this error when i searched... but this popped up this morning:
{
"INDEX_ERROR": "PARSING_FAILED"
}

Yep, I have the same error

 

[Baalzie]

Ditto like the rest today. Parsing failed error...

 

[triste_lune]

Hello

same here : when doing full refresh :
{
"INDEX_ERROR": "PARSING_FAILED"
}

when manually updating one game no issue

Thank

 

[WillyJL]

Well, it did, didn't it? It told us the thread was 'missing'..

That was not in logs. And that was not something that I could use to gauge which ones are affected. It was the same effective result as any thread that actually doesn't exist, of which there are 800 thousand (I reject requests for ids above 1 million, currently largest thread Id is under 250k). I didn't look as again it would be futile, but I'm sure there's a decent chunk of non-existent thread ids cached that never ever existed, just because when an api is exposed to the internet people will fuck with it, and there's bound to be someone that has used the api to check bogus thread ids that never even existed on the forum. So just invalidating cache for threads that my cache api had cached as missing would do it to a lot more threads than just those affected.

Actually, using Fiddler isn't even necessary, you've got the code that makes up the cache, so it could be enhanced do the same thing. Coz somewhere, somehow it got convinced those threads went 'missing', all that's needed is to find out why.

which I already did, so idk why we're still on this lol
There's no need for fiddler here, and headers would be a less reliable way to detect whatever is happening. It was a logic issue on my end, now the logic issue is fixed, and if the actual underlying issue persists (as we see from people below your post) now I will see the logs for it

 

[WillyJL]

WillyJL
Something else you might want to take a look at. Looks like download mirrors don't get synced with the actual thread.
Example: 16 Years Later!
Forum:
View attachment 4433962

F95Checker:
View attachment 4433963

When the first update notification came through, the thread only had a Mega link for the Windows version, and the DDL didn't even have a Windows version. Had to report it to get it added to the DDL and to have them add more Windows mirrors.
But the Download tab didn't sync the WIndows mirrors after doing a recheck. It even had those listed when the thread still only had the Windows Mega link.

As always, I can't fetch all threads always from f95zone. I don't have any way to know if download links changed besides version number changes and latest updates posts changing (though they do not detect download link changes, only information changes). If downloads links change and nothing else, I have no way to detect it. And as we already know, I can't make it fetch all threads from f95 when any user refreshes it because that would defeat the whole point of a cache api. So, it will refreshed next in 1 week when cache is invalidated (or 2 days if the game version is not tracked by latest updates so it invalidates cache more often).
DDL goes directly to f95zone, it's not cached on my end. If the issue also happened with DDL, this sounds to me like something actually went wrong with that thread on f95zone end itself.

 

[WillyJL]

i didnt find this error when i searched... but this popped up this morning:
{
"INDEX_ERROR": "PARSING_FAILED"
}

Yep, I have the same error
View attachment 4434199

Ditto like the rest today. Parsing failed error...

Hello

same here : when doing full refresh :
{
"INDEX_ERROR": "PARSING_FAILED"
}

when manually updating one game no issue

Thank

So the change I made yesterday did fix the logic issue as I expected, now it notices there is an issue, instead of just marking the thread as missing. I will check the logs when I'm in pc in a few hours.

 

[WillyJL]

WillyJL Any chance the server is down?

No. I heard f95zone itself was down for a while, maybe that's what you mean
If you mean the error that the others posted, I will look at it in logs in a few hours

 

[FaceCrap]

As always, I can't fetch all threads always from f95zone.

I understand, but that means the 'Full Refresh' option no longer works like before. Because when it went directly, that would have been picked up.
So you don't store the datetime data for when a thread was last queried?
Otherwise you could compare it to the date value in the json result to see if it's been modified since.
Also, this was an actual update, wouldn't it had to have the thread parsed in that case anyways instead of relying on the expiration cycle?

DDL goes directly to f95zone, it's not cached on my end. If the issue also happened with DDL, this sounds to me like something actually went wrong with that thread on f95zone end itself.

That was just extraneous info. Some mod probably forgot to add the mirrors. It's what lead to noticing the mirror difference

11.0.2 beta 1437
I found a couple of bugs.

confirmed, same here. But the tab does get removed though.

 

[simi00]

When I check one by one, then it works, but error shows up on game ID 74065 ( https://f95zone.to/threads/the-hard-way-v0-38-1-muffin-maker.74065/ ). Other games (checked on 50 games) works. I have around 400 games in list.

Other game with parse error:
https://f95zone.to/threads/the-innocent-mother-ch-2-spies.4211/

 

[WillyJL]

and if the actual underlying issue persists (as we see from people below your post) now I will see the logs for it

So the change I made yesterday did fix the logic issue as I expected, now it notices there is an issue, instead of just marking the thread as missing. I will check the logs when I'm in pc in a few hours.

doctorx6969 simi00 Baalzie triste_lune turns out that this error in particular was due to f95zone database downtime, not to the previous issue with threads missing. it didnt receive valid data from f95zone so of course parsing wasnt successful. it would have attempted again in 6 hours, i fixed it now (5 hours after it happened). if a databse error like this ever happens again, it will be caught as a F95ZONE_UNAVAILABLE error instead and retried after 15 minutes.
i have invalidated cache for all the affected threads (3983 3959 136922 151469 79941 1132 1224 5541 76200 785 2796 6996 1297 135918 42937 47958 10703 4431 126890 20812 210467 202 727 26392 30383 4211 132123 214170 74065), should be fine now

 

[WillyJL]

When I check one by one, then it works, but error shows up on game ID 74065 ( https://f95zone.to/threads/the-hard-way-v0-38-1-muffin-maker.74065/ ). Other games (checked on 50 games) works. I have around 400 games in list.

Other game with parse error:
https://f95zone.to/threads/the-innocent-mother-ch-2-spies.4211/

when you get an actual error reported like this (saying the f95checker cache api returned an error) it means it knows there was an issue and logged it for me to inspect, no need to investigate further, just notify me
it was different last week with the missing threads, those went unnoticed (considered thread as missing, but not an api error), and needed info from users to diagnose

 

[WillyJL]

I understand, but that means the 'Full Refresh' option no longer works like before. Because when it went directly, that would have been picked up.
So you don't store the datetime data for when a thread was last queried?
Otherwise you could compare it to the date value in the json result to see if it's been modified since.

of course it doesnt work like before, basically nothing works like before, it *cannot* work like before.
your f95checker client never talks to f95zone directly when it comes to getting thread info.
as stated in the github readme, now the full refresh means syncing data from the cache api. it doesnt force the cache api to recache the thread. if it did, we would be back at square 1: everyone forcing a full refresh and blasting f95zone till we DDoS it.

i do store the timestamp for last time a game was refreshed, on both sides:
- your client records the last time the thread data changed on the cache api. it works kinda like a "version" of cached data. any time the cache api updates the cached data, it saves the timestamp, then when your client refreshes it checks all the timestamps, if the cache api's cached data timestamp is different than what your client has saved, it syncs the whole thread info from the cache api.
- and on the cache api, i record when each thread was last fetched from f95zone, when the last time any of the contained data actually changed (this is to notify client that they should sync the new data, and avoids syncing if after cache expiration the new data is the exact same), and the time when cache is due to expire (not all threads have same cache expiry, like some errors can have as little as 15 minutes of cache, and non-existent threads are cached for 14 days)

Also, this was an actual update, wouldn't it had to have the thread parsed in that case anyways instead of relying on the expiration cycle?

yes. and it did. the issue is that mods pushed the update, someone refreshed the thread in f95checker, it was cached, and only *after* the downloads were changed. at that point, nothing changed that the cache api could detect.
its always human error. i cannot fix the 0.01%. sorry.

 

[simi00]

doctorx6969 simi00 Baalzie triste_lune turns out that this error in particular was due to f95zone database downtime, not to the previous issue with threads missing. it didnt receive valid data from f95zone so of course parsing wasnt successful. it would have attempted again in 6 hours, i fixed it now (5 hours after it happened). if a databse error like this ever happens again, it will be caught as a F95ZONE_UNAVAILABLE error instead and retried after 15 minutes.
i have invalidated cache for all the affected threads (3983 3959 136922 151469 79941 1132 1224 5541 76200 785 2796 6996 1297 135918 42937 47958 10703 4431 126890 20812 210467 202 727 26392 30383 4211 132123 214170 74065), should be fine now

Thank you. It works.