Others Java New Game Manager/Thread Formatting

[Nox29]

First of all, this is my very fist time posting in any forum, so I am sorry if I´m posting in the wrong section or do something else wrong.

To give a little background information, I am currently planning to develop a new Game Manager or rather a Manager for all sorts of NSFW content. To achieve this they way I envisioned it, I need to get information from all sorts of websites like for example pornhub, hanime and of course f95zone. Sadly, most if not all of said websites do not provide a public API and since I only know how to code in Java, i can´t use the unofficial API´s f95zone already has.

In other words, I am trying to make my own Java API for f95zone.
I´m already able to extract basic information like the title, version and so on from any thread, but I have trouble with the download section. I want my manager to feature automatic downloads and thus need to access the links inside the download section every thread has.

My problem is, I have no idea how I should determine which download link leads to which file. While some games for example only provide a single download link for the game, others like many renpy games for example provide download links for different parts of the game or standalone seasons.
Any idea how I can determine based on the formatting of each thread, which donwload link is for what?

Sorry if anything is unlcear, if I failed to explain something properly please ask me about it, thanks in advance.

 

[InnocentLurker]

...Wait what are you trying to make?

To be honest it sounds like Java isn't the language you want to go for. Something like Python would have more packages for interacting with the web, as well as being easier for other users to read.

 

[_retorik_]

If you can code in Java you can code in most languages. 95% of programming languages is the same stuff.

The most innovative thing in programming languages over the last 30 years is python indentation.

 

[Alcahest]

Any existing APIs won't help you with this on this site since the info on the game threads is added straight up to the webpage. The site doesn't know what is what more than you do looking at the page.

Game opening posts are setup in the same way, more or less, based on the uploading rules (look at the template preview): https://f95zone.to/threads/game-uploading-rules-2019-06-25.524/.

For most games, you can know what the links are for if you look at the typical ways they are setup even when they have different releases, episodes and so on. But since the formatting is not hardcoded, you can never be 100% sure you can cover all games.

 

[Nox29]

...Wait what are you trying to make?

To be honest it sounds like Java isn't the language you want to go for. Something like Python would have more packages for interacting with the web, as well as being easier for other users to read.

Firstly, thank you for your fast reply.

To keep it simple, for now I want to make an application which basicaly works like steam just for f95zone.
I know there are a few game managers out there which kind of do that, but as far as I know none of those supports integrated/ automated downloads or updates.

While I know Python would be a good choice for this as well, my problem would actualy be the same if I used it.
In detail, my problem is that not all threads follow exactly the same formatting rules. I can already extraced pretty much every information i need from a thread, including all download links, but I need some logic to determine if a link leads to a main download, a standalone season, a standalone event build and so on.

I could hardcode it, but optimaly the program should react dynamicaly so it recognizes every thread.
So basicaly what I´m asking for is an algorithm (in pseudo code/normal text) to do that.

I hope that clarifies my goal.

 

[Nox29]

Any existing APIs won't help you with this on this site since the info on the game threads is added straight up to the webpage. The site doesn't know what is what more than you do looking at the page.

Game opening posts are setup in the same way, more or less, based on the uploading rules (look at the template preview): https://f95zone.to/threads/game-uploading-rules-2019-06-25.524/.

For most games, you can know what the links are for if you look at the typical ways they are setup even when they have different releases, episodes and so on. But since the formatting is not hardcoded, you can never be 100% sure you can cover all games.

Thank you for your rely.

As for the API, I figured as much so I wanted t make an API (or rather web-scraper) myself providing me the information from a thread I need.

I actualy looked at the template preview and it´s pretty consistent as far as I know, only the downloads section on most threads differs a little.

Do you think I should just look for the links where I´m sure they exist and then provide every other link as sorts of a recommendation, so the user can chose themselves what the link is?

Another idea I had was that I could make it possible to download files directly from my application and then save which link was used (not the url, more like the position in the thread), so that the same link is used for updating the game.

 

[Nox29]

If you can code in Java you can code in most languages. 95% of programming languages is the same stuff.

The most innovative thing in programming languages over the last 30 years is python indentation.

I would propably be able to code in python pretty easily, what I am unsure of is frameworks (or what they are called in python). My project currently is relying heavyly on javafx and spring, with selenium for browser automation. Is the same possible in python?

 

[Nox29]

hi

And here I was, happy for a reply xd...

 

[Alcahest]

Thank you for your rely.

As for the API, I figured as much so I wanted t make an API (or rather web-scraper) myself providing me the information from a thread I need.

I actualy looked at the template preview and it´s pretty consistent as far as I know, only the downloads section on most threads differs a little.

Do you think I should just look for the links where I´m sure they exist and then provide every other link as sorts of a recommendation, so the user can chose themselves what the link is?

Another idea I had was that I could make it possible to download files directly from my application and then save which link was used (not the url, more like the position in the thread), so that the same link is used for updating the game.

I have no opinion on how you should do it. I doubt there is a good way to do it, and then it might not be worth doing.

 

[InnocentLurker]

To keep it simple, for now I want to make an application which basicaly works like steam just for f95zone.
I know there are a few game managers out there which kind of do that, but as far as I know none of those supports integrated/ automated downloads or updates.

What's the pornhub integration you mentioned then?

Is this for managing games you already have to download new updates, or for searching content?

To be honest it sounds like a very high-scope idea for something you can already get out of bookmarks.

I could hardcode it, but optimaly the program should react dynamicaly so it recognizes every thread.
So basicaly what I´m asking for is an algorithm (in pseudo code/normal text) to do that.

What I would do is, if you can already access the download links in a thread, get the version name of the last installed version and compare it to the current version, then prompt the user to update when the current version's number is larger.

That assumes you're making an update manager.
If not I don't know how you'd compete with the existing f95 search alpha, to be honest.

My project currently is relying heavyly on javafx and spring, with selenium for browser automation. Is the same possible in python?

Selenium is supported in Python. Not sure that's the tool you'd want to use since it's designed to emulate human interactions not to web scrape, but I'm not familiar with any alternatives.

 

[Nox29]

I have no opinion on how you should do it. I doubt there is a good way to do it, and then it might not be worth doing.

Thank you for your help still, if I figure something out I will give updates.

 

[Nox29]

What's the pornhub integration you mentioned then?

Is this for managing games you already have to download new updates, or for searching content?

To be honest it sounds like a very high-scope idea for something you can already get out of bookmarks.


What I would do is, if you can already access the download links in a thread, get the version name of the last installed version and compare it to the current version, then prompt the user to update when the current version's number is larger.

That assumes you're making an update manager.
If not I don't know how you'd compete with the existing f95 search alpha, to be honest.


Selenium is supported in Python. Not sure that's the tool you'd want to use since it's designed to emulate human interactions not to web scrape, but I'm not familiar with any alternatives.

As for the pornhub thing, my overall goal is to make an application which bundles all sorts of nsfw content like games, videos and comics from all sorts of websites and present it in a organized way(using tags, watch/playlists of sorts and so on).
However I have to start somewhere, so for now I am only bothering with f95zone content.
In short, all porn in one place.

As for the game part of the application, yes the main focus is updating games you already own.
Checking if a game can be updated and getting all download links form the thread is easy, but like I said above due to the incosistent formatting of thread on this website I can´t determine for sure which link leads to a main download, a walkthrough, a patch or whatever content is provided in the download section.

1. example:

DOWNLOAD
Win: MEGA - ANONFILES - GOFILE - MIXDROP - UPLOADHEAVEN

2. example:

DOWNLOAD

Season 4
Win/Linux: ANONFILE - MEGA - MIXDROP - UPLOADHAVEN - PIXELDRAIN
Mac: ANONFILE - MEGA - MIXDROP - UPLOADHAVEN - PIXELDRAIN
Android: ANONFILE - MEGA - MIXDROP - UPLOADHAVEN - PIXELDRAIN
Other: COMPRESSED

Season 3
SPOILER

Season 1-2 + Special Episodes
SPOILER

Extras: Fan Signature - Fan Made Walkthrough - Multi-mod - Gallery Mod

As you can see, the formatting is inconsistent.
Things like "Win:" and "Win/Lin:" are rather easy to deal with, because their variations are limited, however having different seasons like in the 2. example is dynamic and thus hard to predict.

As for why I use Selenium, to mimic interactions on the website like pressing buttons or clicking links. You could propably do the same without it, but I´m new to the whole webscraping thing and for now It´s the most intuitive way for me to get stuff from the f95zone since I pretty much just tell it to do what I normaly do on f95zone.
Aside from that, I use it for easy browser intergration so you can just use the f95 search insite the application.

 

[InnocentLurker]

As you can see, the formatting is inconsistent.

This might just be outside the scope of something you could ask someone to solve for you on a forum to be honest.

Trying to solve this dynamically would require finding patterns in the different formats... I'm personally not willing to open a hundred links to find a consistent method.
That's what I'd suggest you do next though, if you want my advice.

 

[Nox29]

This might just be outside the scope of something you could ask someone to solve for you on a forum to be honest.

Trying to solve this dynamically would require finding patterns in the different formats... I'm personally not willing to open a hundred links to find a consistent method.
That's what I'd suggest you do next though, if you want my advice.

To be honest I was mostly wondering if the formatting is completely random, which it seems to be. For now I will simply only take the Donwload links in the line directly under the "DOWNLOADS" text and assume its the link to the latest version. All other links will be optional.

Thank you for you help and sorry for bothering!

 

[_retorik_]

I just scrolled over your messages but if I understood you are cloning google for porn ...

I would go with Javascript and libs like cheerio. If you are doing this to learn how to do it go ahead ... you will learn a lot. But if you have another focus you might be trying to chew a lot more than you can take.

 

[Nox29]

That description seems to be accurate.

As for why I do it, yes mostly to lear from it. I learn faster and more efficient if I can practice on something I somewhat care about. Also, aside from the webscraping I develop something similar in Java for my job, so it´s practice as well.

Aside from that I am annoyed about my pc being so messy, so I wanted a proper way to sort thing most of all.

 

[Nox29]

Quick update, I actualy managed to extract Links in a somewhat usable way now.
While I didn´t manage to associate many lines to more than one main download (yet!), if there aren´t multiple seasons or something like that my solution already works.

So basicaly everythink that looks like this works:

DOWNLOAD
Win/Linux: ANONFILE - GOFILE- MEGA - MIXDROP
Mac: ANONFILE - GOFILE- MEGA - MIXDROP
Android (v0.06): ANONFILE - MEGA - NOPY - UP2SHARE - ZIPPYSHARE
Other: ANDROID*

Extras: Italian Translation* - Spanish Translation*

 

[Nox29]

This might just be outside the scope of something you could ask someone to solve for you on a forum to be honest.

Trying to solve this dynamically would require finding patterns in the different formats... I'm personally not willing to open a hundred links to find a consistent method.
That's what I'd suggest you do next though, if you want my advice.

A late update.
After some trying around I found that you don´t realy need a detailed pattern.
Using the Regex "[^:]*:\\s[^-]*((\\s-\\s[^-]*)+)?" I can identify all lines with links in them (like " Win/Linux: ANONFILE - GOFILE- MEGA - MIXDROP " for example).

 

[Nox29]

I think I´m making some good progress with the API for this website.
For now I´m running it on my localhost and I can already get any normal attribute from a thread like name, author, version and so on using HHTP requests.
As far as I understand, anyone should be able to use the API with any programming language because of that.

 

[_retorik_]

If I understand, yes. If you serve and expose that api somewhere anyone would be able to query with any language ... or using tools