Tool Offline advanced F95 tag search tool (Console application)

[xj47]

I found the tag-searching options on this site a bit lacking so, having too much time on my hands, I created my own search tool that uses an offline database of game data collected from the site.

The tool allows you to:
- Exclude tags
- Use OR operators
- Use brackets to form more complex search criteria

For example you could run the query "female-protagonist, 2dcg, !ntr, !no-sexual-content, (bdsm / slave), (ren'py / rpgm), completed" to get all games that:
- Have the female-protagonist & 2dcg tags
- Don't have the ntr or no-sexual-content tags
- Have either the bdsm or slave tag
- Is either a renpy or RPGM game
- Is marked as completed

(for the purposes of the tool "prefixes" like "completed" and "rpgm" are treated like normal tags)

In more technical terms, you have a first-order-logic statement where the AND, OR, NOT operators are: , ! /
(and the whole statement is evaluated left-to-right with no operator priority)

That said, this is a console application so you need a bit of technical knowledge to use it and it's not super user-friendly.
The tool is really just a python script alongside a (sqlite) database file.
I've converted it to an executable for windows users, but if you aren't using Windows or just want to run the script directly you do that as well. Just need to have python 2.7 installed and the sqlite3 library.

----

How to actual use the tool:
You run the program with the search query you want and get a list of results where for every game you have: The F95 game-thread title, the url to the thread, the full list of tags for that game-thread. If you include "-plain" to the end of your query you just get the list of urls.

I'd recommend messing around with your search query until the result list looks about right. Then add the -plain option and write the program results to a .txt file (Ex: search female-protagonist, !ntr, rpgm -plain > results.txt). Then open the .txt file with a text editor like Notepad and copy the urls into your browser. Google Chrome has a nice extension called "TabList" that lets you open a list of urls at once which is very handy for this.

I admit, it's a bit cumbersome but better than nothing.

Changelog:

March 2nd, 2019
- Updated game database. Most recent entry is now

You must be registered to see the links

- Changed how search query is processed to use operation priority instead of a simple left-to-right evaluation. A query like "bdsm, rpgm / !lesbian, ren'py" now evaluates as "(bdsm, rpgm) / ((!lesbian) , ren'py)"

 

[Air Space Frame]

Tool is nice and easy to use. Any chance of a database update? Or even modifying the script to be able to update and adding instructions on how users can manually update it?

 

[xj47]

Tool is nice and easy to use. Any chance of a database update? Or even modifying the script to be able to update and adding instructions on how users can manually update it?

Thanks. I mainly made it for personal use but it's nice to hear that there are other people using it. I've uploaded a new version with an updated database and improved method of processing search queries.

I've been using a web scraper to update the database. Since web-scraping can cause an excessive load on the servers I talked with some people from the site and said I wouldn't make those parts of the script public.
Honestly I don't think enough people use this tool for it to be an issue but whatever. I said I wouldn't so I wont.

 

[Air Space Frame]

Thanks. I mainly made it for personal use but it's nice to hear that there are other people using it. I've uploaded a new version with an updated database and improved method of processing search queries.

I've been using a web scraper to update the database. Since web-scraping can cause an excessive load on the servers I talked with some people from the site and said I wouldn't make those parts of the script public.
Honestly I don't think enough people use this tool for it to be an issue but whatever. I said I wouldn't so I wont.

Thanks for the update. It would be much appreciated if, when you do your personal periodic updates to the database, occasionally update this thread as well. (and write a post saying "updated", since the system doesn't seem to send an alert otherwise). Or maybe throw something up on a github, etc. As long as the instructions are clear (like how it is now) it shouldn't be too difficult for non-programmers like me to use.

It's a shame the site doesn't have better tag searching features implemented yet.

 

[Kaakman]

It doesn't work for me. I try typing search -help but it just says the command search doesn't exist

 

[xj47]

It doesn't work for me. I try typing search -help but it just says the command search doesn't exist

Confirm that you're in the correct folder before running the command. Typing "dir" in the command line will list the files in your current directory which should include a file called "search.exe"
Depending on how your computer is setup you may also need to run your commands as "search.exe <stuff>" instead of "search <stuff>"

 

[Kaakman]

Confirm that you're in the correct folder before running the command. Typing "dir" in the command line will list the files in your current directory which should include a file called "search.exe"
Depending on how your computer is setup you may also need to run your commands as "search.exe <stuff>" instead of "search <stuff>"

Oh i figured it out, the shift click trick opens powershell for me. Using cmd and cd'ing to the dir works.

 

[MyGirlsNowYoink]

Damn is this application dead/abandoned? was looking through Tools n Toturial for something like this and guess i'm to late now.

 

[xj47]

the internal database is kind of outdated at this point but I could refresh it.
Main issue is that threads do get updated over time so unless I re-scrape everything a lot of the old entries will be inaccurate (might say v0.1 when the game is actually done)

I think the command-line interface is too awkward for most people so it didn't see much use.

 

[MyGirlsNowYoink]

the internal database is kind of outdated at this point but I could refresh it.
Main issue is that threads do get updated over time so unless I re-scrape everything a lot of the old entries will be inaccurate (might say v0.1 when the game is actually done)

I think the command-line interface is too awkward for most people so it didn't see much use.

Is there no way for people to re scrape it themselves? rather than leave the burden on you? to do it every time threads update?

 

[xj47]

Is there no way for people to re scrape it themselves? rather than leave the burden on you? to do it every time threads update?

There is but I deliberately did not upload the scraping scripts as a courtesy to the site. Web scraping does strain the server a bit and if unexperienced people are using it they may run it more often then necessary.

 

[MyGirlsNowYoink]

I see