So I'm building the release of Island SAGA right now. That said I wanted to talk about how I was handling paths on Linux to run natively, how I'm doing it now and why.
As I point out in the nerd notes of my changelog, Linux is sensitive to file name case. While there's some tricks you can play with ext4 to make the filesystem sort of case insensitive, based on my limited research the requires an option when creating the filesystem. An option I neglected to add when building my raid array. Given that I really don't want to lose a lot of data (again) that is simply not an option.
Note: There is logic that limits this behavior to Linux only so Windows users won't see this behavior at all.
What I was doing was this:
On game startup, scan the entire game directory. Create a map between the lowercase filename and the case sensitive file name. When a request for a file comes in, check to see if it exists. If it does, great use it. If the file couldn't be found convert the name to lower case and ask the map what the real name should be. If the map didn't know, just pass along the request as it was, (and typically let the game crash.)
Of course due to how RPG Maker MV encoding works, it couldn't be that simple. I had to look at file extensions and if it was one of the encoded types, switch that to the decoded extension in the map and result. I even made this work such that when not encoded, and the file on disk was a .webp, I would map the .png extension to it so play testing would just work.
While not super efficient, with SSDs and how Linux file systems work this is pretty reasonable. That was until I started working on Island SAGA.
Island SAGA is a beast of a RPG Maker MV game. Unlike a game like Summer Memories which is pretty small and contains less than 7,000 files, Island SAGA hits you with over 14,000 files and they are not small. In my development environment this balloons to 36,000+ files due to source code control and other things I need. This means the files can be spread out on disk a lot, enough to negate the benefits of a SSD/RAID setup. Initially I didn't see these problems because filesystem caching would resolve thing quickly. But if I worked on other stuff for a while and came back, the first startup would take 2-3 minutes (before the loading screen showed up.)
After thinking about the causes for a while I came to the conclusion that my inefficient scan at startup was the cause. I went back to the drawing board.
Scanning the disk every time had to go, so I needed to have the map already built on startup. Well I could scan the filesystem and build up the map on first startup, but that's not exactly a great user experience. Besides I would have to be careful if I used full paths to look and rebuild the map if the files on disk changed or the user moved where the game was. What if I just built the map to begin with, and kept the paths relative to the game root? It wouldn't handle new files very well, so mods on top of my stuff would probably have trouble but I'm already making a set of breaking changes so unless someone took my stuff and modded that, they would break anyways.
So I took the map as it existed in my development environment and wrote it to disk. Yes it wouldn't work for encoded files, but I figured I'd take it one step at a time.
This first attempt wasn't encouraging. The resulting JSON file was 14 megabytes. That's not awesome. I suppose it is all text and I could use compression. But looking at
You must be registered to see the links
drops off at about 750,000 bytes. Having a file of 14,000,000+ bytes means the lz-string might actually take longer to load than looking at the filesystem. Well, the author did say for larger items you should use LZMA, so I pulled in the
You must be registered to see the links
. And using LZMA shrunk that file from 14,299,926 bytes to 908,193 bytes which is about 6% of the original size. Experiments showed I could load this in around 800 milliseconds, (0.8 seconds) but then the result still needed to be parsed.
Looking at my file map I saw that my .git directory was being included, as well as a bunch of stuff for my builds. So I targeted the www directory exclusively. (Yes, this has a problem with the scenario folder in Dieselmine games, but that's something I could fix later.) I also started to use compact output for the JSON. Finally, I limited the map to files that have an uppercase letter in them. Coupled with lzma, this brought down the JSON archive size to 58,126 bytes for development, 23,834 bytes for production use. (The development version still has to map .webp to .png files.)
I finally had my solution. Wrap it up in a python script and you get:
Python:
#!/usr/bin/env python3
import json
import lzma
import os
from pathlib import PurePosixPath
real_path = []
# Set the directory you want to start from
rootDir = 'www'
base = PurePosixPath("/")
for dir_name, _, fileList in os.walk(rootDir):
for file_name in fileList:
real_path.append(base.joinpath(dir_name, file_name))
file_map={}
enc_file_map={}
for path in real_path:
p = PurePosixPath(path)
inputSuffix = p.suffix
if(inputSuffix == ".webp"):
inputSuffix = ".png"
p_str = str(p.parent.joinpath(str(p.stem)+inputSuffix))
p_str_lc = p_str.lower()
if p_str_lc in file_map:
print("Upper case/lower case collision with %s" % p_str)
exit(1)
else:
if(p_str_lc != p_str):
enc_file_map[p_str_lc]=p_str
file_map[p_str_lc] = str(p)
elif(p.suffix == ".webp"):
file_map[p_str_lc] = str(p)
my_filters = [
{"id": lzma.FILTER_LZMA1, "preset": 2}
]
with lzma.open("pathdata.xz", "w",format=lzma.FORMAT_ALONE, filters=my_filters) as f:
f.write(bytes(json.dumps(file_map, separators=(',', ':')), "utf-8"))
with lzma.open("pathdata.enc.xz", "w",format=lzma.FORMAT_ALONE, filters=my_filters) as f:
f.write(bytes(json.dumps(enc_file_map, separators=(',', ':')), "utf-8"))
The only issues are:
- If someone is using the Turkish language set (or other language that does this) the javascript toLower() function may result in a filename that doesn't match
- Windows won't handle the webp mapping for a playtest
- The core needs to change to fix this, so it isn't any different than before
There are more changes I want to make to Island SAGA, but I want to make sure that the existing changes are working properly first