Ren'Py How do I get RenPy to compile a single .rpy file while loading a game?

[TrooperJustinHare]

Good day! I'm making a mod for a RenPy 7.4.11 game (and I'll answer beforehand that this game has Steam workshop support, and the author allows creating any modifications for his game). The mod itself is a tool to auto initialize resources of the mod (i.e. images, audio, etc) while loading the game (yes, built-in RenPy autoinit won't do.).
Long story short, I just discovered that if, instead of running a script to search for and initialize files, you create a file with the file paths already set with the help of a script, the game will load much faster than with the first autoinit option. But there is a problem: when creating this .rpy file in the script, in order for RenPy to compile it, you need to restart the game.

def process_files(self):
    """
    Processes the mod files.
    If the write_into_file value is True, it writes the mod's resources to a separate file rather than initializing them. To further initialize the mod's resources from the file, you need to restart the game.
    """
    if self.write_into_file:
        with builtins.open(self.modPath + "/assets.rpy", "w") as log_file:
            log_file.write("init python:\n    ")
            for type, file_name, file in self.modFiles:
                if type == "sound":
                    log_file.write("%s = \"%s\"\n    " % (file_name, file))
                elif type == "image":
                    log_file.write("renpy.image(\"%s\", \"%s\")\n    " % (file_name, file))
                if type == "sprite":
                    log_file.write("renpy.image(\"%s\", %s)\n    " % (file_name, file))
    else:
        for type, file_name, file in self.modFiles:
            if type == "sound":
                globals()[file_name] = file
            elif type == "image":
                renpy.image(file_name, file)
            if type == "sprite":
                renpy.image(file_name, eval(file))

I did some digging in the RenPy documentation and found that RenPy supports the CLI and can pass the "compile" argument to it, but this won't work since it's designed for a folder (like a game folder), not just a single file, and also create a bunch of junk files inside of it after compiling.
So, is there any way to compile a single .rpy file?

 

[anne O'nymous]

    if self.write_into_file:
[...]
    else:
        for type, file_name, file in self.modFiles:
            if type == "sound":
                globals()[file_name] = file
            elif type == "image":
                renpy.image(file_name, file)
            if type == "sprite":
                renpy.image(file_name, eval(file))

*sigh*

init -1 python:
    for type, file_name, file in self.modFiles:
        if type == "sound":
            setattr( store, file_name, file )
        elif type == "image":
            renpy.image( file_name, file )
        if type == "sprite":
            renpy.image( file_name, eval( file ) )

Problem solved, next question?


Edit: changed the init level (there's in fact no need to be so early) and added the missing eval

 

[TrooperJustinHare]

Thank you, but in what way does this fix my problem?
My problem is that if my autoinit is in "write_into_file" mode (self.write_into_file), then after creating the .rpy file with the files, it will not compile after creation, but will only compile after restarting the game.
Also, I should have made this clear, but there is a reason to use "eval" because I am initializing a sprite that is a Composite() with a ConditionSwitch, not just a single image. To initialize it, I create a big line of code and then use eval to have RenPy handle it.

Spoiler: Sprite example

You don't have permission to view the spoiler content. Log in or register now.

Spoiler: An example of initializing a sprite using RenPy

You don't have permission to view the spoiler content. Log in or register now.

Spoiler: And an example of processing a sprite through autoinit

You don't have permission to view the spoiler content. Log in or register now.

I do understand that using "eval" is bad usecase and yet there's a reason for it.

To remove any misunderstandings in the future, I'll upload the entire autoinit .rpy file, but be careful, it's too big (~600 lines of code).

Spoiler: I'm not joking

You don't have permission to view the spoiler content. Log in or register now.

 

[anne O'nymous]

My problem is that if my autoinit is in

No. Your problem is that, to quote you, "if, instead of running a script to search for and initialize files, you create a file with the file paths already set with the help of a script, the game will load much faster than with the first autoinit option."
The only difference between the script I gave and the one you would create is the loop, and that I forgot the eval that you can easily put back yourself. For information, the time difference for 10,263 images is 0.004 second against the loop approach. Something that is more than bearable...

What also imply that the main issue you have isn't that Ren'Py treat direct declaration faster, but that your autoinit class is extremely slow.

Also, I should have made this clear, but there is a reason to use "eval" because I am initializing a sprite that is a Composite() with a ConditionSwitch, not just a single image.

*sigh*

To remove any misunderstandings in the future, I'll upload the entire autoinit .rpy file, but be careful, it's too big (~600 lines of code).

*sigh again*

Let's be clear here, you wrote a whole fucking big autoinit class, that clearly take a (relatively speaking) astounding long time to proceed, and now you want to optimize it.

And, instead of starting by the most basic and obvious optimizations, you thought about on fly recompilation of a rpy file, something that, between the writing operation to create it, and the recompilation time, will add several seconds to the whole loading process...

This, while not relying on the modFiles list at intermediary, and not having to browse the whole file list twice, while filtering the said list file when you cache it, in order for it to only have your mod files, would already have divided, by a factor that is probably near to two, the time needed for the said class to proceed its main task.

And all this while the use of DynamicImage, or

You must be registered to see the links

, would permit to avoid most of the autoinit part. Something like:

default whateverCharBody = "defaultBody"
default whateverCharClothes = "defaultClothes"
default whateverCharEmotion = "defaultEmotion"

image whateverCharName = ConditionSwitch(
    "persistent.sprite_time=='sunset'",
    im.MatrixColor(im.Composite( (100,100),
                                (0, 0), DynamicImage( "images/path/to/mod/character/[whateverCharBody].ext" ),
                                (0, 0), DynamicImage( "images/path/to/mod/character/[whateverCharClothes].ext" ),
                                (0, 0), DynamicImage( "images/path/to/mod/character/[whateverCharEmotion].ext" )),
                    im.matrix.tint(0.94, 0.82, 1.0)
                ),
    "persistent.sprite_time=='night'",
    im.MatrixColor(im.Composite( (100,100),
                                (0, 0), DynamicImage( "images/path/to/mod/character/[whateverCharBody].ext" ),
                                (0, 0), DynamicImage( "images/path/to/mod/character/[whateverCharClothes].ext" ),
                                (0, 0), DynamicImage( "images/path/to/mod/character/[whateverCharEmotion].ext" )),
                    im.matrix.tint(0.63, 0.78, 0.82)
                ),
    True,
    im.Composite( (100,100),
                                (0, 0), DynamicImage( "images/path/to/mod/character/[whateverCharBody].ext" ),
                                (0, 0), DynamicImage( "images/path/to/mod/character/[whateverCharClothes].ext" ),
                                (0, 0), DynamicImage( "images/path/to/mod/character/[whateverCharEmotion].ext" ))
                )

label whatever:
    show whateverCharName

    menu:
        "Wear casual clothes":
            $ whateverCharClothes = "casual"
        "Wear skimpy clothes":
            $ whateverCharClothes = "skimpy"
   [...]

By the way, why use renpy.image() in the rpy you build in the fly, when the image statement would have added a security level that do not exist with its Python equivalent?

Have also to be noted that, depending on the original game code, the condition switch is possibly partly useless. Ren'Py is fast enough to tint in real time, but the original code need to permit it. Note that it can also possibly be an issue coming directly from the original code.
As for the persistent.sprite_time, a value that depend on the current context, what clearly seem to be the case here, is not a persistent value. But, here again, this can be an issue with the original game.

 

[TrooperJustinHare]

Thank you for reminding me about "layered images" and "dynamic image", it is useful for me, but not in this particular case.

And, instead of starting by the most basic and obvious optimizations.

They are definitely not basic and by far not obvious, If several people with different experience and knowledge of RenPy have been working on it for a couple of weeks. And as a result, it brought it to its current state. With the necessary functionality, and optimized as much as our knowledge allows.
I should elaborate on this topic. You see, there is a reason why initializing sprites (and there's a ton of 'em) is done this way, and why putting a sprite's clothes/emotion in a variable using DynamicImage is not suitable for us at all. Yes, this is a good optimization option, but in our particular case it is inconvenient. The class itself was created with the functionality that we need and the names of the sprites that we are comfortable working with. In this regard, convenience trumps optimization for us.
Therefore, the question is specifically about compiling the .rpy file, not about optimizing the entire script as a whole. If I just wanted to brazenly ask for help optimizing this huge script, I would have written so. But thanks anyway for looking into this.

By the way, why use renpy.image() in the rpy you build in the fly, when the image statement would have added a security level that do not exist with its Python equivalent?

That's a… good point, didn't know about that, I'll change it.

 

[Turning Tricks]

Sorry to insert myself, as most of this is way above my experience level, but the use of Image Manipulators im.xxxxx in Ren'py has been deprecated and PyTom has stated they should not be used as they can cause problems and, in almost all cases, have been replaced with Ren'py functions.

You must be registered to see the links

 

[TrooperJustinHare]

Sorry to insert myself, as most of this is way above my experience level, but the use of Image Manipulators im.xxxxx in Ren'py has been deprecated and PyTom has stated they should not be used as they can cause problems and, in almost all cases, have been replaced with Ren'py functions.

You must be registered to see the links

It was deprecated since RenPy 7.4.0, and our team's using 7.4.11, so it's not a big deal, I think. But that's a good advice, I'll put a TODO thingy into the code.

 

[<Code/>]

Is there a reason you need to import the files yourself? Or are people adding the mods after the game has started running?

 

[TrooperJustinHare]

Is there a reason you need to import the files yourself? Or are people adding the mods after the game has started running?

The game in question is Everlasting Summer (and

You must be registered to see the links

)

As I said earlier, the game has native support for modifications and Steam Workshop. Modifications are not just simple cheat menus or LP widgets, but stories with their own graphics, music, and scenario. And there are a lot of these files (especially sprites), and built-in RenPy autoinit does not apply to them (especially not to the file init practices that this community is used to). So people have to init each image, music (path to music, to be exact), fonts, Composite sprites, etc in the .rpy file.

Spoiler: Example

You don't have permission to view the spoiler content. Log in or register now.

To be honest, I do not know how Steam Workshop handles mods for games on RenPy. All I know is that with more mods, the game takes longer to load. I'll assume that mods are loaded at the launch of the game. And that's why I need to compile an .rpy file with assets into .rpyc at the game launch stage.

 

[anne O'nymous]

*sigh*

Therefore, the question is specifically about compiling the .rpy file, not about optimizing the entire script as a whole.

You totally missed the important part of my post...

This kind of code:

init python:

    modFiles = []
    modFiles.append( [...] )
    [...]

    for type, file_name, file in modFiles:
        if type == "sound":
            setattr( store, file_name, file )
        elif type == "image":
            renpy.image( file_name, file )
        if type == "sprite":
            renpy.image( file_name, file )

Take 0.004 second more to declare 10,263 images than the

init python:
    renpy.image( [...] )

RPY file you create on the fly and want to compile.

4 microseconds of difference is not noticeable.
4 microseconds for 10,263 iterations do not traduce in seconds for an unoptimized code with the same number of iterations.

You're chasing the wrong problem, and clearly since so long that you're totally unable to imagine that it cannot the problem you have.

So people have to init each image, music (path to music, to be exact), fonts, Composite sprites, etc in the .rpy file.

And it's what an approach relying on DynamicImage is doing. And it's doing it in the exact way that, according to you, make the use of explicitly defined none dynamic images being mandatory.

To be honest, I do not know how Steam Workshop handles mods for games on RenPy. All I know is that with more mods, the game takes longer to load. I'll assume that mods are loaded at the launch of the game. And that's why I need to compile an .rpy file with assets into .rpyc at the game launch stage.

*sigh*

Steam Workshop handles mods for games on Ren'Py the exact same way that it handles mods for whatever game, by downloading the needed files at the given location.
And obviously that Ren'Py need more time to launch the game, since it isn't designed to allow/disallow mods. It will load every piece of code, whatever if it correspond to an enabled mod or a disabled one. And like I have no doubt that most of them are as out of context as yours, it need minutes to load.

 

[TrooperJustinHare]

RPY file you create on the fly and want to compile.

4 microseconds of difference is not noticeable.
4 microseconds for 10,263 iterations do not traduce in seconds for an unoptimized code with the same number of iterations.

I was about to write a whole wall of text explaining why that’s not the case for me, but then I decided to test the last autoinit version just in case. And yeah, you’re right. The whole issue with performance loss between initializing stuff on the fly and through a .rpy file comes down to the fucked up autoinit pathfinding function , not because RenPy is somehow faster at processing .rpy files like I thought.

In the previous version, If you run the game with mod in game folder, everything works fine, paths are found quickly. But if you download the mod (like production version of a mod) from the Steam Workshop and then launch the game, the pathfinding system starts searching all the way up through the workshop folder (because the mod isn't in game folder, but in workshop), which really kills performance when initializing on a loading stage. That's why I thought the .rpy option was much faster, because the paths are already written to the .rpy, so RenPy just needs to initialize the files.
Now there's no performance loss between dev and prod versions due to shitty pathfinding func. Thank you kindly.

And it's what an approach relying on DynamicImage is doing. And it's doing it in the exact way that, according to you, make the use of explicitly defined none dynamic images being mandatory.

As I said before, this is a very good optimization option, but in our case it is inconvenient. I think we will create a separate version of auto-initialization using layeredimages (we worked with this a bit because it wasn't suitable for us).

Basically, you could say the problem’s solved, but the idea of compiling an rpy file on the loading stage (or even mid-game) is interesting. I wonder if there’s any way to actually do that.