Tool Ren'Py Tool for building update patches for Ren'Py games

[jenora]

Description

I made a small python script that builds update patches for Ren'Py games. The goal is to easily make update patches so you don't have to re-download a game every update.

You specify a patch folder, old game folder, and new game folder. The script will find all the newly added files and changed files in the "[game name]\game" folder. It then copies these files to the "[patch folder]\game" folder, which can then be compressed and distributed. This patch can then be downloaded and then the games folder can be copied into old version of the game and it will be updated to the new version. The patch will include all new and edited scripts, images, videos, and audio from the new game version.

I'll probably make some update patches for games I play.

Limitations

The patch will ignore Ren'Py archives (.rpa), since they defeat the purpose of these patches. Because of this, if the game includes archives they will have to be extracted using one of the RPA extractors or they'll have to be manually added to the patch. It also does not copy any of the cached bytecode from any cache folder or any saves. These can be added to the patch folder manually if you wanted to. I haven't tested it yet, but I believe the patches built with this tool should work across platforms (unless the game specific scripts use something that isn't cross platform).

Usage

Batch Script Versions

Copy the batch script into the folder with the newer version of the game, the folder with the exe in it, and run it. Copy the location for created patch and the old version when prompted. The rest should run automatically.

Python Versions

python renpy_patch_builder.py "path/to/patch/folder" "path/to/old/game/version" "path/to/new/game/version"

"path/to/old/game/version" and "path/to/new/game/version" should point to the base game folder with the game executable in it, not the"[game name]\game" folder.

Requires

I tested it on python 3.7.1, but it should work on any of the python 3.x versions. If you use a different python 3 version and it doesn't work, let me know and I'll try to fix it.

This script doesn't require any 3rd party libraries.

You must be registered to see the links

Drop it into the folder and follow the above instructions

You must be registered to see the links

Here is the python 3 version.

You must be registered to see the links

I've made a version that should work with the python shipped in Ren'Py. The usage is the same.

 

[SWAJ]

Thank you for this tool! I will try it out on my project

 

[anne O'nymous]

I tested it on python 3.7.1, but it should work on any of the python 3.x versions.

Which imply that, for now, it will be difficult for most of the devs to use it.

The large majority of them aren't coders, it would have been way more useful for them to have a script that can use Ren'py's embed Python ; and to couple it with a batch script that will take care of this, like unren (among others) do.

 

[jenora]

Which imply that, for now, it will be difficult for most of the devs to use it.

The large majority of them aren't coders, it would have been way more useful for them to have a script that can use Ren'py's embed Python ; and to couple it with a batch script that will take care of this, like unren (among others) do.

I made a version for the python version shipped with Ren'Py and also put it in a batch script. The batch script certainly isn't the prettiest thing in the world, but I may come back and redo it. Thanks for the suggestions.

 

[Blup Blup]

I discovered this thread after a very frustrating night getting a Ren'Py game up-to-date with patches. Those were plainly wrong and the result wasn't the same as a freshly downloaded complete version.

Two years ago I created a proof of concept to make binary patches for a Visionaire game. Most of the files are equivalent to the .rpa files in Ren'Py. The new content is added at then end and updated media gets the same location in the file. Don't now if this is also true for .rpa else the same method could be used.

Anyway, the reason why the patch method didn't work has multiple reasons. First was not a clean path from the version I had to the new version. The patches were labeled with a specific version in the first post, bu some had readme.txt files which contained another version. Also the versions must exactly match. Minor version differences could work, but in this case it clearly didn't. The problem is that a file can have changed between my version and the version used to create a patch to an even newer version.

Another problem is only checking the game folder, which again is wrong. A newer version could us a newer Ren'Py version as well. See this example:

--- checksums_10.0.2.md5.sort    2021-07-12 12:23:28.803121772 +0200
+++ checksums_11.0.1.md5.sort    2021-07-12 12:23:45.223489230 +0200
+c0a70854842ea69a94ca075e8af9df76  ./CollegeKings-32.exe
-c1ced1caa891e99b56b4d236dc6fc1f1  ./CollegeKings-32.exe
+c420e73e85dd51117810428ee94b2664  ./CollegeKings.exe
-e5d0f902a5002554bccd30c12bf55d6e  ./CollegeKings.exe

There can also be files removed which were present in a previous version. In most cases this will probably not be a problem, but again it is not the clean way to do this. After patching the new version should be 100% identical to the exported version of the developer.

The proof of concept I mentioned above was used by another player to create a binary patch. First it checks if the old version is 100% identical with the version used to create the patch. If not a fat warning is given that this could mean problems. Next it uses xdelta3 binary patches to update files. So not simple copying the newer version, but only the differences between the files. The mentioned Visionaire game grew by a few hundred megabytes, but if we had used the approach of the whole changed file the patch would almost be as big as the game itself. With the xdelta3 the patch was a small amount -- some overhead is expected -- greater then what the game had grown. After applying all the patches it again ran a check to see if the new version was 100% identical with the newer version used to create the patch.

I hope that this explanation will help in creating more robust patches for Ren'Py games as well. So next time it wouldn't be as frustrating and in the end not working as it did in my frustrating night .

 

[anne O'nymous]

The new content is added at then end and updated media gets the same location in the file. Don't now if this is also true for .rpa else the same method could be used.

RPA file are archives, therefore you need to edit the index table if you want to add something.

Anyway, the reason why the patch method didn't work has multiple reasons.

No, it have only one reason: People don't know what they are doing.
Making a patch for Ren'py is simple:

• Enable the label duplication ;
• Put your new/changed files, both scripts and images, into a RPA file ;
• Give to this archive a name that will make it proceeded after the other RPA files ;
• Explain that this RPA file have to be put in the "game/" directory, where the other RPA files are.

And that's all. If you follow the same rules than for the save compatibility, the only problem you can have is if you updated your version of Ren'py and used the features added to this version.

Also the versions must exactly match.

No. As long as you don't use the additions that came with the new version, this doesn't matter at all. Until the 7.4.x, that started the port to Python 3.x, Ren'py had a really good and strong backward compatibility.
The few issues found were all regarding the parser. But like you talk about RPA, this isn't a problem, the pre-proceeded version of the scripts are included. And while few syntax changed, the result, it, stayed the same.

A newer version could us a newer Ren'Py version as well. See this example:

--- checksums_10.0.2.md5.sort    2021-07-12 12:23:28.803121772 +0200
+++ checksums_11.0.1.md5.sort    2021-07-12 12:23:45.223489230 +0200
+c0a70854842ea69a94ca075e8af9df76  ./CollegeKings-32.exe
-c1ced1caa891e99b56b4d236dc6fc1f1  ./CollegeKings-32.exe
+c420e73e85dd51117810428ee94b2664  ./CollegeKings.exe
-e5d0f902a5002554bccd30c12bf55d6e  ./CollegeKings.exe

What say nothing. The exe are built in the same time than the distribution, they can perfectly have a different MD5 hash while using the exact same version of Ren'py.

There can also be files removed which were present in a previous version. In most cases this will probably not be a problem, [...]

In all cases it will not be a problem.

After patching the new version should be 100% identical to the exported version of the developer.

There's absolutely no reason for this.
Ren'py games do not rely on compiled code and embedded data, but on interpreted code and external data. What matter is that the AST will be the same once loaded into the RAM. And you can perfectly have tenth version of the same rpy files without a single problem. As long as they are proceeded in the right order, it's the up to date one that will update the AST, which will then match the one of the game exported by the author.

The proof of concept I mentioned above was used by another player to create a binary patch.

Oh my fucking insane god... I've did crazy things with Ren'py, including patching the AST directly into the RAM, but a binary patch for Ren'py goes beyond all reason. You do NOT make a binary patch for something that is NOT binary in the first place.
As said above, RPA are achives, therefore there's a structure to follow beyond the sole binary approach ; and there's tools to add files into them. As for the rpyc files, they are Pickle data. Therefore, they follow a structure even more complex than the RPA files.
But anyway, as also said above, just put an updated version of the script, enable label duplication, and ensure that this particular file will be proceeded last, and you have a fully working patch.

I hope that this explanation will help in creating more robust patches for Ren'Py games as well.

As for me, I hope that you'll just look at what Ren'py files are next time you want to explain how to patch them. It would save me a headache.

So next time it wouldn't be as frustrating and in the end not working as it did in my frustrating night .

Next time you've to patch a Ren'py game:

• Use unren to extract the content of the original RPA archives ;
• Delete the RPA files ;
• Use
  You must be registered to see the links
  on the patch RPA files ;
• Move the files you get into the game directory and confirm all replacement.

At worse you'll get few "duplicated label". Close Ren'py, launch again, dot.

 

[Blup Blup]

There's absolutely no reason for this.

Ok, I see then no point in further discussion. May be Ren'Py can survive the plainly wrong patch procedure, but that is NOT how an excuse to do it that way! A good patch should afterwards 100% be the same as the original.

 

[anne O'nymous]

A good patch should afterwards 100% be the same as the original.

And it's exactly what happen with the method I explained. I said it explicitly when I wrote that "what matter is that the AST will be the same once loaded into the RAM".

But it's my fault. Why the hell did I assumed that, because you talked like you did, you had at least some knowledge regarding Python, and therefore Ren'py, internals ?
Yet, the fact that I explicitly said that rpyc files are Pickles data, therefore

You must be registered to see the links

, should have raised some suspicion in your mind. At worse, if you didn't knew what Pickles is, google would have been glad to tell you about "python pickles". Serialization not being something new, and being so wildly used nowadays, I don't see someone with enough knowledge to attempt a writing like the one you did, not knowing what it is, and what it imply for a binary patching approach.
This especially when put in perspective with the fact that I talked about the presence of an

You must be registered to see the links

. It's far to be a novelty in codding, and someone who talk about binary patching should have at least basic knowledge regarding how compilers works ; and therefore know what an AST is. Then, the fact that I place it into the RAM of an active program, should have made you understand my sentence, and avoid writing something as stupid as what I quoted here.



For the, many, readers who don't have the needed knowledge, here's what it mean:

Compilers take lines of code wrote in a language understandable by humans, and transform it into binary codes understandable by machines. But for this it use an intermediary step, the AST.
As its name suggest, the Abstract Structure Tree is a structure that store an abstract representation of the code as it is in the language understandable by humans. Its content is not anymore understandable by humans, and not yet understandable by machines.
The fact that Python, and by extension Ren'py, rely on an AST stored into the RAM, mean that what they load from their files (rpyc and pyo) isn't yet understandable by machines ; since they can't understand an AST, they can even less understand what created it.
It also imply that Python act like a real time compiler. Every time there's an instruction to process, it take it from the AST, transform it into something that a machine can understand, and send it to the said machine. What is the way all interpreted languages works. Those who don't rely on an AST just take the instruction from somewhere else, but it's another topic.
And finally, it also imply that it's only when the AST is fully loaded into the RAM, that the comparison between two versions of a program make sense. Simply because before this moment, the information have no meaning.

For this last point, imagine that you've a box full of lego bricks. What are they meaning, what are they representing ? They are just bricks. Part of a potential larger set, but nothing more than this affirmation of a possible future.
Lets say that you want to use those bricks to make the flag of your country. What bricks you'll use matter only for their color.
Now, imagine that you aren't given a box of lego bricks, but a tray with ordered bricks. And there's someone else who's gave another tray, with the same number of bricks, arranged in the same order. Then you are asked to build the flag of your country ; that is the same for you two, obviously.
For both your flags to be strictly identical, what matter is that you used the same size, and color, of brick at the same place. He used a 4 points brick here, you've to use a 4 points brick here also. It doesn't matter if you took the second brick of this size and color, while he took the fifth one. It doesn't matter if he started to build the flag from the left, while you started to build it from the top. The only thing that matter is that the bricks are the same at each places.

It's, as you surely imagine, a little more complicated for an AST, but the logic is the same.
It doesn't matter if "this entry" of the AST stayed the same during the whole loading process, or if it have been changed five times. What matter is that it have the expected content at the end of this loading process.
Therefore, it doesn't matter if the content of the files is the same or not. What matter is that once fully loaded, they lead to the exact same AST. And it's precisely what happen with the patching method I explained above.