Ren'Py How to make choices impact latter

[anne O'nymous]

Here, let me present to you my rudimentary migration tool:

You must be registered to see the links

Hmm, what is the interest?

If you don't want to pollute the store, you can perfectly use a

You must be registered to see the links

.

For the transfer from a game to another, there's the

You must be registered to see the links

class.

And if you want to have variables that will only exist during a play session, like by example a time counter for choice or variables for a mini game, you can either rely on renpy.dynamic() or store them in a defined object:

init python:

    class MyUnsavedStore( renpy.python.RevertableObject ):
        pass

define tmp = MyUnsavedStore()

label whatever:
    $ tmp.counter = 1.0
    [...]

Or, since most of the time they are used by a screen and only by it, declare then as local to the screen through the

You must be registered to see the links

screen statement.

 

[Mispeled]

Hmm, what is the interest?

A custom store is just the store with an extra module name for allowing you to have a more modular naming convention. You are still keeping all of those values in memory at runtime.
The point isn't to keep the data persistent, the point is to have that data in cold storage rather than kept in memory at runtime because it's all useless data in a current version of a game but might become relevant in a later version.
Like having a database to store stuff in. You don't want to keep your entire database loaded in memory at runtime.

Edit: and before you say "jsondb", it wouldn't work with rollback and also isn't unique to a save file since it's a database across saves. (Very good for stuff like unlocking photo gallery, saving preferences, or remembering if a player has done something at least once, like to skip tutorials or know if they completed the game once already)

Edit 2: Here, let me actually present this in a more comprehensible way:

Feature	store	persistent	jsondb	MultiPersistent	Migration Helper
Per-save data	✅ Yes	❌ No	❌ No	❌ No	✅ Yes
Supports rollback	✅ Yes	❌ No	❌ No	❌ No	⚠ No (But need to be changed to do so)
Persistent across sessions	✅ Yes (Per save file)	✅ Yes	✅ Yes	✅ Yes	❌ No (by design)
Safe for migrating data	❌ No	⚠ Risky	⚠ Risky	⚠ Partial	✅ Yes
Version upgrade support	❌ No	❌ Manual	❌ Manual	⚠ Manual	⚠ Built-in (Might need some tweaking)

Edit 3: Migration Helper uses a defined object. That's the whole point. It stores it in a defined object for a session and, with a hook to the save callback, store it in the save file like a cold storage so that it can be migrated later on if the data becomes relevant in a future version.

 

[anne O'nymous]

The point isn't to keep the data persistent, the point is to have that data in cold storage rather than kept in memory at runtime because it's all useless data in a current version of a game but might become relevant in a later version.

Then an even big "what is the interest"?

By itself, so with an empty AST and no images, Ren'Py core use near 200MB before you even launch the game. It's not the less than 100 KB that your own variables will add that will change something.
Let say that you use a ten characters ID for each choices, stored in a set, with 1,000 menus in your game you would just add 0.03% occupation (0.96% counting the rollback) to the very minimal RAM used by Ren'Py.

And, obviously, if you add the AST, so the game itself, you can easily add 100MB. With the way Ren'Py handle them, few personal screens can easily add another 10MB, what is more space than you'll ever use with your own data, even counting the rollback.
In the end, for an average completed Ren'Py game, a set of ten characters ID, stored in a set, of 1,000 menus would occupy less than 0.01% of the RAM used by Ren'Py, rollback included.

But it's not what you're doing. Instead you're creating a dictionary, stored in an object. What mean that you'll occupy more space that if you used a regular variable or, better, a set:

init python:

    def estimateUsage( o ):
        '''
            reference: https://code.activestate.com/recipes/577504/

            Estimate the size used *by an attribute*.
        '''

        try:    from itertools import chain
        except: canWorks = False
        try:    from collections import deque
        except: canWorks = False

        all_handlers = {tuple: iter,
                        deque: iter,
                        frozenset: iter,
                        list: iter,
                        dict: lambda d: chain.from_iterable(d.items()),
                        set: iter,
                       }
        seen = set()
        default_size = sys.getsizeof(0)

        def sizeOf(o):
            if id(o) in seen: return 0
            seen.add(id(o))

            s = sys.getsizeof(o, default_size)
            for typ, handler in all_handlers.items():
                if isinstance(o, typ):
                    try:    s += sum(map(sizeOf, all_handlers[typ](o)))
                    except: s += default_size
                    break
            else:
                if hasattr( o, "__dict__" ):
                    try:    s += sum(map(sizeOf, all_handlers[__builtins__["dict"]](o.__dict__)))
                    except: s += default_size
                if hasattr( o, "__slot__" ):
                    try:    s += sum(map(sizeOf, all_handlers[__builtins__["dict"]](o.__slot__)))
                    except: s += default_size

            return s

        return sizeOf(o)

define myDict = {}
define mySet = set( [] )

label start:
    $ s0 = estimateUsage( myDict )
    $ myDict["some_value"] = 1
    $ s1 = estimateUsage( myDict )
    $ myDict["some_other_value"] = 2
    $ s2 = estimateUsage( myDict )
    "Empty: [s0]\nOne entry: [s1] ([s1-s0])\ntwo entries: [s2] ([s2-s1])"
    $ s0 = estimateUsage( mySet )
    $ mySet.add( "ABCDEFGHI0" )
    $ s1 = estimateUsage( mySet )
    $ mySet.add( "ABCDEFGHI1" )
    $ s2 = estimateUsage( mySet )
    "Empty: [s0]\nOne entry: [s1] ([s1-s0])\ntwo entries: [s2] ([s2-s1])"

The first variable in your example add 94 bytes, the second 106. Against a constant increase of 70 bytes for a set of 10 characters ID.
It lead to a difference of 116 bytes just for the two entries in this example. What lead to a difference a bit over 3KB if you want you migration_helper to be rollback compliant, what it currently isn't.

And this is another problem with your approach, because you can't have something like this:

label whatever:
    menu:
        "Go to the park.":
            $ migration_helper.set_var("first park day", 2)
        "Go to the gym.":
            $ migration_helper.set_var("first gym day", 2)

If the player rollback to change his decision, the two will be stored in your object. You systematically need to have the same key name for all choice, with only the value changing, something that limits the possible usage.

Like having a database to store stuff in. You don't want to keep your entire database loaded in memory at runtime.

Yet you want it 100% available at anytime, because you never know when you'll need it...

Edit 2: Here, let me actually present this in a more comprehensible way:

There's so many wrong in your table that I wouldn't even know where to starts.

Edit 3: Migration Helper uses a defined object. That's the whole point. It stores it in a defined object for a session and, with a hook to the save callback, store it in the save file like a cold storage so that it can be migrated later on if the data becomes relevant in a future version.

What mean that it's:

• Stored in the memory;
• Pollute the global space;
• Isn't rollback compliant;
• Need to be loaded each time a save file is loaded, else you would lost everything previously stored.

The only differences with a dictionnary that would be saved automatically is that it don't participate to the rollback and need more manipulations to be operated.

So, yeah, what is the interest?

 

[Mispeled]

I'm realizing just how limited Ren'py is the more I'm working with it.
First, let me clarify something:

But it's not what you're doing. Instead you're creating a dictionary, stored in an object. What mean that you'll occupy more space that if you used a regular variable or, better, a set:

A set only stores values. That’s fine if you just want a boolean "present or not" flag, but it’s useless if you actually want to store key-value pairs, which is necessary if you want to support arbitrary types, or more complex data, as part of migration or future-proofing.

The point was to persist key-values for migration purposes, using `save_json_callback`, and not saving them through the store.

If the player rollback to change his decision, the two will be stored in your object. You systematically need to have the same key name for all choice, with only the value changing, something that limits the possible usage.

You're right, my implementation is incomplete and is in fact missing rollback-compliance.
I could make the dictionary MultiRevertable to make it rollback-compliant, but as you noted, it saves it in the store and defeats the point of it being meant for cold storage.

Because of how rigid the rollback system is, being couple to the save/load system, and the fact that it's not possible to intercept or filter data before serialization because of the lack of callbacks provided by Ren'py, the only way to do what I was trying to do (cold storage and rollback-compliant) is to create a parallel rollback system but that is honestly too much work for optimizing a genre of game that could be written in Visual Basic and still run at 2000 FPS on Windows Vista with a potato hard drive and half a stick of ram duct-taped to a PCI-less motherboard.

What mean that it's:

• Stored in the memory;
• Pollute the global space;
• Isn't rollback compliant;
• Need to be loaded each time a save file is loaded, else you would lost everything previously stored.

It doesn't need to be loaded, that's the point of a cold storage. It's only stored in memory for the session so it can be migrated later on when it's actually needed instead of polluting across sessions.

The only real limitation is the lack of rollback compliance, and yeah, I could fix that with way more code, but honestly? Not gonna do that because Ren’Py is about as flexible as a brick with concrete shoes, so I’m not gonna lose sleep over it.

So, yeah, what is the interest?

As I’ve explained, the interest is for future-proofing and migration, especially for games with lots of custom or evolving data. For most projects, this is an optimization that’s overkill. If you hit the scale where it matters, it’s a sign that Ren’Py is not the right engine for the job, and that’s a design tradeoff to make early.
So I suppose tolerating bad coding practices is just part of the Ren'py experience.

TL;DR: Yeah, don't use my migration helper, it's incomplete, and Ren'py is like a toaster, great for bread and bagels, but if you need anything fancier or a more evenly toasted bread, you're shit out of luck.

 

[peterppp]

TL;DR: Yeah, don't use my migration helper, it's incomplete, and Ren'py is like a toaster, great for bread and bagels, but if you need anything fancier or a more evenly toasted bread, you're shit out of luck.

 

[anne O'nymous]

I'm realizing just how limited Ren'py is the more I'm working with it.

Sorry, but If you say this, then you don't know much about Ren'Py, nor even about Python.

A set only stores values.

I took set only as comparison point, because it's what I used above in the thread. But the issue isn't that set would be better, it's that your approach solve nothing.
This being said, it's perfectly possible to store tuples in a set. Like it's not too difficult to mimic your key/value approach with a set, as long as the value is scalar (by definition only since with Python everything is an object).

def add( key, value ):
    def getType( value ):
        if isinstance( value, int ):
            return 'I'
        elif isinstance( value, float ):
            return 'F'
        else:
            return 'S'

    for atom in mySet:
        if atom.startswith( key+'|' ):
            mySet.remove( atom )
            break
    mySet.add( "{}|{}{}".format( key, getType( value ), value ) )

def get( key ):
    for atom in mySet:
        if atom.startswith( key+'|' ):
            idx = atom.index('|')
            t = atom[idx+1]
            if t == 'I':
               return int( atom[idx+2:] )
            elif t == 'F':
               return float( atom[idx+2:] )
            else:
               return atom[idx+2:]

Your first example would use 76 bytes in a set instead of 94 in a dict. The second would use 88 bytes instead of 106. 18 bytes of differences, therefore a bit more than 0.5KB counting the rollback.
For those who are curious, using a set of tuples would increase the size by 72 bytes by entries.

Not that I advocate for this. Using a dict in such case is obviously a better approach since natural. But you said that it's not possible, I just explain that it is.

That’s fine if you just want a boolean "present or not" flag, but it’s useless if you actually want to store key-value pairs, which is necessary if you want to support arbitrary types, or more complex data, as part of migration or future-proofing.

And creating an object that will rely on a dictionary to store them is nothing else than recreating Ren'Py native stores, in a more complex way. This for no actual benefit since it's harder to operate, use more RAM, and take more space in the save file.

The point was to persist key-values for migration purposes, using `save_json_callback`, and not saving them through the store.

As I said, the first can be done with the MultiPersistent class. And, being designed especially for this, this class do not need that you keep the save directory name constant.

As for the second, the question is still the same: what is the interest?


In memory, every single variables created in a RPY file is stacked in the store. So, it change nothing at this level. At most it would use less RAM, but at the price of the none rollback compliance.

Like you include it in the save, it make no difference with it being saved through the store. Except that you need one more operation to save it, and one more to load it.
There's possibly also issues with the autosave and quicksave, but being at works I can't check if the callback also apply for both.

Like it's in the JSON file and not the pickled copy off the store, it need more space to be saved.

Worse, it's not saved through the store only because it's not rollback compliant. The instant it will become rollback compliant, it would be twice in the save file; once in the pickled file, once in the JSON one.
And if you keep it none rollback compliant, then it raise other issues due to it's persistence, like by example if the player restart the game. And obviously what I previously shown, with both values being saved because the player did a rollback to make another choice.

I could make the dictionary MultiRevertable to make it rollback-compliant, but as you noted, it saves it in the store and defeats the point of it being meant for cold storage.

And this didn't triggered something in your mind? Like by example that "cold storage" is totally useless here because it give absolutely no benefits?

[...] it's not possible to intercept or filter data before serialization because of the lack of callbacks provided by Ren'py, [...]

It's totally possible to do it. It would be dirty (so I'll not give the code), but it would works.

Methods are just attributes pointing to a code instead of a value, what mean that they can perfectly be rewrote without having to create a dedicated class inheriting from the original. And the name of the variables to are in a set.

But the question stay the same, what is the interest?

It's saved with the store, and so? As I said, it would take an insignificant amount of space, both in the RAM and in the save file. So it's not a big deal.
And it can not be passed to another game, but there's already a class dedicated for this. So it's not the ideal.

The only possible use I see is if an update isn't save compliant. But creating a harness to keep the values, while you've no guaranties that they'll still be significants, isn't the way to deal with such issues.

the only way to do what I was trying to do (cold storage and rollback-compliant) is to create a parallel rollback system but that is honestly too much work for optimizing [...]

I stop you right now, it would optimize absolutely nothing at all, it's the opposite.

Writing and reading the JSON file take more time than loading the same data from the pickled file. And it would also take more space in the RAM.

It doesn't need to be loaded, that's the point of a cold storage. It's only stored in memory for the session so it can be migrated later on when it's actually needed instead of polluting across sessions.

It need to be loaded... because what happen is the strict opposite of what you are saying.

None rollback compliant data are fucking persistent...


I have two playthroughs for your game.

I play a bit of the first one, you store my choices in your migration_helper object.

I load a save from the second playthrough. Your migration_helper object is not updated, all the choice I made are stored in it, even the ones regarding choices that I haven't encoutered yet in that route.
I play a bit of that route, my choices are added to your migration_helper object, in top of the ones that were already there.

I load the save from the first playthrough. Your migration_helper object now have the choices I made for both routes...

I return to the second playthrough. Your migration_helper object is now fully polluted for both routes.

Not gonna do that because Ren’Py is about as flexible as a brick with concrete shoes, so I’m not gonna lose sleep over it.

Sorry but

As I’ve explained, the interest is for future-proofing and migration, especially for games with lots of custom or evolving data.

And as I explained, you are doing the opposite, and doing it for absolutely no reasons.

If you hit the scale where it matters, it’s a sign that Ren’Py is not the right engine for the job, and that’s a design tradeoff to make early.

No, if you hit the scale where it matters, it's the sign that you don't know Ren'Py enough.

Not that Ren'Py is the engine for everything, but it can do everything. And when you know it enough, it can do most of it relatively easily.

TL;DR: Yeah, don't use my migration helper, it's incomplete, and Ren'py is like a toaster, great for bread and bagels, but if you need anything fancier or a more evenly toasted bread, you're shit out of luck.

Well, Lust Madness isn't out of luck for his map system and dressing doll systems in Lust Hunter. Like Winged Cloud weren't out of luck for their old school 3D maze in Sakura Dungeon, 9 years ago. And those are only two examples.


TLR: You waste your time complicating your code, for no actual benefits and a lot of risks to hit a wall that would force players to restarts the game from scratch. And the worse is that it teach you nothing regarding Ren'Py.

 

[osanaiko]

Mispeled I get the feeling that you are an experienced programmer who is trying to force Renpy into working a certain way that is non-idiomatic.

Putting program state into an external data file is not "the renpy way" at all. I might have misunderstood, but I don't see anything in your proposal that couldn't be done using internal renpy data structures in the Store.

 

[anne O'nymous]

Mispeled I get the feeling that you are an experienced programmer who is trying to force Renpy into working a certain way that is non-idiomatic.

It's probably me being me, but he sound more like a recently graduated student.

Putting program state into an external data file is not "the renpy way" at all.

It's been two decades that it's not the way at all. RAM isn't anymore an issue, and drive access are by definition way slower than RAM access. Especially with OOP approach that blurred the wall between code and data.
Anyway, due to the way they works, and the structures they provides (hash, dict, etc.), it have never been the way for script languages. Serializing, and even more reading, those structures need way more time for this approach to be efficient and present a gain in regard of their storage in RAM.

I might have misunderstood, but I don't see anything in your proposal that couldn't be done using internal renpy data structures in the Store.

It's the main issue I have: What is he trying to achieve?

Because "future-proofing and migration" explain absolutely nothing. You don't need to save the data differently to be future-proof, and you need to store them fully outside of the usual save process to ensure a correct migration. But he's doing the first, and not the second.

 

[Mispeled]

Well, I'm not gonna respond to ann anymore because he opened with an insult, misconstrued most of what I said, and immediately followed his initial insult with being wrong with his set implementation that only works on primitive types and not on arbitrary objects.
Next, it's gonna be serializing a string into a Set to mimic a Dict.

Mispeled I get the feeling that you are an experienced programmer who is trying to force Renpy into working a certain way that is non-idiomatic.

Putting program state into an external data file is not "the renpy way" at all. I might have misunderstood, but I don't see anything in your proposal that couldn't be done using internal renpy data structures in the Store.

No, I was trying to make it work within the Ren'py framework.
If the idea for Ren'py was to never be able to save into an external file other than through the store, then how does one explain the existence of config.save_json_callbacks which goes against this very idea by offering a way to save non-store values directly to the save file? Or the existence of JSONDB and MultiPersistent which both saves data to the disk.

What I was trying to achieve here, and I've said it a few times now, is fill the missing use case of persistent data that isn't on the store but is save-dependent.
JSONDB and MultiPersistent are both save-agnostic. config.save_json_callbacks is the closest because it is save-dependent and meant for non-store data, but it lacks the rollback-compliance. Getting around that problem requires a solution that goes outside the Ren'py framework itself because rollback is tightly coupled to the store. Which is the point at which I said it's too much, because at that point it would be better to modify the engine itself.

No offense, but the "Ren'py way" is just an excuse for poorly designed systems. It's not the first time I run into questionable design with Ren'py. Like the OpenGL integration (Model Based Rendering) for example, which is done via code injection, which is a debugging nightmare. It was clearly added into Ren'py as a sort of patch to modernize it with newer GPU.
And I don't really fault Ren'py for that, it's an open source engine, a lot of the modules are gonna be put together with duct tape and it works for what it does, which is perfectly fine for the kind of games made on Ren'py.

 

[anne O'nymous]

Well, I'm not gonna respond to ann anymore because he opened with an insult,

Well, I just said the truth, you don't know much about Ren'Py. And both your posts and your code show it relatively explicitly.
If it's an insult for you, it's your problem, not mine.

[...] then how does one explain the existence of config.save_json_callbacks which goes against this very idea by offering a way to save non-store values directly to the save file?

It's simple, as implied by

You must be registered to see the links

, it's to save additional information that you'll be able to show to the players in the load screen, without the need to firstly load the save file. This come in addition to the

You must be registered to see the links

variable and permit to give a better description of the save file.
A game like The Inn show it's application.

Or the existence of JSONDB and MultiPersistent which both saves data to the disk.

Already explained what the MultiPersistent class is.
Personally I use it in my extended variables viewer to share it's configuration between all Ren'Py games. But it's usual use is the one gave by

You must be registered to see the links

:
"Multi-Game persistence is a feature that lets you share information between Ren'Py games. This may be useful if you plan to make a series of games, and want to have them share information."
It permit to assure the continuity for games that need to be split in multiple games due to their size. Like Fetish Locator did, among many others.

What I was trying to achieve here, and I've said it a few times now, is fill the missing use case of persistent data that isn't on the store but is save-dependent.

And as I said, as well as like

You must be registered to see the links

, all data declared in a RPY file are on the store. Absolutely all of them. This simply because "store" is the global scope of the game.

Getting around that problem

Except that the problem exist only in your head...

requires a solution that goes outside the Ren'py framework itself because rollback is tightly coupled to the store.

Proof that it isn't as much as you think, if you open the console and type store.migration_helper you'll access your object. Yet it isn't part of the rollback...

No offense, but the "Ren'py way" is just an excuse for poorly designed systems.

And this is just an excuse for a poorly understood documentation...

Will I'm far to agree with all design choices that PyTom made, Ren'Py is fuckingly well designed, integrating three different native languages, plus Python, in a so smooth way that you see it as a framework that it isn't.

Like the OpenGL integration (Model Based Rendering) for example, which is done via code injection, which is a debugging nightmare.

Ren'Py do not integrate OpenGL. Ren'Py use OpenGL as one of its possible three display interface.



Now, like you think that you can judge Ren'Py design, let's take a look at your code...

init python:
    import renpy.store as store
[...]
init python:
    import renpy.store as store
    import time

You are importing "renpy.store" twice...
In the same scope...
Over itself since you're already in "store"...
And for nothing since the variable store.store (and therefore also store due to Ren'Py's scope abstraction) already point by default to "renpy.store"; in addition to not being needed since it represent the current variable scope.

By the way, the time module is already imported and available through renpy.time.

        def all_vars(self):
            return dict(self._vars)

Why do you translate a dict into a dict?

        def set_var(self, name, value):
            self._vars[name] = value

        def get_var(self, name, default=None):
            return self._vars.get(name, default)

        def remove_var(self, name):
            if name in self._vars:
                del self._vars[name]

__setattr__, __getattr__ and __delattr__ magic methods would make the object way easier to use. Since you care so much about proper design, it's how your class should have been designed.

init python:
   [...]
    migration_helper = MigrationHelper()

As said by

You must be registered to see the links

, this is the reason why your object isn't part of the rollback:
"Variables that have their value set in an init python block are not saved, loaded, and do not participate in rollback."

default some_value = 0
label migration_helper_samples():
    $ renpy.say(None, "Our global some_value is: " + str(some_value))
    $ migration_helper.set_var("some_value", 1)
    [...]
    "We just added some_value with a value of 1 to our Migration Helper."

You don't need to use renpy.say. You already understood that the sayer is optional, but you clearly missed the fact that Ren'Py come with

You must be registered to see the links

. What you wanted is "Our global some_value is [some_value]".

Also, the default some_value = 0 declared the variable in the store and as savable.
Therefore the variable is present twice. Once in the global scope, and once in your migration_helper (that is itself in the global scope), and it will be save twice, once in the pickled file and once in the JSON one.

    show screen migrate_slots

This is a none blocking statement. If the player click outside of a button in the screen, the game will progress. What you needed here is to use

You must be registered to see the links

.


Looks like you aren't really in position to spit on Ren'Py design...
And one could wonder if the reason why you "run into questionable design with Ren'py" isn't mainly due to he fact that you didn't care to read the doc...

 

[Mispeled]

It's called prototyping, maybe you've heard of it. But thanks for the pointless code review like I was submitting code to the Ren'py GitHub and not just testing something.

Do you also critique artists who sketch on a napkin instead of spending hours painting?

Actually, don't answer that, I really don't care.

 

[Cenc]

One thing we can always guarantee on this forum, post any code - there's always someone to come along and tell you its wrong, and that their way is better. Regardless of whether it is or isn't.

Also you guys are talking about stuff wildly OT to the OP - who's probably completely confused since the resolution to his issue was to order the variable to before the jump, not after.

There also the whole defining choice twice... but the point is made.