Ren'Py Using databases in Ren'Py

[Dould255]

I would like to use a database of a large number of characters, similar to how HHS does with SQLite in their custom engine. This would allow the game to keep track of a large number of characters, compile stat averages, and do all kinds of interesting things with joins, selects etc.

The obvious thing to try is SQLite for Python, create a database for the entire application and tables for each new save and trying to avoid table collisions however I can. Sadly, that doesn't work since the tables are persistent through saves and rollbacks. This means if you change something and then reload or rollback, that change has still taken place, making saves and rollbacks meaningless.

Next up, I looked at the library Pandas. At first, it looks to be pretty much what I want, but it's not included in Ren'Py, and adding it for all three major OSes would be more work than seems reasonable. It is not just a simple .py file that could be added to the directory. Numpy, however, is included in Ren'Py, and has a lot of the desired functionality but does not support mixing data types in a single matrix, which I would need to do to save character data.

I could of course try to make my own data type to do everything I want, but that would be a lot of work, and probably quite inefficient by comparison. So, what would you guys recommend? Can any of the above solutions work if I do something differently, or maybe there is some other way of doing what I want done?

 

[Palanto]

Well renpytoms answer to this question was: "No, not possible." So if you find a working database solution give me a sign I do it with character class objects, which store each characters information... Yes it can be a lot of work to set up a character class object at first, but when you're done it's just a single line of code for each character

So it most likely depends on how many and what kind of stats it should store and how complicated your functions are going to be. So it can take from 5 minutes to a couple of days to programm a suitable class for yourself.

 

[anne O'nymous]

@Palanto Games gave you the sad truth...
The only way to use a database with Ren'py is to tweak Ren'py, probably by updating the store module, or directly in the context module. Each time Ren'py will backup the game context (so create a new entry in the rollback table), you backup the whole database. And each time Ren'py restore a previously saved context, you restore the correct version of the database.
So, technically it's possible, but it will slow down the game and I don't even talk about the size which will raise very quickly.

This said, he also gave you the solution used by everyone. Use an object to fake a database, and let Ren'py deal with it. Not only it will be rollback compliant, but it will also be more practical since the computation will be performed directly.
Anyway, a Ren'py game don't need a real database, even SQLlite. There isn't enough dynamical data for this. The only possible use is if you plan a really complex game, like Sakura Dungeon and its 3D old school maps by example. In this case, a database can simplify the creation of the game, but in the end it's just a configuration file, and plain text files can do the same ; using TAB to separate the fields and RC to separate the row.

 

[Dould255]

Good news, pervert nerds!

You must be registered to see the links

seems to do everything I want, and being pure python, can be popped right into the project and just work! It supports building and modifying tables using regular python syntax, and can do stuff like selections, sortings, and joins!
Did I just blow your minds or what?

 

[anne O'nymous]

Like the whole database is stored in memory, it should be fully rollback compliant yes.
But be careful, Ren'py will not save, nor include in the rollback process, objects created at init time. So, you'll have to create it at start time, which will lead to some complication if you update the table between two releases. Nothing hard to deal with, but you must plan it right to avoid problems.
There's also few case where objects are really not rollback compliant. It shouldn't happen with recent versions of Ren'py, but still I recommend you to do some tests before using it your game. Something like this :

label start:
  "1"
  [create a table]
  "2"
  if [test if the table exist]:
    "table exist"
  else:
    "argggg"
  [update the table]
  "3"
  if [test if the table is correctly updated]:
    "table updated"
  else:
    "argggg"
  [update the table]
  "4"
  if [test if the table is correctly updated]:
    "table updated again"
  else:
    "argggg"
  "5 - Save, exist, load here please"
  if [test if the test exist]:
    "table exist"
  else:
    "argggg"
  if [test if the table have the expected value]:
    "table correctly saved and loaded"
  else:
    "argggg"

Then you play with it, going back and forth. Using the numbers as check points, and the test to control that the rollback don't mess with the tables and their values.
When save/exit/load, you must really fully quit the game, not just save then load. Else Ren'py will still have the value in its memory, and it will mess with your test. By example the table can be forgot while saving, but like it's still in Ren'py memory, your test will don't show it.

 

[Dould255]

Yeah, it didn't really work. For one, it wasn't pickleable, due to using weak references, and neither unpicklable, due to an infinite recursion. I managed to bodge both of those though, so I could load and save, but at that point, I realized it didn't rollback properly.

I guess I'll have to make my own, after all. Is a pure python implementation really not something that would benefit from being a ready-made library?

 

[Palanto]

Well it is as it is ren'py is still being in development so to say, who knows what the future might bring, but for now it's limited in that regard

 

[anne O'nymous]

I guess I'll have to make my own, after all.

The question is, do you really need a database ? It's important, because making you own DB from scratch will need times. If it's for something which isn't really necessary, it's probably better to find a compromise between a DB and an object.
The less worse way to do it Ren'py compliant is probably to use dict of list of dict.
[I made it on the fly, don't expect the notation and code to be correct, it's more the spirit than the code]

{ tableName: [ { fieldName : value, fieldName: value }, { fieldName : value, fieldName: value } ],
tableName : [ { fieldName : value, fieldName: value }, { fieldName : value, fieldName: value } ] }

So you'll have something like this :

   class MyDB():
       DB = {}
       def _getTable( self, name ):
           if name in self.DB: return self.DB[name]
           return None
       def _getRow( self, table, row ):
          if len( table ) >= row: return table[row]
           return None
       def _getField( self, row, name ):
           if name in row: return row[name]
           return None
       def newTable( self, name ):
           if not name in self.DB: self.DB[name] = []
          return None
       def newRow( self, tableName, values = {} ):
           if tableName in self.DB: self.DB[tableName].append( values )
          return None
       def updateField( self, tableName, row, fieldName, value ):
           if tableName in self.DB:
              if len( self.DB[tableName] ) >= row:
                  if fieldName in self.DB[tableName][row]:
                        self.DB[tableName][row][fieldName] = value
          return None
      def getField( self, tableName, row, fieldName ):
          t = self._getTable( tableName )
          if not t is None:
             r = self._getRow( t, row )
             if not r is None:
                 return self._getField( r, fieldName )
         return None

There's better way to code it, but like I said, it's just to give you the spirit. You can add other methods to simulate more advanced queries, using this base to navigate through the DB itself.
You can also works like EzTable. Instead of storing the table inside self.DB, you return the dict. It's probably a better approach for Ren'py, but not sure of the performances. It's should also ease the creation of the table, since you can wrote something like :

init python:
   myFirstTable = [ { field1: "aaa", field2: "bbbb" }, { field1: "ccc", field2: "dddd" } ]

Like you don't need full SQL support, and apparently didn't planed to have linked queries (else EzTable wouldn't be made for you), it should do the trick.
Too bad that I'm already late on my mods and still have my extended variable viewer to improve, while have real life which don't let me sleep. Else it would have been an interesting challenge.

 

[Dould255]

You're right, I don't need an entire SQL database, but I kind of want one. For one, it allows for a lot of interesting operations, that I might never use, but they're nice to have. What I really want is being able to filter, search and sort rows by any value, and doing averages and the likes on the values in a certain column.

For context, I really only need one "proper" table, that I would use to store the information on the citizens of a fictional world, so the more rows I could reasonably afford to have the better, with a lower bound around a couple of thousand. I'm not sure they all need names, but the ones who do, would have their names randomised from a decimated version of US census data on the most popular names.

When I said "make my own" I didn't mean making my own proper database, but rather some ad-hoc solution, similar to what you described. For search, filter, sort, etc. I imagine a suitable approach could be list of dicts, one for each stat, pointing to a list of rows along the lines of:

[stat_name:{stat_value:[row], stat_value:[row, row, row]}]

, which could be updated whenever the corresponding table was, or on a index() function call, and then use the keys of this index for the different operations, and finally looking up the corresponding rows. Is there something I'm missing in this approach?

 

[gue5t]

Have you considered

You must be registered to see the links

?

 

[Dould255]

I had not, that looks great! PyDbLite pickles and unpickles properly without needing to bodge anything, so that actually seems really nice.
However, it doesn't support rollback, which kind of sucks. But what I've come to realise, is that I will probably do all stat changes at end-of-week/month/whatever anyway, so if I block rollback past those, I can still support rollback in text, and being able to undo every decision makes for bad gameplay anyway imo.

So I'll probably go for PyDbLite in the end, thanks!

 

[Deleted member 167032]

As i really do not know but can you not use Access database?

 

[anne O'nymous]

As i really do not know but can you not use Access database?

If you mean the DB coming with Office, then the answer is no, he can't, because:
His game will also be played on Mac OS X, Linux, and possibly Android. I highly doubt that it's rollback compliant. Anyone who'll play the game would need to have it installed on his computer. He would have to write a Ren'py script installing the DB. It's probably too much for what he want to do.

 

[Dould255]

I develop on GNU/Linux, so using a Windows-only database is not too appealing. Not to mention it's proprietary software. Disgusting.

Spoiler

You don't have permission to view the spoiler content. Log in or register now.

 

[Studio Errilhl]

Also... why would you use something that hasn't been working well since it was released, when there are literally hundreds of other options for _real_ databases, which are open source and works cross-platform?