Ren'Py Displayable to In-Memory PNG Bytes for Base64?

[zonepepus]

Hello !!!!!!!!

I'm working on a project where I need to convert complex displayables (Composites) directly to in-memory PNG bytes for base64 conversion and API transmission. Is it possible to do it?


Thank you so much!!!

 

[n00bi]

I have a hard time understanding this question.

Heres the thing, png files are loaded into memory.
So lets say you have 2 png files, each of these files will occupy a its own region in the memory.

Loading of png into memory usually involves several steps as its a compressed file format.
Reading the data, decoding it, creating a in memory representations of the file etc.
"There is a reason people use 3rd party libs "

The actual compositing process happens either on the cpu or the gpu.
These days its mostly done on the gpu by using opengl, directx etc.

The composite image is then transferred to the framebuffer on the video card to be displayed.
If the compositing was done on the gpu. its directly transferred into the framebuffer, ie by shaders etc.

Can you grab that image from the framenbuffer, absolutely.
This will be the hardest part tho as you need to dive into the api of advance frameworks.
This method is a bit hackish and requires a bit understanding how the hole video pipeline works.
you can probably find tools for this already.

creating base64 encoding isnt hard. you will spend 2min with chatgpt and he will make you some code.
or spend 1h and make one yourself if you know how it works.. there is also a ton of online encoders.

I and not a renpy developer. but there must be some functions inside renpy that creates overlays/handles images etc.
If you can trace back that function(s) you can modify them and do whatever you want with the data.

I have no idea what you mean by API transmission.
An api is not something you transmit. its just a set of rules for building and interacting with different programs.
It just defines how how you do the "communication". regarding protocol, data structures etc etc.

base64 can be send over the network fine.
However. i do not like that. if you plan on sending the png over the network leave it raw.
as base64 increases the size of the data transfer.

Encoded Size (bytes) = ((Input Size (bytes) /3 ) × 4 + (Padding Bytes)
So a 100KB image becomes 133.33KB
((100/3)*4) +0 = 133KB. 33% more data to transfere.

However if you're trying to embed the png into a webpage or something then thats another story.
* background-image: url(data:image/png;base64,iVBORw0KGgoAA......)
* const B64_Image_touch = "data:image/png;base64,iVBORw0KGgoAA....)
...

So Is it possible to do it? , Yes ofc it is, is it worth the effort, that's another story and be your thing to decide.

 

[Deleted member 7747919]

I believe it's possible to create a custom renpy displayable wrapper that allows you to retrieve pixel values directly to create a bitmap. It might not be as complicated as n00bi describes (though I agree that in general determining which pixels are on screen is complicated)

Definitely use a 3rd party lib to convert to PNG. If it just needs to be some sort of file format, something like jpg might be easier. Or even just a plain bitmap.

I believe that the core python libraries have a way to convert to Base64, so you wouldn't even need ChatGpt for this step.

 

[n00bi]

It might not be as complicated as n00bi describes

Yeah,, Not something you want to do unless you're really into opengl,directx and c/c++

But you could probably take a look at the file called.
renpy\display\im.py

There is a class called Composite,
I dont know what version renpy hes using. but in 8.3.4. inside the Composite class there is a function called load.

    def load(self):

        if self.size:
            size = self.size
        else:
            size = cache.get(self.images[0]).get_size()

        os = self.oversample
        size = [s*os for s in size]

        rv = renpy.display.pgrender.surface(size, True)

        for pos, im in zip(self.positions, self.images):
            rv.blit(cache.get(im), [p*os for p in pos])

        return rv

You can probably modify it like.
but you probably need more checks and whatnot as this will be called time Composite is.

        Same as Before
        ....
        # Composite each image onto the blank surface
        for pos, im in zip(self.positions, self.images):
            rv.blit(cache.get(im), [p * os for p in pos])
        
        # Added code to save the composite.
        # Do whatever you like with rv here..
        # Save the composite image to a file (e.g., "composite.png")
        import pygame.image
        pygame.image.save(rv, "composite.png")
        
        return rv

 

[anne O'nymous]

I believe it's possible to create a custom renpy displayable wrapper that allows you to retrieve pixel values directly to create a bitmap.

Yes and no. At displayable level, you don't have access to images, but to a Render object. It's possible to get the final image if you dig deep enough into Ren'Py's internal. But with no guaranties that you'll actually have the final image, because it will be created late since Ren'Py works by layers, that only are assembled when the full result will be actually displayed.

This being said, Ren'Py have two screenshot functions, and one of them should do precisely what OP want,

You must be registered to see the links

. I'm not sure what format is used for the data, but since it's a screenshot, it's a combination of all the layers without the overlays (same has when you hit "H"), and it should be possible to access the data representing the raw image itself.

But you could probably take a look at the file called.
renpy\display\im.py

It wouldn't really help it. As I said above, "rv" isn't an image but a Render object. And accessing the actual image need to dig really deep in the object, with no guaranties that you'll have the actual final result.
What OP would need to do is to create its whole displayable inheriting from Composite, then tweak its render method doing the digging before it return "rv" to Ren'Py.


Edit: Updated the link

 

[Turning Tricks]

Yes and no. At displayable level, you don't have access to images, but to a Render object. It's possible to get the final image if you dig deep enough into Ren'Py's internal. But with no guaranties that you'll actually have the final image, because it will be created late since Ren'Py works by layers, that only are assembled when the full result will be actually displayed.

This being said, Ren'Py have two screenshot functions, and one of them should do precisely what OP want,

You must be registered to see the links

. I'm not sure what format is used for the data, but since it's a screenshot, it's a combination of all the layers without the overlays (same has when you hit "H"), and it should be possible to access the data representing the raw image itself.




It wouldn't really help it. As I said above, "rv" isn't an image but a Render object. And accessing the actual image need to dig really deep in the object, with no guaranties that you'll have the actual final result.
What OP would need to do is to create its whole displayable inheriting from Composite, then tweak its render method doing the digging before it return "rv" to Ren'Py.

I think your link was wrong there...

You must be registered to see the links

 

[anne O'nymous]

I think your link was wrong there...

Yeah, it happen sometimes. I browse the doc locally (since it come with the SDK), and I don't always remember to update it to the last version. So sometimes it don't match anymore the online version.

This being said, now that they are grouped, renpy.render_to_surface() become more visible, and it can possibly be a better option. A the screenshot one need for the displayable to be actually displayed, while this one don't. PyGame.surface aren't actually raw data, but from memory the image would be easier to access than with a Ren'Py Render.

But, obviously, all the points listed above by n00bi stay. Whatever the way to access the image, the base64 conversion and all don't feel like a good idea.
If OP told us what he actually try to achieve, perhaps could we've found a better approach.

 

[n00bi]

It wouldn't really help it. As I said above, "rv" isn't an image but a Render object. And accessing the actual image need to dig really deep in the object, with no guaranties that you'll have the actual final result.
What OP would need to do is to create its whole displayable inheriting from Composite, then tweak its render method doing the digging before it return "rv" to Ren'Py.

If i understand this correct. renpy is based on some sdl and pygame.
And too me it looks like rv is a pygame surface, that means you can grab the texture from it.
"A pygame Surface is used to represent any image, from pygame docs."

If you look in renpy\display\pgrender.py you will see: pygame.Surface and pygame.Surface.subsurface, used all over the place.
Also you will see the usage of blit in the for loop from my prev post:
rv.blit(...)
Which is a common method used to draw one image on top of another.

So technically you can in the load function save the surface as a image. or do whatever you like with it.
But will it give you the result you're after.. probably not. as you said. renpy works in layers.
You are way more familiar with renpy than i am tho.

Btw. do you mean tweak the render function in the Composite class?
I was looking in my renpy sdk. its version 8.3.4, my Composite class does not have a render function.
Or you mean override it by adding this function to the class ?
Or is this something that is in verion 7.x ?

Anyway. i still have no clue what the op is trying to achieve, it would be really helpful if he posted some clarifications.

 

[anne O'nymous]

And too me it looks like rv is a pygame surface, that means you can grab the texture from it.

As I said, it isn't. It's a Ren'Py Render object.

To roughly summarize, it's closer to an object stocking the different steps needed to reach the final image, than the final image itself. There's a PyGame Surface behind, but if I remember correctly the times I messed with custom displayable, you need to go two levels down the Render object to access it. But then you would barely be able to use it because nothing in Ren'Py interface is designed to deal with it. Even the Render object don't accept it as entry.

Ren'Py have been designed with this abstraction in mind because each displayable is subject to transformations, while transformations depend on a timeline and can alter the size of the image. I guess that it permit to harmonize everything, since using directly a PyGame Surface would need to explicitly reset the image for each iteration, while a Render object make this implicit.
Anyway, a dedicated class what mandatory just due to this possible timeline.

So, starting by the Render object you would have to do by yourself what renpy.render_to_surface() seem to do if I understood it's description correctly. Not really interesting

Btw. do you mean tweak the render function in the Composite class?
I was looking in my renpy sdk. its version 8.3.4, my Composite class does not have a render function.

It have it, but by inheritance from ImageBase.

You must be registered to see the links

is the method called by Ren'Py when it want to get the Render object, or sometimes diplayable object, corresponding to the displayable.
Basically, it's where you build the image as intended by the current size and time. For basic displayable, you just resize the image, for more complex one you use time to pace your effect, and/or merge multiples images.

Anyway. i still have no clue what the op is trying to achieve, it would be really helpful if he posted some clarifications.

It's the main issue here, because depending on what he want to do, there possibly not even need to actually care about Ren'Py's internal.
I mean, he talk about a Composite displayable, so strictly speaking loading the base image in a PyGame Surface, then merging the other ones to it, would be easier and faster than trying to get the result of the said Composite displayable. It would even be possible to cache the result if the displayable is used more than once. Perhaps even to start a thread at init time, to cache them all in advance without slowing down the game.

But all depend on what his Composite displayable actually do, and what he want to do with the result.
He want to turn it into Base64, so I assume that the API he talk about isn't local to the game, and transit through a protocol that don't accept mime entities. But in the same time it don't looks like he expect the image to be altered by the API, since he didn't ask how to rever the process. :/