Tool Ren'Py UnRen.bat v1.0.11d - RPA Extractor, RPYC Decompiler, Console/Developer Menu Enabler

[VepsrP]

I did so far not even test the deobfuscation... he, he. Its in the unused function 'assert_is_normal_rpyc', so its without value.
Regardless i spotted these 2 mentioned problems:

• line 236 and 239 should be the same variable/name i think: 'header' -> 'header_data'
• line 247/251 and 259 the same: 'uncompressed' -> 'raw_contents'

And more of your exceptions after a test. Two of them, many repeats:

• Line 71 needs a byte prefix b'RENPY RPYC' literal on the string. Thats one of your errors.
• Line 291 and 308: ValueError has no "message" member so remove it from 'e.message'. Just 'e' Thats the other error i got.

With summertimesaga as test the deobfuscation did now run without problems.

However another problem: We missed some of the double colons and in one case its even missing. Summertimesaga produced a bunch of them. See attached file.
This was/is exactly my thought. Py2 is end of life anyway and already supported by the unrypc dev. I will see if he takes the work from here for a official py3 version. Very possible we overlooked some problems and he sees them immediately.
For the normal unren.bat. This idea pops every 2 months up. Not a bad idea per se, but people missing the fact this app isnt really maintained anymore and makes it for more unknowing users harder to use.

I kind of removed the extra colon from use, it's strange that it came back. At the expense of errors - is it all from the deobfuscation file? if so, I didn't touch it, like, as soon as the problems with e.message passed.

 

[VepsrP]

Awesome work VepsrP , and awesome discussion even though I'm not fully following lol. I know you built this for Python 3, but I'm gonna ask what I always ask when someone fiddles with un'ren, which is does it work for Summertime Saga ?

Madeddy Already answered above, as you can see, while there are still problems.

 

[yoyomistro]

Madeddy Already answered above, as you can see, while there are still problems.

Wow, I'm silly. I skimmed through his last post and didn't see summertime saga written literally twice lol... Well, from my understanding, 1. It's something fixable 2. The deobfuscation worked, which was the main obstacle with previous iterations due to all the "custom code" they made? I understand the errors thrown in previous versions were fixable, but I'm too lazy to do that so a version of unren that makes STS work out the box is the holy grail for me.

 

[VepsrP]

Wow, I'm silly. I skimmed through his last post and didn't see summertime saga written literally twice lol... Well, from my understanding, 1. It's something fixable 2. The deobfuscation worked, which was the main obstacle with previous iterations due to all the "custom code" they made? I understand the errors thrown in previous versions were fixable, but I'm too lazy to do that so a version of unren that makes STS work out the box is the holy grail for me.

This is already more difficult, because I did decompilation using by python3, which excludes the UnRen tool(original). But I'll see what I can do about it.

 

[Madeddy]

I kind of removed the extra colon from use, it's strange that it came back. At the expense of errors - is it all from the deobfuscation file? if so, I didn't touch it, like, as soon as the problems with e.message passed.

... Well, from my understanding, 1. It's something fixable 2. ...

The errors we got and believed they are fixed before yesterday have nothing to do with deobfuscation. They are from the conversion to py3 and th expected incompatiblitys in the old code base.

Deobfuscation is just active if we call unrpyc with "--try-harder arg" on CLI and we do this just if the normal decompiling not works or we know the game uses deobfuscation.
I played with the deobfuscation just now after i thought we have the other part running. Lets set this aside for later - it's still the py3 conversion which not work(s|ed) like thought. (The colons etc.)

... The deobfuscation worked, which was the main obstacle with previous iterations due to all the "custom code" they made? ...

We do not need deobfusction for SummertimeSaga. It's decompiling just fine. And running. Don't fear good Sir, your addiction to Aunts, Cows, sexy Teachers and Sisters is safe with us. Happy milking fellow Cookie lover.

On a side note folks: Do not waste time and search for the colon error reason etc. I figured it out and fixed it already last evening. At least i think so. Two solutions which both work and we must now decide which one we use. Later more...
Cheerio!

 

[VepsrP]

We do not need deobfusction for SummertimeSaga. It's decompiling just fine. And running. Don't fear good Sir, your addiction to Aunts, Cows, sexy Teachers and Sisters is safe with us. Happy milking fellow Cookie lover.

I do not know where you got such information, but neither on python2(from game) nor on the python3 I could not properly decompile the script of this game(literally just tried) The most interesting thing is that on the python2 blocks are lost, and on the python3 the colons is lost. It's just epic.
Update: Well, I figured out the extra colons, just cut them off, if they are in 'condition'. But now there are empty blocks and this is on Python 3, where in theory everything should be type-top, heels, makeup... Don't pay attention, I just remembered one ad. In short, must again look for what is wrong with this decompilation.

 

[Madeddy]

...on the python2 blocks are lost, and on the python3 the colons is lost. It's just epic.

Lost blocks? colons? No. Nothing here. You mean lost Sumertimesaga cows and milking yoyomistro perverts?
WALL of text and boring code talk warning! RUN! If you can... try it...

So. Lets see: I'm still waiting for it to blow up in my face, but so far it worked on Mythos, 53XHomecoming and STS.
Reverts in sl2decompiler (sry, thats your changes VepsrP ):

1. I reverted all the changes which removed the colons. My theory was: "They're there and setup like this for a reason and the second colons came from something else." I expected it somewhat, but it got just clear after some thinking and experimenting. This where effects of another bug. See 2
2. I reverted the change in the call to self.print_keywords_and_children in line 108 from None into ast.location[1]. The None value can not be replaced with "whatever" on a whim. Would be more fun, but does not work.

The missing None value did give the old error TypeError: '>' not supported between instances of 'int' and 'str' from the line 313( in the orginal unrpyc v1.1.5): if i.location[1] > last_keyword_line: . So why? We need to understand it, so we can fix it. Explaination:

In py2 this was allowed:

In [23]: 0 > 0
Out[23]: False
In [24]: 0 > None
Out[24]: True
In [25]: 5 > None
Out[25]: True
In [26]: -1 > None
Out[26]: True

In py3 its forbidden:

In [36]: 0 > None
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-36-84134e7bdf39> in <module>
----> 1 0 > None

TypeError: '>' not supported between instances of 'int' and 'NoneType'

Means, we need to let None stay for other things like the correct setting of the colons, but go around it in the integer comparison (line 313). If we would want to remove the use of None type from these variable, it must be from the point on where it is set. It would also be needed to change every code part where the None effect has.


Loaded it up to my fork in a separate(!) branch. Please test folks. My fixes on this are from line 313 to 339 in sl2decompiler.

You must be registered to see the links

• There are no changes beside py3 compatibility. All other works should be separate...
• Removal of py2 remnants.
• I want to use path-like objects with pathlib.
• Change % style placeholder and str.format to f-strings.
• The same for a pep8 refactoring.(For the most part... I use 90 char lines and other small stuff differently)

Have fun.

 

[VepsrP]

Lost blocks? colons? No. Nothing here. You mean lost Sumertimesaga cows and milking yoyomistro perverts?
WALL of text and boring code talk warning! RUN! If you can... try it...

Okay I used your replacement for 'None' when writing blocks. I agree, just changing it to something else in the function call was wrong. But I decided that zero is not quite what is needed, because earlier, when 0 > None, it should be true, if I do not confuse anything, but if you replace 'None' with 0(Line 318 in sl2decompiler), then in theory it turns out to be false. And then I decided to replace 'None' with the smallest number in the int type. Which I did.

last_keyword_line = -sys.maxsize if last_keyword_line is None else last_keyword_line

In principle, now SummertimeSaga and other games decompile normally. I also figured out the colons in the IF and USE blocks, which are either there or not. I decided to just trim the colon at the end of the parameter if there is already a colon there. Thus, there will be no extra ones, and the right one will always be(I hope).

self.write("%s %s:" % (keyword(), condition.rstrip(':')))

self.write(" id %s" % ast.id.rstrip(':'))

Also, in this build, I took your deobfuscation code, so that I would not bother with it anymore.

 

[VepsrP]

Wow, I'm silly. I skimmed through his last post and didn't see summertime saga written literally twice lol... Well, from my understanding, 1. It's something fixable 2. The deobfuscation worked, which was the main obstacle with previous iterations due to all the "custom code" they made? I understand the errors thrown in previous versions were fixable, but I'm too lazy to do that so a version of unren that makes STS work out the box is the holy grail for me.

In general, I tried. But the scrap of the python2 does not give out the necessary blocks in some places, and an attempt to call them led to the "Out of memory" error. Plus, in some cases, there is a message about the encoding "cp866", where it came from there, it is not clear, it is probably written somewhere in the game files. In general, so far I can only suggest using a normal person's Python3 instead of a RenPy scrap specifically for this game. Plus, if you want to use UnRen, then complain to the creators of the decompiler with the attached errors, as soon as they release a new fix, we may replace it in the tool, or maybe Madeddy will come up with something.

 

[yoyomistro]

I'll give both your changes a try. I've dedicated some time recently to pick back up on seriously trying to learn Japanese, but you've both motivated me to set some time apart to pick up Python. Hopefully getting back into coding won't be too difficult; I made some halfhearted attempts a C++ some years ago but I think I have the will to dedicate some real effort here lol. We'll see how it goes.

 

[Madeddy]

... But I decided that zero is not quite what is needed, because earlier, when 0 > None, it should be true, if I do not confuse anything, but if you replace 'None' with 0(Line 318 in sl2decompiler), then in theory it turns out to be false. And then I decided to replace 'None' with the smallest number in the int type....

Believe me i did the same thinking about this problem yesterday and tested some.

Its also the reason for the control print in there at line 323/324. Both sides of the comparison where in tests with 3 games never "zero". Logically it makes sense to never have a "line 0" and so i used it. If this really gets to be a issue we could go with -1. But int sys.maxsize? Ever heard the word "overkill"?

I also figured out the colons in the IF and USE blocks, which are either there or not. I decided to just trim the colon at the end of the parameter if there is already a colon there....

What is the problem with these colons again? I have no errors or extra colons... They where all from the replaced "None" in the function call imho and we repaired it. Or?

... there is a message about the encoding "cp866", where it came from there, it is not clear, it is probably written somewhere in the game files. In general, so far I can only suggest using a normal person's Python3 instead of a RenPy scrap specifically for this game....

You try to run it over RenPy's python? Gives you just many more grey hair.... cp866 is a old firend of mine i think... Its a missing encoding module. RenPys python is simply castrated: No multiprocessing, missing encoding modules... etc.

...t you've both motivated me to set some time apart to pick up Python. Hopefully getting back into coding won't be too difficult; I made some halfhearted attempts a C++ some years ago ...

Good. It will be fun.
Starting with c++ is hard i think. Some HTML +CSS + javasript or python is better for starting with writing code. Ask if you're stuck on something.

 

[yoyomistro]

Believe me i did the same thinking about this problem yesterday and tested some.

Its also the reason for the control print in there at line 323/324. Both sides of the comparison where in tests with 3 games never "zero". Logically it makes sense to never have a "line 0" and so i used it. If this really gets to be a issue we could go with -1. But int sys.maxsize? Ever heard the word "overkill"?
What is the problem with these colons again? I have no errors or extra colons... They where all from the replaced "None" in the function call imho and we repaired it. Or?
You try to run it over RenPy's python? Gives you just many more grey hair.... cp866 is a old firend of mine i think... Its a missing encoding module. RenPys python is simply castrated: No multiprocessing, missing encoding modules... etc.
Good. It will be fun.
Starting with c++ is hard i think. Some HTML +CSS + javasript or python is better for starting with writing code. Ask if you're stuck on something.

I did Javascript and HTML before I did C++, I just haven't done anything serious in a long time. I'm not a total noob but I can't pretend I have an active codebase I work on or something lol.

 

[Walter Victor]

Believe me i did the same thinking about this problem yesterday and tested some.

Its also the reason for the control print in there at line 323/324. Both sides of the comparison where in tests with 3 games never "zero". Logically it makes sense to never have a "line 0" and so i used it. If this really gets to be a issue we could go with -1. But int sys.maxsize? Ever heard the word "overkill"?
What is the problem with these colons again? I have no errors or extra colons... They where all from the replaced "None" in the function call imho and we repaired it. Or?
You try to run it over RenPy's python? Gives you just many more grey hair.... cp866 is a old firend of mine i think... Its a missing encoding module. RenPys python is simply castrated: No multiprocessing, missing encoding modules... etc.
Good. It will be fun.
Starting with c++ is hard i think. Some HTML +CSS + javasript or python is better for starting with writing code. Ask if you're stuck on something.

Do you mean that my decades of Cobol experience won't help with this!?

 

[VepsrP]

Believe me i did the same thinking about this problem yesterday and tested some.

Its also the reason for the control print in there at line 323/324. Both sides of the comparison where in tests with 3 games never "zero". Logically it makes sense to never have a "line 0" and so i used it. If this really gets to be a issue we could go with -1. But int sys.maxsize? Ever heard the word "overkill"?
What is the problem with these colons again? I have no errors or extra colons... They where all from the replaced "None" in the function call imho and we repaired it. Or?
You try to run it over RenPy's python? Gives you just many more grey hair.... cp866 is a old firend of mine i think... Its a missing encoding module. RenPys python is simply castrated: No multiprocessing, missing encoding modules... etc.
Good. It will be fun.
Starting with c++ is hard i think. Some HTML +CSS + javasript or python is better for starting with writing code. Ask if you're stuck on something.

1. Well, hyperbole is about me.

2. I lost the colons when trying to decompile the SummertimeSaga code. It was in one line, where the IF block ended with a function call, and if I wrote like you, then two colons appear instead of one, I can send error logs if you don't believe me. Why this happens, I do not understand myself.
Update: Problems with extra and missing colons are gone. Was it really the wrong substitution of None for children[1]?.. Probably... However, I do not remember at what point I fixed this place yesterday, probably after I had solved all the other problems. Well, now everything seems to be fine, except that there are some warnings, but they are not critical, as I understood.


3. About the encoding - I tried to decompile the same code using UnRen (the one I modified, not 0.91) and the decompiler in it. and on one of the files, he told me about the problem with this cp866, but where it comes from, I did not understand.

 

[VepsrP]

Awesome work VepsrP , and awesome discussion even though I'm not fully following lol.

I found the right joke for that.

Translation:
Senior developers are arguing about the implementation of the project.
I listen and don't understand half of their arguments.

 

[Madeddy]

Do you mean that my decades of Cobol experience won't help with this!?

No, no.It will be helpful because it gives the mindset of a programmer. Problem will be to find others which have experience in this and are still with us.
Btw: Was "Bismark" a cool guy?

...Was it really the wrong substitution of None for children[1]?.. Probably.....

Hm, yes. I think so. Replacing None did put a positive line-number in there and some comparison came with True back instead False.(And we got a extra colon from it.) Theoretical...

Anyway.
1) In sl2decompiler line 301 we have current_line = (0, []), This value goes a bit later in our problem comparison
if i.location[1] > last_keyword_line: on the right side. So you where right: Its possible "last_keyword_line is set to zero in some cases. i go now with -1 as linenumber for None.
2) We forgot the right side of the comparison. What if there pops someday also a None up(Don't know if possible)? We would get the same error again. So i changed the code there again and did the same convert for the right side.

ln_num_kw = keywords_by_line[-1][0] if keywords_by_line[-1][0] is not None  else -1  # changed org. line with renamed var
children_with_keywords = []
children_after_keywords = []
for i in children:
    ln_num_child = i.location[1] if i.location[1] is not None else -1  # new line
    if ln_num_child > ln_num_kw:  # changed org. line
        children_after_keywords.append(i)
    else:
        children_with_keywords.append((i.location[1], i))

Comments on code diffs., all other is original code.

 

[Walter Victor]

Btw: Was "Bismark" a cool guy?

I always found Otto to be an overbearing prig!

 

[VepsrP]

1) In sl2decompiler line 301 we have current_line = (0, []), This value goes a bit later in our problem comparison
if i.location[1] > last_keyword_line: on the right side. So you where right: Its possible "last_keyword_line is set to zero in some cases. i go now with -1 as linenumber for None.
2) We forgot the right side of the comparison. What if there pops someday also a None up(Don't know if possible)? We would get the same error again. So i changed the code there again and did the same convert for the right side.

ln_num_kw = keywords_by_line[-1][0] if keywords_by_line[-1][0] is not None  else -1  # changed org. line with renamed var
children_with_keywords = []
children_after_keywords = []
for i in children:
    ln_num_child = i.location[1] if i.location[1] is not None else -1  # new line
    if ln_num_child > ln_num_kw:  # changed org. line
        children_after_keywords.append(i)
    else:
        children_with_keywords.append((i.location[1], i))

Comments on code diffs., all other is original code.

And with the change from point 2, we come back to the problem from point 1. But I think that this change is generally meaningless and that's why. If you study the comments to the code and the decompiler dump, then the location parameter is missing only in the SLBlock element. And as we know, a SLBlock cannot be a child of another SLBlock. There will always be another element that will be the parent element for the next block(SLIf, SLFor, etc.). Which in turn means that there is no None in children element, since the SLBlock is output via other functions.

 

[Madeddy]

Possible. I did'n study and trying to understand it this deep. You're way ahead there.

I put the changes of the port forward to the unrpyc devs to take a look and well... They know the code better as we are and see if something wrong is or unneeded like my additional None replacement.
They should at least be interested because the port is inevitable at some point.

 

[VepsrP]

Possible. I did'n study and trying to understand it this deep. You're way ahead there.

I put the changes of the port forward to the unrpyc devs to take a look and well... They know the code better as we are and see if something wrong is or unneeded like my additional None replacement.
They should at least be interested because the port is inevitable at some point.

Good. In the meantime, we can do more interesting things. I just came up with an interesting idea on how to make life even easier for windows users. Yes, it is for them(Windows users are my favorites). Although I think that for the rest it can also be done, I'm just not a layman in other systems, and there is nothing to test on.