Extracting data from bin file

[ahoh]

You can look at the python script for the details.
The summary is that the base file needs to be decrypted by Xoring each byte with an extended key byte. Each byte in the extended key is generated from a base-key + the offset in the file.
Once decrypted the file has a pretty simple archive format with a header that specifies the files contained within and the rest of the file just holding the raw binary data.

This is genuinely fascinating and impressive! Thanks for sharing!

I was poking around in the extracted folder and saw that each scene folder has a bunch of ".zbn" files which I believe to be the available differences for those scenes. I imagine the .zbn to just be some proprietary file extension the author made up since I've never seen it before and cant find any info about it.
Do you have any insight into how one would go about converting/extracting these files to .png?

I tried everything I can think of but all I could figure out is that maybe it has something to do with NScripter (uniextract scans pointed me in that direction)
I'm new to this kind of stuff so I apologize if I'm missing something obvious, But I'm very interested in learning!

 

[xj47]

This is genuinely fascinating and impressive! Thanks for sharing!

I was poking around in the extracted folder and saw that each scene folder has a bunch of ".zbn" files which I believe to be the available differences for those scenes. I imagine the .zbn to just be some proprietary file extension the author made up since I've never seen it before and cant find any info about it.
Do you have any insight into how one would go about converting/extracting these files to .png?

I tried everything I can think of but all I could figure out is that maybe it has something to do with NScripter (uniextract scans pointed me in that direction)
I'm new to this kind of stuff so I apologize if I'm missing something obvious, But I'm very interested in learning!

I had a look at those files as well, I agree that it's some sort of change-set applied to a base image and likely specific to this project. Don't really see the connection to NScripter though.

The zbn files seems to have the format:
"size" value (4-bytes)
zlib-compressed section that when uncompressed is size bytes long

The uncompressed section in turn has the format:
mystery value (4 bytes, haven't figured this one out yet)
size-1
data-section-1
size-2
data-section-2
size-3
data-section-3
size-4
data-section-4

Haven't got much further than that though.
I'd speculate that each data section corresponds to a color channel (RGBA) but if they do I don't know how the data is encoded.

I've looked at the game in a debugger but haven't found the critical functions yet.

 

[ahoh]

I had a look at those files as well, I agree that it's some sort of change-set applied to a base image and likely specific to this project. Don't really see the connection to NScripter though.

The zbn files seems to have the format:
"size" value (4-bytes)
zlib-compressed section that when uncompressed is size bytes long

The uncompressed section in turn has the format:
mystery value (4 bytes, haven't figured this one out yet)
size-1
data-section-1
size-2
data-section-2
size-3
data-section-3
size-4
data-section-4

Haven't got much further than that though.
I'd speculate that each data section corresponds to a color channel (RGBA) but if they do I don't know how the data is encoded.

I've looked at the game in a debugger but haven't found the critical functions yet.

Interesting!
I was poking around with a hex editor (was looking for a file signature) and came to the same conclusion about them being modifiers for the base scene image. Id never have thought to do it that way!
Thanks for taking the time to look at this and explain.
What tools/programs do you use by the way?

 

[xj47]

What tools/programs do you use by the way?

For this I didn't use many tools, just HxD (hex editor) and x64dbg (debugger).
For miscellaneous tasks I write python scripts. So for example the zlib decompression was done with a short python script.

Other tools I frequently use are:
- Process Monitor (allows you to see what a program is doing, mainly useful for seeing read operations)
- Process Explorer (expanded version of task manager)
- ILSpy (If the program uses managed code such as C# then you often get pretty readable code)
- Dll Explorer (For unmanaged code can see public function exports)

Depending on the task at hand I sometimes create injection-DLLs with function hooks as well. This essentially allows you to overwrite functions in the target application and/or create wrapper functions so it has a lot of uses. Hooks are more useful in a "cracking" scenario though where you want to bypass or disable some DRM system. Here I mainly just want to understand how the zbn format works so modifying the game's behaviour isn't that helpful.

 

[xj47]

ahoh
Hey, I figured out the .zbn file format.
It's more complicated than I had initially anticipated. Basically section-3 is a sequence of 2-bit instructions that move a write-head around, sections 1 & 2 provide a sequence of colors, and section 4 contains some misc data for moving the write-head.

Here's a python script to extract those files. I've also included a .txt that explains the file format properly.

I think the zbn format is literally only used for one game, so I doubt this tool will see much use. But hey, I had fun figuring it out.

 

[ahoh]

ahoh
Hey, I figured out the .zbn file format.
It's more complicated than I had initially anticipated. Basically section-3 is a sequence of 2-bit instructions that move a write-head around, sections 1 & 2 provide a sequence of colors, and section 4 contains some misc data for moving the write-head.

Here's a python script to extract those files. I've also included a .txt that explains the file format properly.

I think the zbn format is literally only used for one game, so I doubt this tool will see much use. But hey, I had fun figuring it out.

Neat! Great work by the way, your code and explanations are really informative and easy to follow. I appreciate you taking the time to explain your findings and creating (really specific) tools. Thank you!

I believe there are 2 other games (same artist and dev) that IIRC are very similar. I'll probably take a look for curiosities sake.

 

[ahoh]

Good news! I spent some more time with it and figured it out.
Here's a python 3 script to decrypt and unpack the data.bin file

If you have a lot of trouble running the script I could also upload the decrypted files manually, I just don't feel like uploading a 0.5 GB file unless I have to.

Hello, hope you are doing well! Sorry to be posting in an old thread but I don't have enough posts to start a conversation and in case someone else has a similar issue I'll ask here.

I have been attempting to use the extract tool on the dev's other projects (namely Kisaragi's Dangerously Erotic Certification Exam! ~Resistance Is Futile~) and have been running into an index error in the readIndex() function when called with the unpackDecrypted() function after dataPlain.bin is made. Traceback tells me specifically line 77.

At first I figured the data.bin of this different game might be incompatible with your script but I went and tried it for the data.bin of the game it was intended for and get the same issue! Which is really throwing me for a loop since I never had any issue with your script before.

This usually would constitute an "oh well" from me but since I am currently in the process of learning python I really would like to learn what I'm missing here!

Here is my exact process in case I'm just doing something wrong:
move data.bin of the original game script was made for into ExtractTool folder,
Open cmd and type 'python' to install it through Microsoft store,
Open python in cmd to make sure it worked,
exit() to get out of interpreter,
cd to the directory of the script,
Type in: 'python extract.py'

decryptArchive() seems to be working fine and dataPlain.bin is made

Then I get:
Traceback (most recent call last):
File "C:\Users\Admin\Desktop\ExtractTool\extract.py", line 115, in <module>
unpackDecrypted()
File "C:\Users\Admin\Desktop\ExtractTool\extract.py", line 99, in unpackDecrypted
for offset, dataLen, path in readIndex(data):
File "C:\Users\Admin\Desktop\ExtractTool\extract.py", line 77, in readIndex
while data[j] != 0:
IndexError: index out of range

Although a lot of the script is beyond me, I cant see how j could be an invalid index for data given that dataPlain.bin is the argument for the function's parameter...

Sorry for such a long question and VERY SORRY if its just something simple I'm doing wrong here! I am on a new PC now since I was last able to successfully use the extract script but cant imagine how that could be causing an index invalid...

 

[ahoh]

xj47
After some more troubleshooting I was able to get the script to work again but only using the data.bin provided by the OP. I guess its from an older version of the game? I've been trying to use the data.bin from version 1.02 (latest update) since that's what I happen to have...

Should have checked this earlier, sorry!

I am still curious as to why these other bin files wont work so I'll provide them here:
World Reformation v1.02:

You must be registered to see the links

Kisaragi's Dangerously Erotic Certification Exam! ~Resistance Is Futile~ v1.10:

You must be registered to see the links

 

[xj47]

Most likely the decryption key got changed.
I extracted the original key manually using a debugger and just included it alongside the script. If the key varies the script would need to be updated to extract the key automatically.

 

[ahoh]

Most likely the decryption key got changed.
I extracted the original key manually using a debugger and just included it alongside the script. If the key varies the script would need to be updated to extract the key automatically.

It's so obvious! I guess I'll try to learn how to use x64dbg and get the keys.

Thanks for your help!

 

[xj47]

It's so obvious! I guess I'll try to learn how to use x64dbg and get the keys.

Thanks for your help!

Do you have any experience working with a low-level language like C or C++?
If not I'd recommend learning the basics of C and preferably the basics of assembly as well before trying to make sense of assembly code using a debugger.

Trying to navigate a program's assembly code is difficult.

 

[ahoh]

Do you have any experience working with a low-level language like C or C++?
If not I'd recommend learning the basics of C and preferably the basics of assembly as well before trying to make sense of assembly code using a debugger.

Trying to navigate a program's assembly code is difficult.

I have some experience with C++ but assembly might as well be hieroglyphics. And yea I got as far as attaching and realized I was in miles over my head haha...

I'm probably going to stick to learning python at least until I'm competent.

But the possibilities with a debugger are truly awesome and poking around in x64dbg really opened my eyes at least!

Thanks for the advice regardless

 

[xj47]

ahoh

I revisited the script to see if I could make it more generic.
I've come up with a program that should be able to automatically extract the decryption key from the game. So far I've confirmed it works for the version of World Reformation I have on my PC and "Midnight Castle Succubus 2" which is made by the same dev.

You could try running it for the games that gave you trouble.
Sadly the mega-links you posted don't help me here since I need to run the game in order to extract the key.
(I've actually spent a fair bit of time thinking about how you might be able to brute-force the key, but yoinking it from the game directly is way easier)

I've also updated the extraction code so it runs faster and can detect if the key is correct in advance.

P.S.
Also don't hesitate to ask me questions. I'm quite happy to talk about this stuff.

 

[ahoh]

xj47

I appreciate you taking the time with me!

I tried out your key extractor and haven't been able to get it to work. It gets to 'Dll successfully injected' after launching the game then times out with no key.bin being created. Game also doesn't close with error message as described in readme...

Since I don't know how to make an exe run with locale emulator from a bat file I've changed the administrative language for non-Unicode programs to Japanese (which seems to work fine for these games), could that be messing it up? I'm also using a virtual machine to run all this (dll injectors are spooky) but cant imagine that being an issue...

Is there another way to make games like these run without locale emulator?

Also the vm being used has no antivirus and windows real time protection disabled.

My own troubleshooting hasn't produced any fruit so here's what I'm most interested in extracting:

You must be registered to see the links

(I pm'ed you)

P.S.
What python IDE do you use?

 

[samuel1154]

Did not have much luck myself trying the 2nd extraction tool Seems like pixel-teishoku games are destined to remain untranslated with how much of a pain in the ass they are to access.

 

[samuel1154]

"Bumping" this I guess just to maybe keep it active, still really want to see these games translated someday.