Tool RPGM SLR - Offline JP to EN Translation for RPG Maker VX, VX Ace, MV, MZ, and Pictures

[Shisaye]

However, after running the tests (of that part), I realized that previous translations only are called if the actual translation failed.
Is this expected normal operation? Wouldn't the LLM benefit always from having access to previous (at least the inmediate previous) translations? I've read from options that too much past context can break the LLM, but always having a single previous batch maybe it's good? I guess not if it's not done that way (I did understand later that this is the single-line translation mode for when the whole batch doesn't work).

DSLR is designed to work with stuff like DeepSeekV3.
Always including a previous full batch would essentially double the money spent, because of that it's only used when the LLM is in a failure loop of providing an unusable result.
As for single translations, the first attempt is automatically skipped if a full batch was accepted.
No request to the LLM is actually made on that one, it just goes through the loop as if the line from the full batch was the first LLM response, checking if it fits the requirements of a usable translation, and it only makes a request on round 2 if it fails the checks.
On round 3 it includes failed attempts to try to get the LLM out of a failure loop.
The "systemContent" should include the context successful translations even on a round 1 request though. If not something would be very wrong here. The very first request will not actually have any context though because DSLR cannot see cells that weren't actually selected for translation.

(I really, really hate regex)

No, no, no.
Regex is love, regex is life!
Every morning you wake up thanking the lord for the eternal happiness and inner fulfillment bestowed upon you by regex puzzles.

I haven't been able to fully test it because my LLM decided to work flawlessly and is getting late. I hope something is usable. Feel free to mark it as "Experimental" or dismiss it all together if you don't feel confident.

Is there a rush? Just tell me when you're fully happy with it, and I'll include the new option with a note that all complaints are to be forwarded to you.

 

[RTS23]

Understandable.

I would never be satisfied, so I must stop at some time. I added an option to add comments, and rewrote the option indications because of the many rules of the context format. For all I tested, it seemed to work, but who knows if I missed some obscure case of "It's not parsing this character term because his names is $[*&%]!."

The sooner I finish modifying the tool, the sooner I start using it. (and seriously, this crap of special symbols should never have been a problem, but we live in a world of degenerates. Escaping special characters should be something done by the language compiler, not the user/programmer).

 

[fantasmic]

I have a report!
Translating RJ234574, and there's some text the parser doesn't see.

Spoiler: An example

You don't have permission to view the spoiler content. Log in or register now.

I found the entries though: they're notes in Enemies.json (of all places...). The entries in the translated json look like this:

"name":"Devil Pet Operation","note":"<a:Ⅼ０１>\n<b:宿直室>\n<c:ペット探し>\n<d:ストレス-10>\n<e:寂しいっす。癒やしが欲しいっす。\n野良猫か野良犬を捕まえてきて欲しいっす。\nペットにするっす。宿直室に来てほしいっす>\n",

Considering this is an MV game from six years ago and it's my first time seeing this, it's probably just some niche plugin and not really a pressing issue, particularly since the json itself is easy to edit. Still, who knows who else is hiding stuff in the "notes" section of enemy listings.
Neither SLR versions 1.144 nor 2.000 grabbed the entries, but I didn't see anything in the v2.001 patch notes relating to this so I (still) didn't update my install yet.

 

[RTS23]

I found the entries though: they're notes in Enemies.json (of all places...). The entries in the translated json look like this:

Considering this is an MV game from six years ago and it's my first time seeing this, it's probably just some niche plugin and not really a pressing issue, particularly since the json itself is easy to edit. Still, who knows who else is hiding stuff in the "notes" section of enemy listings.
Neither SLR versions 1.144 nor 2.000 grabbed the entries, but I didn't see anything in the v2.001 patch notes relating to this so I (still) didn't update my install yet.

Maybe an unnecesary question, but what color are these lines tagged? If they're black, then my guess is that the text sanitation isn't prepared for those kind of lines. If they're of pretty much any other color, then it probably isn't expected to be translated by the tool with the default options. Check the manual.html/"Core Concepts"/Understanding Cell Color Tags" for the color code explainations. (On this line, maybe the option already exists and I didn't see it but if not, I would like to be able to search color tags.)

Anyways, you can still try to select manually those lines and right-click translate them (applying a different color tag blacklist than the default), though you'll have most probably to manually update the fields with text you see on the game, keeping the original code so it doesn't break.

I guess these kind of things are found from time to time.

 

[fantasmic]

RTS23 The lines don't show up in the project at all, so I had to hunt them down manually in the json files. It makes sense to skip the "Notes" section for enemies since there usually isn't anything important in there, unlike the notes sections for skills and items, which are often used by scripts to expand rpgm's functionality. Like, even in this case, the notes aren't even being used for actual enemies.

 

[RTS23]

RTS23 The lines don't show up in the project at all, so I had to hunt them down manually in the json files. It makes sense to skip the "Notes" section for enemies since there usually isn't anything important in there, unlike the notes sections for skills and items, which are often used by scripts to expand rpgm's functionality. Like, even in this case, the notes aren't even being used for actual enemies.

Oh, that's something completely different of what I was thinking reading "neither SLR versions 1.144 nor 2.000 grabbed the entries". The tool literally didn't grab the entries from the original game files, so they do not appear in the spreadsheet.
For what I've seen using the tool in a single game, notes are parsed but tagged in blue iirc, so they don't get translated by default but parsed in case the user wants to know some developer context. Thus, I consider that both notes and lines printed in the game menu should be parsed, it's just that notes should be tagged in color and the text in black so it gets translated.

I'm talking without any knowledge, so I may be wrong, but I would say that's probably on "www\addons\rmmv\rmmv.js". I gave it a look, but didn't see at first look anything that may try to get that, so there's 2 possibilities; (1) that script takes and parses the missed text to another script where it gets mistakenly trimmed, or (2) I missed it because I don't understand anything.

If you wanna try to make a patch, you can use the F12 console when generating the spreadsheet from the game files to maybe find our easier the pipeline it follows. At some point of the path this .json is being parsed trimming the part that should pass to the spreadsheet, and the other way around when generating the translated files after finishing the translation.

 

[Shisaye]

The parsers are originally based on by now very old ones by dreamsavior, which I have changed over the years. One of those changes was, that I made it stop collecting never relevant garbage from animations and tilesets.
But apparently dreamsavior had lumped enemies into the same case operation. So it also stopped collecting certain entries from enemies, which until now fell under the radar because I don't really remember a project in which a dev actually put relevant notes into the enemy json.

I'll fix it next update, but just so RTS23 can sleep at night, the relevant bit is at around line 1260 and to make it collect those entires you would just add this after the 'case "enemies":' line, which will also fix it for "classes":

thisFetch = rmmvData.fetchCommonData($currentData,["name", "note"], [$fileInfo['filename']]);
$RESULT[$path] = thisFetch;
break;

Edit: in case you're wondering why it was still collecting "name" entries, that's because it was now using the "mapinfos" case operation further down.

 

[Shisaye]

Understandable.

I would never be satisfied, so I must stop at some time. I added an option to add comments, and rewrote the option indications because of the many rules of the context format. For all I tested, it seemed to work, but who knows if I missed some obscure case of "It's not parsing this character term because his names is $[*&%]!."

The sooner I finish modifying the tool, the sooner I start using it. (and seriously, this crap of special symbols should never have been a problem, but we live in a world of degenerates. Escaping special characters should be something done by the language compiler, not the user/programmer).

Erm... you modified included the TranslationEngineWrapper here, which is the wrapper of SugoiV4 SLR even though you added an option for DSLR.

TranslationEngineWrapper = SLR
TranslationEngineWrapper2 = Google, Bing, etc.
DSLRTranslationEngineWrapper = DSLR

Since I'm guessing you made no changes except the option menu order I'll just make that change myself.

Edit: Wait... SLR doesn't have a set options menu like DSLR, that's something I only added for the new engine.
What the heck did you do?

Edit2: If I use the ComparePlus plugin on it it says the files are identical.
...Did you include the wrong file?

 

[RTS23]

Erm... you modified included the TranslationEngineWrapper here, which is the wrapper of SugoiV4 SLR even though you added an option for DSLR.

TranslationEngineWrapper = SLR
TranslationEngineWrapper2 = Google, Bing, etc.
DSLRTranslationEngineWrapper = DSLR

Since I'm guessing you made no changes except the option menu order I'll just make that change myself.

Edit: Wait... SLR doesn't have a set options menu like DSLR, that's something I only added for the new engine.
What the heck did you do?

Edit2: If I use the ComparePlus plugin on it it says the files are identical.
...Did you include the wrong file?

This is my power... blindness allows you to make errors that nobody would... like sharing an unmodified file as "modified."

You're right, I did only modify TranslationEngineWrapper.json to add `'systemPromptadditionTrim',` at line 1405, just between `'systemPromptaddition',` and `'temperature',`, for enabling the option at Options.

And don't worry, I will not sleep anyways. I'm writing the monospace word wrapper while taking into account some characters that print using two monospaces. I'll share it as a script within the extra red button from the toolbar. You then decide if to embedded it onto your "Word wrapper" function as "m" (of monospace). That way it's somewhat integrated in the tool though, of course, it will not follow your smart word wrapping.

 

[Shisaye]

I've released v2.002.

Fixed the note stuff not being collected in enemies and classes.
Included the rts trim option for someone is unable to increase context length of their used LLM for whatever reason, or wants to shave off some tokens for premium models.

 

[Shisaye]

This is my power... blindness allows you to make errors that nobody would... like sharing an unmodified file as "modified."

You're right, I did only modify TranslationEngineWrapper.json to add `'systemPromptadditionTrim',` at line 1405, just between `'systemPromptaddition',` and `'temperature',`, for enabling the option at Options.

And don't worry, I will not sleep anyways. I'm writing the monospace word wrapper while taking into account some characters that print using two monospaces. I'll share it as a script within the extra red button from the toolbar. You then decide if to embedded it onto your "Word wrapper" function as "m" (of monospace). That way it's somewhat integrated in the tool though, of course, it will not follow your smart word wrapping.

...but why?
Making that wrapping system took a long time and it actually considers a lot of stuff. It even has a section to wrap certain scripts with specific formatting requirements and ... stuff.

 

[RTS23]

...but why?
Making that wrapping system took a long time and it actually considers a lot of stuff. It even has a section to wrap certain scripts with specific formatting requirements and ... stuff.

Because there isn't a monospace option, no? I tried with all options listed in your word wrapper, but neither seemed to work in my case. Also, having a log of warnings and errors of the file and row with problems helps a lot in review. For instance, if the message doesn't fit into the message window, have some log that it warns you.

Here's my monospace word wrapper. It's dumb, but it tries it's best. It has the user variables at the start, so they can be modified easily. The idea is to put all console logs to also the window as any other function, but I'll leave that to the maybees.

Tried to do smart punctuation, realized spaces weren't counting correctly, corrected it so now it should, though there's no punctuation logic whatsoever other than "please, don't line break parenthesis." The console shows the errors and warnings, telling if a odd number of parenthesis symbols have been detected or it had to line break text between parenthesis (my LLM decided to not follow syntax rules, and added random 「 at the start of many sentences).
I also tried to regex parenthesis that do not have a closure/start. Without the closure is easy enough, /\[(?![^\[\]]*\])/ for ( without ) and \[(?![^\[\]]*\]) for [ without ]. But found out that regex doesn't allow for unlimited lookback, so I didn't find any way of selecting only the parenthesis/bracket that do not have a start. That's the reason I count in my script any parenthesis symbol equivalent, which is not good, but didn't find any other easy way.
Oh, and regex with global flag is poison. It looks great until it doesn't find something is there because of a random internal index that seems that isn't reseting, so it then depends of previous searches with the same regex...


Feel free to discard it if you see it usless or if yours does a better job (probably).
I just wanted something that works (and if it doesn't warns me) and avoids me having to check with scripts and redoing work.

 

[Shisaye]

I also tried to regex parenthesis that do not have a closure/start. Without the closure is easy enough, /\[(?![^\[\]]*\])/ for ( without ) and \[(?![^\[\]]*\]) for [ without ]. But found out that regex doesn't allow for unlimited lookback, so I didn't find any way of selecting only the parenthesis/bracket that do not have a start. That's the reason I count in my script any parenthesis symbol equivalent, which is not good, but didn't find any other easy way.

What are you talking about?
What are you even trying to do?
If the lookahead/lookbehind stuff causes issues, why not use capture groups instead?

Edit:
Just in case you don't know how capture groups work in javascript.
If this is the code:

let text = "text text text]";
let text2 = "[text text text]";
let regex = /^[^\[]*(\])/;
let matchthing = text.match(regex);
let matchthing2 = text2.match(regex);

Then "matchthing[1]" is "]" and "matchthing2[1]" is "null".

 

[RTS23]

What are you talking about?
What are you even trying to do?
If the lookahead/lookbehind stuff causes issues, why not use capture groups instead?

Edit:
Just in case you don't know how capture groups work in javascript.
If this is the code:

let text = "text text text]";
let text2 = "[text text text]";
let regex = /^[^\[]*(\])/;
let matchthing = text.match(regex);
let matchthing2 = text2.match(regex);

Then "matchthing[1]" is "]" and "matchthing2[1]" is "null".

I tried that and ****ing regex101 doesn't match anything but javascript does.
How can I not hate the regex father, the special characters son and the javascript spirit?

Anyway, javascript does match it, but it does only if it doesn't have the global flag (I really don't understand anything now, I though the global flag is what allowed multiple matches, but I guess it's getting group and match and is not really multiple "matches"), but it still only gets a single "]" (`let text = "text text text] ]";` only returns one "]") and it return null if there is any "[" in the line (EDIT: if there's any "[" before of all "]" without start) (eg, `let text = "text [text text] ]";` returns null).
That's what lookbehind is supposed to solve, but due to the constrained limit to look back, I haven't been able to find a suitable regex.

PD: Mine for matching brackets without end /\[(?![^\[\]]*\])/g gets all brackets that do not have an ending bracket in a single execution (well, is global flag, so it probably has some bug with the internal counter). Of course, manual inspection is required since you cannot trust the translator to make the last bracket be the correct one (there've been instances where it changed the brackets to apostrophes and put the brackets to the whole sentece/row, local LLMs of relatively low size have a long way).

I'm trying to do 2 things:
1) Monospace word wrapper for games with monospace font (fixed 60/56 characters, with the exception of some characters that take 2 for some reason), that ensures no text goes over the window (I'm at a point that I don't care for fancyness), and if it can't, report it somewhere (log, window screen, some color tag or way to search it easily, anything). That is what my script does (if I didn't miss any special case after the validation).
2) Regex for finding bracket (any kind of bracket) anomalies, like open but not closed and the other way around. That could also include when the original text has brackets and the translation lacks. Some kind of warning or report. I can do this with Automation scripts for now.

 

[Shisaye]

2) Regex for finding bracket (any kind of bracket) anomalies, like open but not closed and the other way around. That could also include when the original text has brackets and the translation lacks. Some kind of warning or report. I can do this with Automation scripts for now.

I still don't understand what you are trying to do. What does this have to do with word wrapping?

 

[RTS23]

Oh nothing, that's why is in it's own point. It's small things I observed from the translation and want to make a script for at least personal use. And curiosity on how things can be done with regex.
I'll probably have to make an Automation script that goes letter by letter like the monospace word wrapper to do a better job of finding parenthesis conflicts adn the such.

 

[Shisaye]

Oh nothing, that's why is in it's own point. It's small things I observed from the translation and want to make a script for at least personal use. And curiosity on how things can be done with regex.
I'll probably have to make an Automation script that goes letter by letter like the monospace word wrapper to do a better job of finding parenthesis conflicts adn the such.

Currently most outer bracket types are never sent to the LLM. They are removed beforehand and later added again.
To be exact these ones at the start:

〔⟨〖〘〚〝〞〟｢〈《「『【（［[({＜<｛｟'´`“

These ones at the end:

])}〕〗〙〛〝〞”｣〉》」』】）］＞>｝｠〟⟩'`´

If the LLM adds outer brackets, those are almost always wrong, and you could just add a step to DSLR to remove them before the original brackets are added back in. I didn't do something like that, yet, because I've never seen DeepSeekV3 do that and I'm not 100% sure there's never a situation in which someone would want them.