Tool Japanese translator based on the OpenIA GPT 3.5 Model and image analysis (OCR)

[K-RT]

I bring you a new open source tool that I have developed which serves to translate Japanese texts into another language using the GPT 3.5 Model. The system allows capturing images from the screen and through visual analysis extracts the Japanese characters (OCR). Then these texts are translated. It can be used to translate manga, video games, or any other source of text

Spoiler: Preview

You don't have permission to view the spoiler content. Log in or register now.

The OpenAI model is not perfect yet, but using the system correctly it has a considerably better performance than traditional translators. Clearly, it cannot be compared to the ability of a human yet... but at this rate, everything may change by the end of the year.

You can find the tool in this

You must be registered to see the links

repository or on my

You must be registered to see the links

.

My goal is to redo a project that I started last year but abandoned due to technical limitations: A system for real-time translation of video games. I am experimenting with the combination of various OCRs and translators in parallel and the use of GPT for the combination of these results, and it looks promising. The big difference with traditional systems is that now we can use the context of a text to enrich and adjust translations.

 

[Extra Lewd]

Looks interest.
Your OCR could be external service?
Like:
- I want get text from X image
- I use OCR for get text
- I use my translate tool (with any language I want)
- I use tool for replace text by translated

 

[K-RT]

I will respond to you in two ways:

1. Yes. If you enter the repo, you can download and use only the backend part (local FastAPI server). This server has the functionality of extracting text from an image. Then, you can use your own translator.
   
   
2. My idea is to add various OCRs and translators to the tool (to support multiple languages and character formats) so that users can select one (or multiple) that best suits their use case. Then, GPT can be very useful for combining these results and generating an even better final result.

Regarding the points you mentioned, the first and second ones can already be performed by VGT. For the third point, if you tell me which translator you would like to have, I can add it as an option in the next version. The fourth point is outside the scope of VGT, as that requires a different type of work on the image

 

[Obiyo]

Try

You must be registered to see the links

for reference. I use it translate doujin/manga with Sugoi model for offline usage.
Never tried GPT model but it's there if you want to try.
Here's my command that I used

Spoiler: cmd

You don't have permission to view the spoiler content. Log in or register now.

 

[shiny-kuki]

Very cool!