Skip to content

Commit

Permalink
feat: add OpenAI translator (#519)
Browse files Browse the repository at this point in the history
  • Loading branch information
jmeiss authored Aug 29, 2023
1 parent 734b31f commit a030090
Show file tree
Hide file tree
Showing 6 changed files with 139 additions and 1 deletion.
22 changes: 22 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -117,6 +117,17 @@ $ i18n-tasks translate-missing --backend=yandex
$ i18n-tasks translate-missing --from=en es fr
```

### OpenAI Translate missing keys

Translate missing values with OpenAI ([more below on the API key](#openai-translation-config)).

```console
$ i18n-tasks translate-missing --backend=openai

# accepts from and locales options:
$ i18n-tasks translate-missing --from=en es fr
```

### Find usages

See where the keys are used with `i18n-tasks find`:
Expand Down Expand Up @@ -424,6 +435,17 @@ translation:
yandex_api_key: <Yandex API key>
```

<a name="openai-translation-config"></a>
### OpenAI Translate

`i18n-tasks translate-missing` requires a OpenAI API key, get it at [OpenAI](https://openai.com/).

```yaml
# config/i18n-tasks.yml
translation:
openai_api_key: <OpenAI API key>
```

## Interactive console

`i18n-tasks irb` starts an IRB session in i18n-tasks context. Type `guide` for more information.
Expand Down
6 changes: 6 additions & 0 deletions config/locales/en.yml
Original file line number Diff line number Diff line change
Expand Up @@ -112,6 +112,12 @@ en:
missing:
details_title: Value in other locales or source
none: No translations are missing.
openai_translate:
errors:
no_api_key: >-
Set OpenAI API key via OPENAI_API_KEY environment variable or translation.openai_api_key
in config/i18n-tasks.yml. Get the key at https://openai.com/.
no_results: OpenAI returned no results.
remove_unused:
confirm:
one: "%{count} translation will be removed from %{locales}."
Expand Down
6 changes: 6 additions & 0 deletions config/locales/ru.yml
Original file line number Diff line number Diff line change
Expand Up @@ -111,6 +111,12 @@ ru:
missing:
details_title: На других языках или в коде
none: Всё переведено.
openai_translate:
errors:
no_api_key: |-
Установить ключ API Яндекса с помощью переменной среды OPENAI_API_KEY или translation.openai_api_key
в config / i18n-tasks.yml. Получите ключ по адресу https://openai.com/.
no_results: Яндекс не дал результатов.
remove_unused:
confirm:
few: Переводы (%{count}) будут удалены из %{locales}.
Expand Down
1 change: 1 addition & 0 deletions lib/i18n/tasks/configuration.rb
Original file line number Diff line number Diff line change
Expand Up @@ -66,6 +66,7 @@ def translation_config
conf[:deepl_api_key] = ENV['DEEPL_AUTH_KEY'] if ENV.key?('DEEPL_AUTH_KEY')
conf[:deepl_host] = ENV['DEEPL_HOST'] if ENV.key?('DEEPL_HOST')
conf[:deepl_version] = ENV['DEEPL_VERSION'] if ENV.key?('DEEPL_VERSION')
conf[:openai_api_key] = ENV['OPENAI_API_KEY'] if ENV.key?('OPENAI_API_KEY')
conf[:yandex_api_key] = ENV['YANDEX_API_KEY'] if ENV.key?('YANDEX_API_KEY')
conf
end
Expand Down
5 changes: 4 additions & 1 deletion lib/i18n/tasks/translation.rb
Original file line number Diff line number Diff line change
Expand Up @@ -2,20 +2,23 @@

require 'i18n/tasks/translators/deepl_translator'
require 'i18n/tasks/translators/google_translator'
require 'i18n/tasks/translators/openai_translator'
require 'i18n/tasks/translators/yandex_translator'

module I18n::Tasks
module Translation
# @param [I18n::Tasks::Tree::Siblings] forest to translate to the locales of its root nodes
# @param [String] from locale
# @param [:deepl, :google, :yandex] backend
# @param [:deepl, :openai, :google, :yandex] backend
# @return [I18n::Tasks::Tree::Siblings] translated forest
def translate_forest(forest, from:, backend: :google)
case backend
when :deepl
Translators::DeeplTranslator.new(self).translate_forest(forest, from)
when :google
Translators::GoogleTranslator.new(self).translate_forest(forest, from)
when :openai
Translators::OpenAiTranslator.new(self).translate_forest(forest, from)
when :yandex
Translators::YandexTranslator.new(self).translate_forest(forest, from)
else
Expand Down
100 changes: 100 additions & 0 deletions lib/i18n/tasks/translators/openai_translator.rb
Original file line number Diff line number Diff line change
@@ -0,0 +1,100 @@
# frozen_string_literal: true

require 'i18n/tasks/translators/base_translator'

module I18n::Tasks::Translators
class OpenAiTranslator < BaseTranslator
# max allowed texts per request
BATCH_SIZE = 50

def initialize(*)
begin
require 'openai'
rescue LoadError
raise ::I18n::Tasks::CommandError, "Add gem 'ruby-openai' to your Gemfile to use this command"
end
super
end

def options_for_translate_values(from:, to:, **options)
options.merge(
from: from,
to: to
)
end

def options_for_html
{}
end

def options_for_plain
{}
end

def no_results_error_message
I18n.t('i18n_tasks.openai_translate.errors.no_results')
end

private

def translator
@translator ||= OpenAI::Client.new(access_token: api_key)
end

def api_key
@api_key ||= begin
key = @i18n_tasks.translation_config[:openai_api_key]
fail ::I18n::Tasks::CommandError, I18n.t('i18n_tasks.openai_translate.errors.no_api_key') if key.blank?

key
end
end

def translate_values(list, from:, to:)
results = []

list.each_slice(BATCH_SIZE) do |batch|
translations = translate(batch, from, to)

results << JSON.parse(translations)
end

results.flatten
end

def translate(values, from, to) # rubocop:disable Metrics/MethodLength
messages = [
{
role: 'system',
content: "You are a helpful assistant that translates content from the #{from} to #{to} locale in an i18n
locale array. The array has a structured format and contains multiple strings. Your task is to translate
each of these strings and create a new array with the translated strings. Keep in mind the context of all
the strings for a more accurate translation.\n"
},
{
role: 'user',
content: "Translate this array: \n\n\n"
},
{
role: 'user',
content: values.to_json
}
]

response = translator.chat(
parameters: {
model: 'gpt-3.5-turbo',
messages: messages,
temperature: 0.7
}
)

translations = response.dig('choices', 0, 'message', 'content')
error = response['error']

fail "AI error: #{error}" if error.present?

translations
end
end
end

0 comments on commit a030090

Please sign in to comment.