Paarthurnax

A Golang translator

Utilisation

Init

Crée les fichiers nécessaire pour traquer les segments dans le temps

Normalize

Normalise les fichiers de traduction afin de réduire le bruit dans les reviews futurs

Translate

Traduit dans toutes les langues définit via DeepL

Init

$ paarthurnax init

2024/09/23 15:44:38 Generating state from disk...
2024/09/23 15:44:38 Processing config/locales/a_new_directory/fr.yml
2024/09/23 15:44:38 Processing config/locales/component/fr.yml
2024/09/23 15:44:38 Processing config/locales/fr.yml
2024/09/23 15:44:38 Persisting state...

Charge les fichiers de traduction source
Génere un Hash pour chaque clé basée sur son contenu
Sauvegarde un fichier d'état .paarthurnax

Init

Quand?

Une seule fois a l'installation de Paarthurnax dans le projet

Prérequis?

Clean state (pas de traductions en attente)

Normalize

$ paarthurnax normalize

2024/09/23 15:59:11 Loading current state from disk...
2024/09/23 15:59:11 Normalizing translations...
2024/09/23 15:59:11 Processing config/locales/a_new_directory/fr.yml...
2024/09/23 15:59:11 Processing config/locales/component/fr.yml...
2024/09/23 15:59:11 Processing config/locales/fr.yml...
2024/09/23 15:59:11 Done!

Charge les fichiers de destination de traduction
Les réécrit directement avec le même système que lors de prochaines traductions

normalize

Quand?

A n'importe quel instant

Prérequis?

Aucun

translate

$ DEEPL_API_KEY=[...] paarthurnax translate

2024/09/23 16:00:41 Loading previous state from disk...
2024/09/23 16:00:41 Generating current state from disk...
2024/09/23 16:00:41 Reconciling states...
2024/09/23 16:00:41 Processing config/locales/a_new_directory/fr.yml
2024/09/23 16:00:41 Processing config/locales/component/fr.yml
2024/09/23 16:00:41 Processing config/locales/fr.yml
2024/09/23 16:00:41 Cleaning up removed files...
2024/09/23 16:00:41 Persisting new state...

Charge les fichiers de traduction source
Charge l'état précédent
Traduit touts les segments modifiés avec DeepL
Réécrit tous les fichiers de traduction cible modifiés

Init

Quand?

A chaque étape de traductio

Prérequis?

Présence du fichier .paarthurnax
Présence de la clé d'api DeepL FREE en variable d'env

Fonctionnement

// Variables
var i string
i := 42

// Fonctions
func (reciver File) Name(param string) (string, error) {
  [...]
}

// Types
type File struct {
  I int
}

Fonctionnement

type State struct {
	Files []translationFile.TranslationFile
}

type TranslationFile struct {
	Path           string
	SegmentsHashes map[string]string // Key: sha1
}

Chargement du state depuis les locales

Fonctionnement

Chargement du state depuis les locales

e := filepath.WalkDir(path, func(path string, d fs.DirEntry, err error) error {
		if err == nil && d.Name() == "fr.yml" {
			if verbose {
				log.Println(fmt.Sprintf("Processing %s", path))
			}

			t, err := translation.Load(path)
			if err != nil {
				return errors.New(fmt.Sprintf("Failed to load translation: %s", err))
			}

			state.Files = append(state.Files, translationFile.TranslationFile{
				Path:           path,
				SegmentsHashes: HashTranslation(t.FlattenedSegments()),
			})
		}
		return err
	})
	if e != nil {
		return nil, errors.New("Unable to process the filestructure: " + e.Error())
	}

Fonctionnement

func Load(path string) (*TranslationFile, error) {
	data, err := os.ReadFile(path)
	if err != nil {
		return nil, errors.New("Error reading file:" + err.Error())
	}

	var yamlData map[string]interface{}
	err = yaml.Unmarshal(data, &yamlData)
	if err != nil {
		return nil, errors.New("Error unmarshaling YAML:" + err.Error())
	}

	if len(yamlData) != 1 {
		return nil, errors.New("the provided YAML file is not a valid translation file")
	}

	locale := utils.MapKeys(yamlData)[0]
	file := TranslationFile{Path: path, Locale: locale, Segments: yamlData[locale].(map[string]interface{})}

	return &file, nil
}

Fonctionnement

Chargement du state depuis les locales

e := filepath.WalkDir(path, func(path string, d fs.DirEntry, err error) error {
		if err == nil && d.Name() == "fr.yml" {
			if verbose {
				log.Println(fmt.Sprintf("Processing %s", path))
			}

			t, err := translation.Load(path)
			if err != nil {
				return errors.New(fmt.Sprintf("Failed to load translation: %s", err))
			}

			state.Files = append(state.Files, translationFile.TranslationFile{
				Path:           path,
				SegmentsHashes: HashTranslation(t.FlattenedSegments()),
			})
		}
		return err
	})
	if e != nil {
		return nil, errors.New("Unable to process the filestructure: " + e.Error())
	}

Fonctionnement

Chargement du state depuis les locales

func (translation *TranslationFile) FlattenedSegments() map[string]string {
	return extractFlattenedSegments(translation.Segments)
}

func extractFlattenedSegments(m map[string]interface{}) map[string]string {
	result := make(map[string]string)

	for key, value := range m {
		switch value.(type) {
		case string:
			result[key] = value.(string)
		case map[string]interface{}:
			for subKey, subValue := range extractFlattenedSegments(value.(map[string]interface{})) {
				result[key+"."+subKey] = subValue
			}
		}
	}

	return result
}

Fonctionnement

Chargement du state depuis les locales

func HashTranslation(m map[string]string) map[string]string {
	result := make(map[string]string)

	for k, v := range m {
		h := sha1.New()
		h.Write([]byte(v))
		result[k] = hex.EncodeToString(h.Sum(nil))
	}

	return result
}

Fonctionnement

Chargement du state depuis les locales

Fonctionnement

Chargement du state depuis les locales

[[Files]]
Path = 'config/locales/a_new_directory/fr.yml'

[Files.SegmentsHashes]
a_key = '2a6dcab3038a7c1d15b427378a3483952761df8c'

[[Files]]
Path = 'config/locales/component/fr.yml'

[Files.SegmentsHashes]
'a_super.deep.key' = 'e50fc81a49f34d67292cdfd88a6e1737be1677a6'

[[Files]]
Path = 'config/locales/fr.yml'

[Files.SegmentsHashes]
'a.deep.b' = '6dcd4ce23d88e2ee9568ba546c007c63d9131c1b'
first_level_key = '153d6addf72e832f95dd72ba7f891ccb47d6d4d2'
likes = '06cee428e9b518ed6181eb637a6a6175fa8d725f'
remote_status = 'ca4830751faf5e537a84eb297e53441a599c7681'
'second_level.plural_key.one' = '137eca41db3d9cb9591ad89ceb5e78379017b67f'
'second_level.plural_key.other' = 'a1ba1d821b7e573d90064bcfe2fd080ff625a0ad'
'second_level.plural_key.zero' = 'ddcc9ab409b7ba7733d96f07e981dda1102b21a6'
'second_level.second_level_key' = '6f9588ba7a1bb1d5a7c987cc9c1211fe81f7a15f'

Fonctionnement

Normalisation des traductions

log.Println("Loading current state from disk...")
nState, err := state.LoadFromDisk("config/locales", false)
if err != nil {
    log.Fatal(err)
}

log.Println("Normalizing translations...")
for _, nFile := range nState.Files {
	log.Println(fmt.Sprintf("Processing %s...", nFile.Path))

	for _, locale := range translationGroup.DestLocales {
		path := strings.Replace(nFile.Path, "fr.yml", locale+".yml", 1)
		file, err := translation.LoadOrCreate(path)
		if err != nil {
			log.Fatal(fmt.Sprintf("%s: %s", path, err.Error()))
		}
		if err = file.Save(); err != nil {
			log.Fatal(err)
		}
	}
}

Fonctionnement

Traduction

log.Println("Loading previous state from disk...")
pState, err := state.Load()
if err != nil {
	log.Fatal(err)
}

log.Println("Generating current state from disk...")
nState, err := state.LoadFromDisk("config/locales", false)
if err != nil {
	log.Fatal(err)
}

Fonctionnement

Traduction

log.Println("Reconciling states...")
for _, nFile := range nState.Files {
	log.Println(fmt.Sprintf("Processing %s", nFile.Path))
	changes := nFile.Changes(pState.GetFile(nFile.Path))
	if len(changes) != 0 {
		log.Println(fmt.Sprintf("Applying changes and translating %s...", nFile.Path))
		group, err := translationGroup.New(nFile.Path)
		if err != nil {
			log.Fatal(err)
		}
		if err = group.Apply(changes); err != nil {
			log.Fatal(err)
		}
	}
}

Fonctionnement

Traduction

const (
	Added   = 0
	Removed = 1
	Updated = 2
)

type Change struct {
	Kind int8
	Path string
}

func (file *TranslationFile) Changes(other *TranslationFile) []Change {
	var changes []Change

	for path, hash := range file.SegmentsHashes {
		if other == nil { // When the other translation state does not exist (new source file)
			changes = append(changes, Change{Kind: Added, Path: path})
		} else {
			oHash, ok := other.SegmentsHashes[path]
			if !ok {
				changes = append(changes, Change{Kind: Added, Path: path})
			} else if hash != oHash {
				changes = append(changes, Change{Kind: Updated, Path: path})
			}
		}
	}

	if other != nil {
		for path, _ := range other.SegmentsHashes {
			if _, ok := file.SegmentsHashes[path]; !ok {
				changes = append(changes, Change{Kind: Removed, Path: path})
			}
		}
	}

	return changes
}

Fonctionnement

Traduction

log.Println("Reconciling states...")
for _, nFile := range nState.Files {
	log.Println(fmt.Sprintf("Processing %s", nFile.Path))
	changes := nFile.Changes(pState.GetFile(nFile.Path))
	if len(changes) != 0 {
		log.Println(fmt.Sprintf("Applying changes and translating %s...", nFile.Path))
		group, err := translationGroup.New(nFile.Path)
		if err != nil {
			log.Fatal(err)
		}
		if err = group.Apply(changes); err != nil {
			log.Fatal(err)
		}
	}
}


var DestLocales = [...]string{"en", "es", "it", "de", "hu", "pt", "pl", "ro", "uk"}

type TranslationGroup struct {
	Path   string
	source *translation.TranslationFile
	files  []*translation.TranslationFile
}

Fonctionnement

Traduction

func (group *TranslationGroup) Apply(changes []translationFile.Change) error {
	translator, err := deepl.New(os.Getenv("DEEPL_API_KEY"), "api-free.deepl.com")
	if err != nil {
		return errors.New("Unable to initialize the translation engine: " + err.Error())
	}
	for _, change := range changes {
		keyParts := strings.Split(change.Path, ".")
		if utils.Includes(keyParts[len(keyParts)-1], []string{"zero", "one", "other"}) {
			if err := handlePluralSegment(keyParts[len(keyParts)-1], group, &change, translator); err != nil {
				return err
			}
		} else {
			if err := handleStandaloneSegment(group, &change, translator); err != nil {
				return err
			}
		}
	}

	for _, file := range group.files {
		if err := file.Save(); err != nil {
			return errors.New("Failed to save updated translation file " + file.Path + ": " + err.Error())
		}
	}

	return nil
}

Fonctionnement

Traduction

func handleStandaloneSegment(group *TranslationGroup, change *translationFile.Change, translator *deepl.Deepl) error {
	for _, file := range group.files {
		if change.Kind == translationFile.Added || change.Kind == translationFile.Updated {
			value, err := group.source.GetSegmentValueAt(change.Path)
			if err != nil {
				return errors.New("Unable to get the value of the source segment " + change.Path + ": " + err.Error())
			}
			valueForTranslation, ctx, err := prepareForTranslation(value)
			if err != nil {
				return errors.New("Unable to prepare for translation " + change.Path + ": " + err.Error())
			}
			translation, err := translator.Translate(valueForTranslation, "fr", file.Locale)
			if err != nil {
				return errors.New("Unable to translate the value of the source segment " + change.Path + ": " + err.Error())
			}
			translation, err = revertTranslationPreparation(translation, ctx)
			if err != nil {
				return errors.New("Unable to revert translation preparation for " + change.Path + ": " + err.Error())
			}

			if err = checkVariableEquity(value, translation); err != nil {
				return errors.New("The translation does not contain the required variables: " + err.Error())
			}

			if err = file.SetSegmentValueAt(change.Path, translation); err != nil {
				return errors.New("Unable to set the value of the source segment " + change.Path + " in locale " + file.Locale + ": " + err.Error())
			}
		} else if change.Kind == translationFile.Removed {
			if err := file.RemoveSegmentAt(change.Path); err != nil {
				return errors.New("Unable to remove segment " + change.Path + " in locale " + file.Locale + ": " + err.Error())
			}
		}
	}
	return nil
}

Fonctionnement

Traduction

func prepareForTranslation(text string) (string, map[string]string, error) {
	ctx := make(map[string]string)
	r, err := regexp.Compile("%{(?P<variable>[a-zA-Z_ -]+)}")
	if err != nil {
		return "", nil, errors.New("Unable to prepare variable extraction: " + err.Error())
	}
	for _, variable := range r.FindAllStringSubmatch(text, -1) {
		ctx[variable[1]] = "<span translate=\"no\">" + variable[1] + "</span>"
		text = strings.Replace(text, "%{"+variable[1]+"}", ctx[variable[1]], -1)
	}
	return text, ctx, nil
}

Fonctionnement

Traduction

func handleStandaloneSegment(group *TranslationGroup, change *translationFile.Change, translator *deepl.Deepl) error {
	for _, file := range group.files {
		if change.Kind == translationFile.Added || change.Kind == translationFile.Updated {
			value, err := group.source.GetSegmentValueAt(change.Path)
			if err != nil {
				return errors.New("Unable to get the value of the source segment " + change.Path + ": " + err.Error())
			}
			valueForTranslation, ctx, err := prepareForTranslation(value)
			if err != nil {
				return errors.New("Unable to prepare for translation " + change.Path + ": " + err.Error())
			}
			translation, err := translator.Translate(valueForTranslation, "fr", file.Locale)
			if err != nil {
				return errors.New("Unable to translate the value of the source segment " + change.Path + ": " + err.Error())
			}
			translation, err = revertTranslationPreparation(translation, ctx)
			if err != nil {
				return errors.New("Unable to revert translation preparation for " + change.Path + ": " + err.Error())
			}

			if err = checkVariableEquity(value, translation); err != nil {
				return errors.New("The translation does not contain the required variables: " + err.Error())
			}

			if err = file.SetSegmentValueAt(change.Path, translation); err != nil {
				return errors.New("Unable to set the value of the source segment " + change.Path + " in locale " + file.Locale + ": " + err.Error())
			}
		} else if change.Kind == translationFile.Removed {
			if err := file.RemoveSegmentAt(change.Path); err != nil {
				return errors.New("Unable to remove segment " + change.Path + " in locale " + file.Locale + ": " + err.Error())
			}
		}
	}
	return nil
}

Fonctionnement

Traduction


func revertTranslationPreparation(text string, ctx map[string]string) (string, error) {
	for value, placeholder := range ctx {
		text = strings.Replace(text, placeholder, "%{"+value+"}", -1)
	}
	return text, nil
}

Fonctionnement

Traduction

func handleStandaloneSegment(group *TranslationGroup, change *translationFile.Change, translator *deepl.Deepl) error {
	for _, file := range group.files {
		if change.Kind == translationFile.Added || change.Kind == translationFile.Updated {
			value, err := group.source.GetSegmentValueAt(change.Path)
			if err != nil {
				return errors.New("Unable to get the value of the source segment " + change.Path + ": " + err.Error())
			}
			valueForTranslation, ctx, err := prepareForTranslation(value)
			if err != nil {
				return errors.New("Unable to prepare for translation " + change.Path + ": " + err.Error())
			}
			translation, err := translator.Translate(valueForTranslation, "fr", file.Locale)
			if err != nil {
				return errors.New("Unable to translate the value of the source segment " + change.Path + ": " + err.Error())
			}
			translation, err = revertTranslationPreparation(translation, ctx)
			if err != nil {
				return errors.New("Unable to revert translation preparation for " + change.Path + ": " + err.Error())
			}

			if err = checkVariableEquity(value, translation); err != nil {
				return errors.New("The translation does not contain the required variables: " + err.Error())
			}

			if err = file.SetSegmentValueAt(change.Path, translation); err != nil {
				return errors.New("Unable to set the value of the source segment " + change.Path + " in locale " + file.Locale + ": " + err.Error())
			}
		} else if change.Kind == translationFile.Removed {
			if err := file.RemoveSegmentAt(change.Path); err != nil {
				return errors.New("Unable to remove segment " + change.Path + " in locale " + file.Locale + ": " + err.Error())
			}
		}
	}
	return nil
}

Fonctionnement

Traduction

func checkVariableEquity(a string, b string) error {
	r, err := regexp.Compile("(?P<variable>%{[a-zA-Z_ -]+})")
	if err != nil {
		return errors.New("Unable to prepare variable check: " + err.Error())
	}
	aResults := r.FindAllStringSubmatch(a, -1)

	var bResults []string
	for _, match := range r.FindAllStringSubmatch(b, -1) {
		bResults = append(bResults, match[1])
	}

	for _, variable := range aResults {
		if !utils.Includes(variable[1], bResults) {
			return errors.New(fmt.Sprintf("'%s' is not present in '%s'", variable[1], b))
		}
	}
	return nil
}

Fonctionnement

Traduction

func handleStandaloneSegment(group *TranslationGroup, change *translationFile.Change, translator *deepl.Deepl) error {
	for _, file := range group.files {
		if change.Kind == translationFile.Added || change.Kind == translationFile.Updated {
			value, err := group.source.GetSegmentValueAt(change.Path)
			if err != nil {
				return errors.New("Unable to get the value of the source segment " + change.Path + ": " + err.Error())
			}
			valueForTranslation, ctx, err := prepareForTranslation(value)
			if err != nil {
				return errors.New("Unable to prepare for translation " + change.Path + ": " + err.Error())
			}
			translation, err := translator.Translate(valueForTranslation, "fr", file.Locale)
			if err != nil {
				return errors.New("Unable to translate the value of the source segment " + change.Path + ": " + err.Error())
			}
			translation, err = revertTranslationPreparation(translation, ctx)
			if err != nil {
				return errors.New("Unable to revert translation preparation for " + change.Path + ": " + err.Error())
			}

			if err = checkVariableEquity(value, translation); err != nil {
				return errors.New("The translation does not contain the required variables: " + err.Error())
			}

			if err = file.SetSegmentValueAt(change.Path, translation); err != nil {
				return errors.New("Unable to set the value of the source segment " + change.Path + " in locale " + file.Locale + ": " + err.Error())
			}
		} else if change.Kind == translationFile.Removed {
			if err := file.RemoveSegmentAt(change.Path); err != nil {
				return errors.New("Unable to remove segment " + change.Path + " in locale " + file.Locale + ": " + err.Error())
			}
		}
	}
	return nil
}

Fonctionnement

Traduction

func handlePluralSegment(part string, group *TranslationGroup, change *translationFile.Change, translator *deepl.Deepl) error {
	for _, file := range group.files {
		affectedKeys := determineAffectedKeysIn(part, file.Locale)
		for _, definition := range affectedKeys {
			pathParts := strings.Split(change.Path, ".")
			localKey := strings.Join(append(pathParts[:len(pathParts)-1], definition.key), ".")

			if change.Kind == translationFile.Added || change.Kind == translationFile.Updated {
				value, err := group.source.GetSegmentValueAt(change.Path)
				if err != nil {
					return errors.New("Unable to get the value of the source segment " + change.Path + ": " + err.Error())
				}

				localValue := strings.ReplaceAll(value, "%{count}", strconv.Itoa(int(definition.tip)))
				valueForTranslation, ctx, err := prepareForTranslation(localValue)
				if err != nil {
					return errors.New("Unable to prepare for translation " + change.Path + ": " + err.Error())
				}
				translation, err := translator.Translate(valueForTranslation, "fr", file.Locale)
				if err != nil {
					return errors.New("Unable to translate the value of the source segment " + change.Path + ": " + err.Error())
				}
				translation, err = revertTranslationPreparation(translation, ctx)
				if err != nil {
					return errors.New("Unable to revert translation preparation for " + change.Path + ": " + err.Error())
				}
				translation = strings.ReplaceAll(translation, strconv.Itoa(int(definition.tip)), "%{count}")

				if err = checkVariableEquity(value, translation); err != nil {
					return errors.New("The translation does not contain the required variables: " + err.Error())
				}

				if err = file.SetSegmentValueAt(localKey, translation); err != nil {
					return errors.New("Unable to set the value of the source segment " + change.Path + " in locale " + file.Locale + ": " + err.Error())
				}
			} else if change.Kind == translationFile.Removed {
				if err := file.RemoveSegmentAt(localKey); err != nil {
					return errors.New("Unable to remove segment " + localKey + " in locale " + file.Locale + ": " + err.Error())
				}
			}
		}
	}
	return nil
}

Fonctionnement

Traduction

func determineAffectedKeysIn(part string, locale string) []pluralDefinition {
	keysDirections := map[string]map[string][]pluralDefinition{
		"zero": {
			"en": {
				{
					key: "zero",
					tip: 0,
				},
			},
        },
        "one": {
            "en": {
                key: "one",
                tip: 1
            }
        }
    }
    return keysDirections[part][locale]
}

Fonctionnement

Traduction

func handlePluralSegment(part string, group *TranslationGroup, change *translationFile.Change, translator *deepl.Deepl) error {
	for _, file := range group.files {
		affectedKeys := determineAffectedKeysIn(part, file.Locale)
		for _, definition := range affectedKeys {
			pathParts := strings.Split(change.Path, ".")
			localKey := strings.Join(append(pathParts[:len(pathParts)-1], definition.key), ".")

			if change.Kind == translationFile.Added || change.Kind == translationFile.Updated {
				value, err := group.source.GetSegmentValueAt(change.Path)
				if err != nil {
					return errors.New("Unable to get the value of the source segment " + change.Path + ": " + err.Error())
				}

				localValue := strings.ReplaceAll(value, "%{count}", strconv.Itoa(int(definition.tip)))
				valueForTranslation, ctx, err := prepareForTranslation(localValue)
				if err != nil {
					return errors.New("Unable to prepare for translation " + change.Path + ": " + err.Error())
				}
				translation, err := translator.Translate(valueForTranslation, "fr", file.Locale)
				if err != nil {
					return errors.New("Unable to translate the value of the source segment " + change.Path + ": " + err.Error())
				}
				translation, err = revertTranslationPreparation(translation, ctx)
				if err != nil {
					return errors.New("Unable to revert translation preparation for " + change.Path + ": " + err.Error())
				}
				translation = strings.ReplaceAll(translation, strconv.Itoa(int(definition.tip)), "%{count}")

				if err = checkVariableEquity(value, translation); err != nil {
					return errors.New("The translation does not contain the required variables: " + err.Error())
				}

				if err = file.SetSegmentValueAt(localKey, translation); err != nil {
					return errors.New("Unable to set the value of the source segment " + change.Path + " in locale " + file.Locale + ": " + err.Error())
				}
			} else if change.Kind == translationFile.Removed {
				if err := file.RemoveSegmentAt(localKey); err != nil {
					return errors.New("Unable to remove segment " + localKey + " in locale " + file.Locale + ": " + err.Error())
				}
			}
		}
	}
	return nil
}

Quoi retenir?

$ paarthurnax -h

A simple and quick translation tool for in place
translation of Rails project

Usage:
  Paarthurnax [flags]
  Paarthurnax [command]

Available Commands:
  completion  Generate the autocompletion script for the specified shell
  help        Help about any command
  init        Initialize the repository
  normalize   Normalize the repository
  translate   Translate the repository

Flags:
  -h, --help   help for Paarthurnax

Use "Paarthurnax [command] --help" for more information about a command.

Code source

https://github.com/eliastre100/Paarthurnax