Add option to strip metadata when parsing PO files

[maennchen]

Intended to be used in Gettext.Backend to speed up compiler and minimize memory usage when meta information is not needed.

Uses roughly a third of memory and is two times faster than including all meta information.

Performance Test

#!/usr/bin/env elixir

Mix.install([
  {:expo, path: "."},
  {:large_translations_catalogue,
   github: "jshmrtn/hygeia",
   branch: "main",
   app: false,
   compile: false,
   depth: 1,
   sparse: "priv/gettext"},
  {:benchee, "~> 1.3"}
])

require Logger

catalogue_base_path =
  Path.join([Mix.install_project_dir(), "deps", "large_translations_catalogue", "priv", "gettext"])

Logger.info("Preparing .mo files")

po_files = Path.wildcard(Path.join([catalogue_base_path, "*", "LC_MESSAGES", "*.po"]))

for path <- po_files do
  mo_path = Path.rootname(path, ".po") <> ".mo"
  Mix.Task.rerun("expo.msgfmt", [path, "--output-file", mo_path])
end

mo_files = Path.wildcard(Path.join([catalogue_base_path, "*", "LC_MESSAGES", "*.mo"]))

Logger.info("Performance Test Expo Parse")

Benchee.run(
  [
    expo_po_parse: fn ->
      for file <- po_files, do: Expo.PO.parse_file!(file)
    end,
    expo_po_strip_meta_parse: fn ->
      for file <- po_files, do: Expo.PO.parse_file!(file, strip_meta: true)
    end,
    expo_mo_parse: fn ->
      for file <- mo_files, do: Expo.MO.parse_file!(file)
    end
  ],
  warmup: 5,
  time: 10,
  memory_time: 2,
  parallel: System.schedulers_online()
)

Results:

13:48:54.184 [info] Preparing .mo files

13:48:54.236 [info] Performance Test Expo Parse
Operating System: Linux
CPU Information: 11th Gen Intel(R) Core(TM) i7-1185G7 @ 3.00GHz
Number of Available Cores: 8
Available memory: 46.76 GB
Elixir 1.17.2
Erlang 27.0.1
JIT enabled: true

Benchmark suite executing with the following configuration:
warmup: 5 s
time: 10 s
memory time: 2 s
reduction time: 0 ns
parallel: 8
inputs: none specified
Estimated total run time: 51 s

Benchmarking expo_po_parse ...
Benchmarking expo_po_strip_meta_parse ...
Benchmarking expo_mo_parse ...
Calculating statistics...
Formatting results...

Name                               ips        average  deviation         median         99th %
expo_mo_parse                    56.77       17.62 ms    ±31.16%       16.07 ms       35.65 ms
expo_po_strip_meta_parse         14.12       70.81 ms    ±14.25%       69.67 ms      102.19 ms
expo_po_parse                     6.54      152.92 ms    ±11.27%      151.50 ms      199.97 ms

Comparison: 
expo_mo_parse                    56.77
expo_po_strip_meta_parse         14.12 - 4.02x slower +53.19 ms
expo_po_parse                     6.54 - 8.68x slower +135.31 ms

Memory usage statistics:

Name                             average  deviation         median         99th %
expo_mo_parse                    4.05 MB     ±0.00%        4.05 MB        4.05 MB
expo_po_strip_meta_parse        11.20 MB     ±0.00%       11.20 MB       11.20 MB
expo_po_parse                   33.67 MB     ±0.00%       33.67 MB       33.67 MB

Comparison: 
expo_mo_parse                    4.05 MB
expo_po_strip_meta_parse        11.20 MB - 2.77x memory usage +7.15 MB
expo_po_parse                   33.67 MB - 8.32x memory usage +29.62 MB

[coveralls]

Pull Request Test Coverage Report for Build ff379f76a243351fcb49ffa3076f519e522cc105-PR-141

Details

• 17 of 18 (94.44%) changed or added relevant lines in 2 files are covered.
• No unchanged relevant lines lost coverage.
• Overall coverage decreased (-0.2%) to 97.936%

---

Changes Missing Coverage	Covered Lines	Changed/Added Lines	%
lib/expo/po/tokenizer.ex	15	16	93.75%

Totals
Change from base Build deb65ba9cbda69768ded2dcfac751dd6f8046828:	-0.2%
Covered Lines:	427
Relevant Lines:	436

---

💛 - Coveralls

[whatyouhide]

@maennchen so what would be a use case where meta information is not needed? Runtime translations? I thought we were still discussing if that's something we want.

[maennchen]

@whatyouhide We need all meta information when we're extracting translations to merge correctly. However when we compile the backend, we don't need any comments etc.

I should choose a better name than runtime, especially in the current context it is misleading. Maybe messages_only or something. Ideas?

[whatyouhide]

@maennchen something like "strip meta" maybe? IIRC that's what we use in the various Elixir compiler stuff.

[maennchen]

@whatyouhide Changed to strip_meta 👍

[whatyouhide]

Left a couple of comments but this looks great.

[maennchen]

@whatyouhide Are we good to merge?

[whatyouhide]

Fantastic @maennchen 💟