Detect encoding changes while parsing diff

[jpraet]

Fix #16329

Independently detect encoding of added, deleted and unchanged sections of each file diff.

Before:

After:

[codecov-commenter]

Codecov Report

Merging #16330 (12f34ec) into main (0393a57) will increase coverage by 1.21%.
The diff coverage is 50.22%.

❗ Current head 12f34ec differs from pull request most recent head f55c6ff. Consider uploading reports for the commit f55c6ff to get more accurate results

@@            Coverage Diff             @@
##             main   #16330      +/-   ##
==========================================
+ Coverage   44.24%   45.45%   +1.21%     
==========================================
  Files         695      710      +15     
  Lines       82341    83952    +1611     
==========================================
+ Hits        36429    38161    +1732     
+ Misses      40014    39654     -360     
- Partials     5898     6137     +239     

Impacted Files	Coverage Δ
cmd/convert.go	0.00% <0.00%> (ø)
cmd/dump.go	0.91% <0.00%> (ø)
cmd/dump_repo.go	0.00% <0.00%> (ø)
cmd/generate.go	0.00% <0.00%> (ø)
cmd/hook.go	0.00% <0.00%> (ø)
cmd/migrate.go	0.00% <0.00%> (ø)
cmd/migrate_storage.go	0.00% <0.00%> (ø)
cmd/serv.go	2.36% <0.00%> (-0.22%)	⬇️
cmd/web.go	0.00% <0.00%> (ø)
cmd/web_letsencrypt.go	0.00% <ø> (ø)
... and 285 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 2614309...f55c6ff. Read the comment docs.

[zeripath]

You're going to need to add a marker to the file header to say the charsets for the files - otherwise you're likely to get an apparently empty diff on re-encoding.

[jpraet]

You're going to need to add a marker to the file header to say the charsets for the files - otherwise you're likely to get an apparently empty diff on re-encoding.

Indeed you can now have something like this:

I agree that a marker in the file header indicating an encoding change would be useful. But I fear that the information will not be reliable, because we only detect the encoding based on the added/deleted sections of the file. To detect the encoding reliably I think the complete old and new version of the file would need to be processed. The diff encoding detection is a best effort in order to attempt to render the diff correctly.

I would expect the commit message to mention the encoding change, providing the necessary context for the user to be able to interpret an apparently empty diff.

[6543]

Didnt look to close but seeing 3 created buffers sound bit memory consuming in the first place

[jpraet]

Didnt look to close but seeing 3 created buffers sound bit memory consuming in the first place

The 3 buffers are being reused for each file, and their combined content should be the same as the previous single buffer (just split out across 3 separate buffers by line type). So, at first sight, memory consumption looks similar to the previous version to me. But I'm new to golang, so I don't know for sure.