unexpand: -a uses Unicode display width instead of byte count for multibyte characters

[ChrisDryden]

Example Input:

"1ΔΔΔ5 99999" (3 spaces after the 5)
Δ (U+0394) is 2 bytes in UTF-8

printf '1ΔΔΔ5   99999\n' | unexpand -a | xxd

GNU unexpand output: keeps 3 spaces (no conversion)

  00000000: 31ce 94ce 94ce 9435 2020 2039 3939 3939  1......5   99999

uutils unexpand output: converts to tab

00000000: 31ce 94ce 94ce 9435 0939 3939 3939 0a    1......5.99999.

This doesn't fully fix the Busybox non utf-8 tests because they have different rules about trailing spaces, but this example came from there.

[ChrisDryden]

Solution should only require changing the use byte count instead of UnicodeWidthChar::width() and adding a test for regression, ideally this should be left for someone that hasn't contributed before since this fix is only a few lines

[egmontkob]

Interestingly, for the input string 䀀 a (that's U+4000, 5 spaces, and an ASCII letter) both GNU an UU emit 䀀<tab>a.

U+4000 is 1 character, consists of 3 bytes in UTF-8, and has a display width of 2.

That is, while for the opening post's example it's clear that GNU uses byte count and UU uses either display width or character count; for this new example GNU still uses byte count (3) but so does UU. It clearly does not use character count (1), nor display width (2), because then it would have to keep the spaces.

I haven't checked the PR, but the code it's removing (or some code it's keeping) probably deserves some further investigation, because prior to that PR the code behaves inconsistently, not fully according to the opening post's observations.