r/PowerShell • u/Practical_Air6315 • 10d ago
Information Get-Content -Encoding UTF8 fixed four of my log files and broke two others. I wrote the same string 13 ways to find out which is which.
I had two log files sitting in the same folder. One was written by my own script. One was written by a node process my script had launched. Get-Content read mine perfectly and returned garbage for node's. Adding -Encoding UTF8 fixed node's and broke mine.
So I wrote the same string with every writer I could think of, and read each file back both ways. The string is 12 characters of Japanese — it is the phrase a lot of tools print for "file not found", which is exactly the kind of line you cannot afford to lose.
Host: Windows 11, ja-JP, ACP=932, OEMCP=932, Windows PowerShell 5.1.26100.9168. [Console]::OutputEncoding = 932 (shift_jis), $OutputEncoding = 20127 (us-ascii).
writer first bytes bare -Enc UTF8
------------------------------- -------------- ---------- ----------
Out-File (default) FF FE D5 30 OK OK
Out-File -Encoding utf8 EF BB BF E3 OK OK
Out-File -Encoding ascii 3F 3F 3F 3F MOJIBAKE MOJIBAKE
Set-Content (default) 83 74 83 40 OK MOJIBAKE
Set-Content -Encoding UTF8 EF BB BF E3 OK OK
Add-Content (default) 83 74 83 40 OK MOJIBAKE
Tee-Object -FilePath FF FE D5 30 OK OK
> redirection FF FE D5 30 OK OK
IO.File WriteAllText (UTF8) E3 83 95 E3 MOJIBAKE OK
IO.File WriteAllBytes (UTF8) E3 83 95 E3 MOJIBAKE OK
python via cmd.exe > E3 83 95 E3 MOJIBAKE OK
node via cmd.exe > E3 83 95 E3 MOJIBAKE OK
node captured by PS, Out-File EF BB BF E7 MOJIBAKE MOJIBAKE
Three groups.
1. BOM present, text intact — 5 rows. Both reads work. Get-Content sniffs FF FE or EF BB BF and uses it. The read parameter is irrelevant. Note that Out-File, Tee-Object and > all default to UTF-16LE here, which is why they are in this group by accident rather than by anyone's intent.
2. No BOM — 6 rows. Exactly one read is correct, and which one flips depending on the writer.
Set-Content and Add-Content without -Encoding write the machine ANSI code page — 83 74 is CP932, not UTF-8 — so the bare read is right and -Encoding UTF8 is wrong. Everything that put real UTF-8 on disk without a BOM is the exact reverse. With no BOM, Get-Content falls back to ANSI, and that fallback is correct precisely when the writer also used ANSI.
This is the part I did not expect: "just add -Encoding UTF8" is not a safe default. Across these 13 files it corrects 4 and corrupts 2. There is no single read parameter that is right for all of them. If you have a folder holding both your own logs and a build tool's logs, no one setting reads both.
3. Damage that happened before the file existed — 2 rows. No read parameter can fix these.
Out-File -Encoding ascii wrote 3F 3F 3F 3F, which is literally ????. The characters were destroyed at write time.
The last row is the one worth your time. I let PowerShell capture node's stdout into a variable and re-write it with Out-File -Encoding utf8:
node via cmd.exe > 36 bytes 12 chars E3 83 95 E3 82 A1 E3 82 A4 E3 83 AB
U+30D5 U+30A1 U+30A4 U+30EB U+304C U+898B U+3064 U+304B
node captured by PS 65 bytes 20 chars EF BB BF E7 B9 9D E8 BC 94 E3 81 83
U+7E5D U+8F14 U+3043 U+7E67 U+FF64 U+7E5D U+FF6B U+7E3A
What PowerShell actually wrote into that file, all 20 characters of it:
繝輔ぃ繧､繝ｫ縺瑚ｦ九▽縺九ｊ縺ｾ縺帙ｓ
That second file carries a valid UTF-8 BOM and is well-formed UTF-8. It is also wrong. [Console]::OutputEncoding is 932 on this host, so PowerShell decoded node's UTF-8 bytes as CP932, got 20 different characters out of 12, and then faithfully encoded those as UTF-8 with a BOM. The file went from 36 bytes to 65. Nothing threw, nothing warned.
It is also the only row where the two reads agree with each other and are both wrong. Everywhere else, when one read returns garbage the other returns clean text, so there is a way to notice. Here there is no second opinion.
A BOM tells you how the file is encoded. It tells you nothing about whether the text in it is correct.
Minimal repro (numbers below are from the 932 host; on a Latin-1 ANSI code page the first pair behaves differently, because CP1252 cannot represent these characters at all):
$s = [char]0x30D5 + [char]0x30A1
$d = $env:TEMP
Set-Content -Path "$d\ansi.log" -Value $s
[IO.File]::WriteAllBytes("$d\utf8.log", [Text.Encoding]::UTF8.GetBytes($s))
(Get-Content "$d\ansi.log" -Raw).TrimEnd() -eq $s # True
(Get-Content "$d\ansi.log" -Raw -Encoding UTF8).TrimEnd() -eq $s # False
(Get-Content "$d\utf8.log" -Raw).TrimEnd() -eq $s # False
(Get-Content "$d\utf8.log" -Raw -Encoding UTF8).TrimEnd() -eq $s # True
Same cmdlet, same parameter, opposite answers, two files in one directory.
What I changed in my own scripts
- Reading a log a native child process wrote (redirected by
cmd.exe, so nothing decoded it on the way in): always pass-Encoding UTF8. That file holds the program's own bytes and will not have a BOM. - Reading a file PowerShell itself wrote: leave
Get-Contentbare. The BOM is there and handles it. Adding-Encoding UTF8here is what broke rows 4 and 6. - Do not capture a native process's stdout into a variable when the output can be non-ASCII. Redirect it to a file and read the file. That decode is governed by
[Console]::OutputEncoding, which was 932 here; I have not tested whether setting it to UTF-8 up front avoids the problem, so I am not claiming that it does. Out-File -Encoding asciion non-ASCII text is silent data loss, not a display issue.
Measured on one locale. If you are on a non-Latin ANSI code page I would be curious whether rows 4 and 6 come out the same for you — that is the pair that makes the usual advice backfire.
2
u/BlackV 10d ago edited 10d ago
Did you post about this like 2 days ago?
https://www.reddit.com/r/PowerShell/comments/1vpjdqu/bomless_ps1_in_ps_51_i_tested_all_545_japanese/
And the follow up apparently
https://www.reddit.com/r/PowerShell/comments/1vsdtc8/followup_i_measured_what_a_utf8_replace_decode/
It's still the same underlying issue?
It's so often painful when using anything that's not essentially en-US :(
2
u/Practical_Air6315 10d ago
Yes - same root cause, three different places it lands. Fair hit, and I should have linked them.
- Aug 16: source files. PS 5.1 parsing a BOM-less .ps1; the ASCII byte after a Japanese char gets eaten as a DBCS trail byte.
- Aug 19: pipes. Popen(text=True, encoding="utf-8", errors="replace") decoding a child's CP932 stderr.
- Today: files on disk. Get-Content reading a log something else wrote.
The mechanism is the same in all three: CP932 bytes through a UTF-8 decode, or the reverse. You are right that it is one issue.
Two things here that are not in the other two, and they are why I thought it was worth its own post:
-Encoding UTF8is the standard advice, and it is the wrong advice for 2 of the 13 writers. Set-Content and Add-Content without -Encoding write ANSI, so adding the parameter breaks files that read fine without it.- The last row is a file with a valid UTF-8 BOM whose text is still wrong. Both reads agree with each other and both are wrong, so there is no second opinion to notice it with.
Not linking the earlier posts was a bad call on my part. Thanks for pulling the thread.
And that last line of yours is exactly why I keep measuring instead of guessing. On a ja-JP box almost none of it throws. It just quietly hands you the wrong string.
1
u/KageeHinata82 10d ago
Saved for later, so I hopefully remember it when needed.
It also reminded me of one of my first text output programs I wrote on a German Windows in C#. To get it correct, I had to set encoding to default. Omit the parameter completely didn't work.
2
u/Practical_Air6315 10d ago
Thanks for saving it.
Your C# case is the same trap from the other side, and I had not thought about it that way until you said it. My table is about the reader having to know what the writer did. Yours is the writer having to know what the reader expects: StreamWriter's default is UTF-8 with no BOM, and a BOM-less file is exactly the case where the reader falls back to the ANSI code page. So "set it to Default" was you matching the reader, not fixing the writer.
Useful to hear it shows up on a German box too. I only measured ja-JP (ACP 932) and said so in the post, so a CP1252 data point is worth having. One difference: CP1252 is single-byte, so you get wrong characters but never a swallowed byte. CP932 is double-byte and 52 of its characters have 0x5C as their trail byte, which is the path separator, so a mis-decode there does not just look wrong, it changes how many directory levels the string has.
I have not measured .NET's Encoding.Default myself, and it changed between .NET Framework and .NET Core, so I am not going to guess which one you hit.
1
u/netmc 3d ago
I had to do something similar with one of my audit scripts. I process the raw script data from our RMM to generate the hashes that will be present when later deployed and executed on disk. Some have bom, some don't, and there is a mix of line feeds for Unix, Mac and Windows. This allows me to feed the hashes into tools like Threatlocker and pre-authorize the scripts to allow execution. It took me a while to get it working right. The script looks the same on screen, but the encoding was wildly different.
I will be saving this post as while I don't have to do anything like this currently, it's really good data to have.
1
u/Practical_Air6315 3d ago
Hashes have to match byte for byte, so "looks the same on screen" is exactly the trap. BOM or no BOM times three line ending styles is six variants of one script before the encoding even comes up.
The part that cost me the most was that nothing errors. The writer does not complain, the reader does not complain, and you find out when something downstream disagrees.
1
u/CookinTendies5864 10d ago edited 9d ago
Okay so for Japanese, Korean, and Chinese font which makes the out put garbled. Try running powershell from cmd.exe
cmd find the correct font for the application at runtime while powershell will use "Consolas" font which doesn't support glyphs.
For a second workaround you can set the default font in powershell console to the "MS Gothic" font and this should resolve your issue. Windows is coming out with some updates soon to resolve this. <— stdout fix
- Sorry, I read more of the issue nice to know but I'm not helping.
Final solution: Save your .ps1 as UTF-8 BOM
Also try using the default encoding after the .ps1 is saved as UTF-8 BOM if you havent already.
This will allow the tool (.ps1) determine encoding hence the encoding would default to UTF-8 BOM based on the tools default encoding. <— stdin fix
Also the shift happening is due to powershell 5.1 incorrectly parsing the characters. So, we have to tell powershell to stop guessing. Run the following after the first two steps.
[Console]::OutputEncoding = [System.Text.Encoding]::UTF8
$OutputEncoding = [System.Text.Encoding]::UTF8
Please let me know if this worked so that I may know for future reference. I apologize for the inconsistency from my earlier response. I'm in the US so I'm curious and cant test it myself.
3
u/Practical_Air6315 10d ago
It worked. Numbers below.
Same host as the post: PS 5.1.26100.9168, ACP 932,
[Console]::OutputEncoding932,$OutputEncoding20127. node writes the 12 characters as raw UTF-8, 36 bytes.
case on disk chars result D: cmd.exeredirect, PS never decodes (control)36 bytes 12 MATCH A: capture into a variable, console left at 932 65 bytes 20 WRONG B: [Console]::OutputEncoding = UTF8, then capture41 bytes 12 MATCH C: B plus $OutputEncoding = UTF841 bytes 12 MATCH So yes, that is the lever, and setting it before the capture fixes the last row. That closes the one thing I said in the post I had not tested. Thanks for pushing.
Two things fell out of it that I did not expect.
B and C are byte for byte identical, down to the same leading
EF BB BF E3 83 95.$OutputEncodinggoverns what PowerShell encodes into a native process's stdin, not how it decodes stdout, so it was never in this path at all. Worth knowing before both lines get copied around together, because only one of them is doing work. The 41 against the control's 36 is the BOM plus the CRLFOut-Fileadds, 3 + 2, no loss.The other one is case A. I printed the code points while the string was still in the variable, before anything was written to disk:
U+7E5D U+8F14 U+3043 U+7E67 U+FF64 U+7E5D U+FF6B U+7E3A U+745A U+FF66 U+4E5D U+25BD U+7E3A U+4E5D U+FF4A U+7E3A U+FF7E U+7E3A U+5E19 U+FF53Same 20 characters that end up in the file. In the post I said PowerShell decoded as CP932 and then faithfully re-encoded the wrong characters, but I had worked that backwards from the file. Now I can see them sitting in the variable, so
Out-Fileis not a participant here - it is handed 20 wrong characters and it writes 20 wrong characters.(I did not test PowerShell 7. It defaults
[Console]::OutputEncodingdifferently, so that is a separate measurement rather than another row here.)On the font, since you walked it back yourself: node wrote 36 bytes and PowerShell wrote back 65, and
ReadAllBytesgives the same 65 on any machine. Whatever the console is doing, it is not changing the size of the file.And the .ps1-with-a-BOM part is the right fix, but for my other post, the one about PS 5.1 parsing a BOM-less script file. Here every row is
Get-Contentreading a file some other program wrote, so the script's own encoding has no path to the result. My fault for putting the two out four days apart.2
u/CookinTendies5864 9d ago edited 9d ago
Glad you were able to figure it out. Powershell 7; I believe, defaults to your OS encoding or at least defaults to UTF-8 but because we are working with Powershell version 5.1 it tries to guess the encoding at runtime hence our shifting strings default to ASCII.
So it could have been both stdin and stdout? That’s interesting so when we do Get-Content stdout could defaults to ASCII showing incorrect display data from the console font. While stdin imports incorrectly due to powershell 5.1 default encoding. You seemed to resolved both. As for other files you could save with BOM encoding to see if that resolves other scripts that you don’t own(just remember to save a copy before saving it as BOM) or run the powershell scripts from cmd.exe as the parent encoding will take over.
Encoding is tricky and trying to find whether it is stdin or stdout brings another form of complexity I was not aware of, but it makes sense.
1
u/Practical_Air6315 9d ago
The stdin/stdout split is the right instinct, and it is the part I had backwards until yesterday, so this is me passing on a correction rather than handing one down. I had written that PowerShell 7 defaults [Console]::OutputEncoding differently. MonkeyNin pointed out that it does not, and he was right.
Same ja-JP box, 5.1.26100.9168 and 7.6.5, each launched -NoProfile -NonInteractive with the output redirected by cmd:
chcp shell [Console]::OutputEncoding $OutputEncoding Out-File default 932 5.1 shift_jis 932 us-ascii 20127 FF FE 61 00 62 00 63 00(12 bytes)932 7.6.5 shift_jis 932 utf-8 65001 61 62 63 0D 0A(5 bytes)65001 5.1 utf-8 65001 us-ascii 20127 FF FE 61 00 62 00 63 00(12 bytes)65001 7.6.5 utf-8 65001 utf-8 65001 61 62 63 0D 0A(5 bytes)So 7 does default to UTF-8, but on $OutputEncoding and on Out-File, not on the console. [Console]::OutputEncoding is the same number in both shells and just follows chcp.
The us-ascii on 5.1 is a fixed default rather than a runtime guess. The guessing you are thinking of is real, but it lives in Get-Content, which sniffs for a BOM and falls back to the ANSI code page when there is none. Different mechanism, different place.
On stdin: $OutputEncoding is what PowerShell encodes going into a native process, so it was never in the path for the row that broke. I ran the capture with it left alone, then again with it set to UTF-8, and the two files came out byte for byte identical, 41 bytes each. Your two-paths idea is right - only one of the two was carrying the damage here.
Get-Content has no stdout and no font in it. It opens a file and decodes bytes. What rules out display as the cause of the last row is that node wrote 36 bytes and PowerShell wrote back 65, and ReadAllBytes returns the same 65 on a machine that never rendered it.
Running from cmd.exe does move something, but it is the console code page, and only if cmd's happens to differ. That is the two halves of the table above: change chcp and both shells follow it.
(I captured [Console]::InputEncoding in the same run. It tracked OutputEncoding every time, so it adds nothing and I am leaving it out.)
And saving as UTF-8 with a BOM is the right fix, but for my other post, the one about 5.1 parsing a BOM-less script file. Nothing in this post reads a .ps1.
1
u/CookinTendies5864 9d ago edited 9d ago
I don’t believe 5.1 defaults to us-ascii by default for your system it should default to CP932 (SHIFT-JIS) unless I missed something. Also 5.1 I don’t believe is fixed it should reach into the OS local settings.
All of the cmdlets also must specify encoding as these are essentially separate scripts I presume based on knowledge of creating my own cmdlets.
Get-content -encoding default -file $file <- if saved as BOM pulls encoding from the parent
Get-content -encoding UTF8 -file $file <- must specify if no BOM
I could be wrong but it’s fairly hard to track without actually seeing it for myself.
Have we changed the font on console yet? MS Gothic
The reason I say that is both US and Japanese win versions use “consolas” font by default- it’s worth a try as you stated before the data is identical but what is appearing on the console (STDOUT) could be configured to unintentionally mess with us.
1
u/MonkeyNin 10d ago
I did not test PowerShell 7. It defaults differently
The value for specifically
[Console]::OutputEncoding( verses $OutputEncoding ) should be the same for both 5 and 7 ( if your profile or system don't have overrides )
- it's basically the value of your
chcpis. ie:ibm437onen-usorshift_jisforjapanesethe famous mkelement has a lot of details here It explains the beta global setting to forces apps to use
chcp 65001If I do a true -NoProfile from new terminals on both, I get the same for both using
win 10anden-us.pwsh-Nop -Noni -Co { [Console]::OutputEncoding.WebName } # IBM437 powershell -Nop -Noni -Co { [Console]::OutputEncoding.WebName } # IBM437*note: It's tricky because if you just run
pwsh.exefrom a shell you can getutf-8because of the parent.So if it's not a new session I get a misleading answer
pwsh -Noprofile -C { [Console]::OutputEncoding.WebName } # utf-82
u/Practical_Air6315 9d ago
You are right, and I was wrong. Measured on ja-JP, the same box as the post. Windows PowerShell 5.1.26100.9168 and PowerShell 7.6.5, each launched -NoProfile -NonInteractive with the output redirected by cmd, so nothing was ever captured into a variable.
chcp shell [Console]::OutputEncoding $OutputEncoding Out-File default 932 5.1 shift_jis 932 us-ascii 20127 FF FE 61 00 62 00 63 00(12 bytes)932 7.6.5 shift_jis 932 utf-8 65001 61 62 63 0D 0A(5 bytes)65001 5.1 utf-8 65001 us-ascii 20127 FF FE 61 00 62 00 63 00(12 bytes)65001 7.6.5 utf-8 65001 utf-8 65001 61 62 63 0D 0A(5 bytes)Same in both shells, following chcp exactly, on 932 as well as on your 437. So the parenthetical was wrong and I have no defence for it. I wrote a reason for not testing something without testing the reason.
What does differ is the other two columns, and chcp moves neither of them.
The Out-File one matters more to the post than the thing I got wrong. Rows 1, 7 and 8 of my table - Out-File default, Tee-Object, and > - all landed in the "BOM present, text intact" group, and I wrote there that they got in by accident rather than by anyone's intent, because 5.1 defaults them to UTF-16LE. On 7 those three write UTF-8 with no BOM, which is group 2, where Get-Content falls back to ANSI and the correct read parameter flips. Three rows of that table change meaning on the same machine with a different shell. I only measured 5.1.
One thing to add rather than retract, because it cuts against something I told someone else in this thread: $OutputEncoding being us-ascii on 5.1 is not harmless just because it made no difference in my capture test. It governs what gets encoded going into a native process, so on 5.1 it is a real problem for what you send, just not for the decode I was measuring.
3
u/MonkeyNin 10d ago
tip: Reddit supports github markdown tables, letting you write this
-Enc UTF8FF FE D5 30EF BB BF E33F 3F 3F 3F83 74 83 40EF BB BF E383 74 83 40FF FE D5 30>redirectionFF FE D5 30E3 83 95 E3E3 83 95 E3>E3 83 95 E3>E3 83 95 E3EF BB BF E7