Fix UTF-8 string corruption across the native/.NET boundary (#1) - #2
Open
TorinKS wants to merge 2 commits into
Open
Fix UTF-8 string corruption across the native/.NET boundary (#1)#2TorinKS wants to merge 2 commits into
TorinKS wants to merge 2 commits into
Conversation
The native C API writes every WD_DEVICE_INFO char[] field as UTF-8, but the managed struct declared those fields as CharSet.Ansi/ByValTStr, so the CLR decoded them using the system ANSI codepage. Every non-ASCII character was corrupted: on a German system "Billboard-Gerät" arrived as "Billboard-Gerät" (U+00E4 is UTF-8 C3 A4, misread as two CP1252 characters). Marshal the fields as raw byte buffers and decode them explicitly as UTF-8 via the new Utf8Buffer.ToStringZ helper. Byte buffers occupy exactly the same space as the ByValTStr fields they replace, so WD_DEVICE_INFO's layout is unchanged and the wrapper stays binary-compatible with an already-deployed WinDevices.dll. A layout test asserts the struct size and every field offset to keep it that way. Also fix SafeStrCopy's truncation path. It clamped the output length and passed that to WideCharToMultiByte, but Win32 does not truncate on a short buffer - it fails with ERROR_INSUFFICIENT_BUFFER and leaves the destination undefined. Now the conversion goes through a temporary buffer and truncation backs off to a UTF-8 character boundary, so a multi-byte sequence is never split. PtrToStringAnsi -> PtrToStringUTF8 for the version build date and error message strings, so the whole boundary decodes consistently. Both are ASCII today, so this is a consistency change rather than a behaviour change. Fixes #1 (encoding portion)
TorinKS
force-pushed
the
fix/utf8-string-marshaling
branch
from
August 24, 2026 15:47
80b4ccf to
aaebaf4
Compare
GitHub moved windows-latest onto the windows-2025-vs2026 image, which ships
Visual Studio 2026. The workflows pin CMake to 3.29, and the
"Visual Studio 18 2026" generator was only added in CMake 4.2, so CMake no
longer recognises any Visual Studio on the runner. It silently falls back to
the NMake Makefiles generator, and because nothing activates an MSVC developer
environment, configuration dies before compiling anything:
-- Building for: NMake Makefiles
CMake Error at CMakeLists.txt:57 (project):
Running 'nmake' '-?' failed with: no such file or directory
CMake Error: CMAKE_CXX_COMPILER not set, after EnableLanguage
Pin the two jobs that actually compile to windows-2022, which is still GA and
carries VS 2022. Note that windows-2025 is not an alternative - it ships VS 2026
as well; windows-2022 is the only label that still provides VS 2022.
The summary and increment-version jobs run PowerShell only and have no toolchain
dependency, so they stay on windows-latest.
This pins the runner image alongside the already-pinned CMake version. Pinning
only the tool while the image floats is what let the two drift apart. Bumping
CMake to >= 4.2 and staying on windows-latest is the alternative when VS 2022
images are eventually retired.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the encoding half of #1.
The bug
The native library is UTF-16 (
std::wstring) end to end and correct. The corruption happens exactly at the interop boundary:SafeStrCopyconverts withWideCharToMultiByte(CP_UTF8, ...)into thechar[]fields ofWD_DEVICE_INFO.WdDeviceInfodeclaredCharSet = CharSet.AnsiwithByValTStr, so the CLR decoded those bytes with the system ANSI codepage.So
ä(U+00E4) becomes UTF-8C3 A4, and decoded as CP1252 that givesä. That is precisely theBillboard-Gerätin the reporter's debugger screenshot. It affects every string field and reproduces on any non-ASCII device string, independent of hardware.Because the screenshot is a debugger watch window, the corruption is baked into the marshalled
stringitself. This is not a console rendering problem, andConsole.OutputEncodingis not involved.The fix
Keep UTF-8 on the wire (the right choice, since the C API is consumable from plain C) and decode it explicitly on the managed side:
WdDeviceInfostring fields becomeUnmanagedType.ByValArrayofbyte, decoded through a newUtf8Buffer.ToStringZhelper.ByValTStrfields they replace, so this stays binary-compatible with an already-deployedWinDevices.dll. A layout test asserts the total size (2672) and all 12 field offsets so a future edit cannot silently break that.WD_DEVICE_INFOinWinDevicesAPI.hfor C consumers.Second bug fixed along the way
SafeStrCopy's truncation path was broken independently of the encoding issue. It clampedcopyLenand passed that as the output size toWideCharToMultiByte, but Win32 does not truncate on a short buffer. It fails withERROR_INSUFFICIENT_BUFFERand leaves the destination undefined. Even had it written, clamping on a byte boundary can split a multi-byte sequence. Non-ASCII strings are exactly the ones that approach the buffer limit, so the two bugs compounded.Now the conversion goes through a temporary buffer, and truncation backs off to a UTF-8 character boundary. The destination always holds valid, NUL-terminated UTF-8.
Consistency
PtrToStringAnsibecomesPtrToStringUTF8for the version build date and error-message strings. Both are ASCII today, so this is a consistency change, not a behaviour change.Verification
Built the native library with MSVC (
WinDevices.dll) and ran everything against it:WinDevicesE2ETests)The 3 native E2E failures are hardware-gated and pre-existing. They assert "Please connect a USB flash drive" and require a specific JetFlash serial number, and no USB mass storage is attached to this machine. They do not touch the changed code path.
Baseline check:
mainfails 40 .NET tests withDllNotFoundExceptionwhen the native DLL is not built. With the DLL present, bothmainand this branch are green, and this branch adds 12 passing tests.New tests cover the exact reported string (
Billboard-Gerätround-trips, and explicitly asserts it does not equal theBillboard-Gerätmojibake), NUL handling, missing terminator, empty and null buffers, malformed UTF-8 degrading rather than throwing, and the struct layout guard.Known gap
The native truncation fix is verified by compilation and review but is not directly unit-tested.
SafeStrCopyhas internal linkage, and its truncation path cannot be driven through the public C API without a device whose name exceeds the buffer. Worth revisiting if the helper is ever made testable.CI
windows-latestmoved to thewindows-2025-vs2026image, which ships Visual Studio 2026. The workflows pin CMake to 3.29, and the "Visual Studio 18 2026" generator was only added in CMake 4.2, so CMake stopped recognising any Visual Studio on the runner and fell back to the NMake Makefiles generator. Configuration then failed before compiling anything. The two jobs that compile are now pinned towindows-2022. Note thatwindows-2025is not an alternative, since it ships VS 2026 as well.Not included
Two follow-ups from the same issue, deliberately left out to keep this reviewable and ABI-safe:
bcdDeviceis already retrieved intoHubConnectionInfo::_deviceDescriptorand simply never propagated. The reporter's second screenshot is USBDeview showing30.98, which confirms it isbcdDevice(0x3098, BCD-formatted) rather than a SCSI INQUIRY revision.Description,DeviceId,DevicePath,FriendlyName,ManufacturerandProductNameall empty.SetFriendlyNameandSetDescriptionare only called fromEnumerateByDeviceClass, never fromEnumerateUsbDevices. This is a wider gap than the issue's "DeviceName would also be good" suggests.Note that "Device Name: 40AY" in the USBDeview screenshot is not the friendly name. USBDeview's own Friendly Name field is empty there, and
40AYlooks derived from the instance ID (USB\VID_17EF&PID_30A9\1S40AY0090E). Exposing the Instance ID as a proper field is probably the better answer than trying to replicate it.Item 1 breaks the
WD_DEVICE_INFOABI and needs a version bump, so it belongs in its own PR.