Skip to content

Fix UTF-8 string corruption across the native/.NET boundary (#1) - #2

Open
TorinKS wants to merge 2 commits into
mainfrom
fix/utf8-string-marshaling
Open

Fix UTF-8 string corruption across the native/.NET boundary (#1)#2
TorinKS wants to merge 2 commits into
mainfrom
fix/utf8-string-marshaling

Conversation

@TorinKS

@TorinKS TorinKS commented Aug 24, 2026

Copy link
Copy Markdown
Owner

Fixes the encoding half of #1.

The bug

The native library is UTF-16 (std::wstring) end to end and correct. The corruption happens exactly at the interop boundary:

  • Native writes UTF-8: SafeStrCopy converts with WideCharToMultiByte(CP_UTF8, ...) into the char[] fields of WD_DEVICE_INFO.
  • Managed reads ANSI: WdDeviceInfo declared CharSet = CharSet.Ansi with ByValTStr, so the CLR decoded those bytes with the system ANSI codepage.

So ä (U+00E4) becomes UTF-8 C3 A4, and decoded as CP1252 that gives ä. That is precisely the Billboard-Gerät in the reporter's debugger screenshot. It affects every string field and reproduces on any non-ASCII device string, independent of hardware.

Because the screenshot is a debugger watch window, the corruption is baked into the marshalled string itself. This is not a console rendering problem, and Console.OutputEncoding is not involved.

The fix

Keep UTF-8 on the wire (the right choice, since the C API is consumable from plain C) and decode it explicitly on the managed side:

  • WdDeviceInfo string fields become UnmanagedType.ByValArray of byte, decoded through a new Utf8Buffer.ToStringZ helper.
  • The struct layout is unchanged. Byte buffers occupy exactly the same space as the ByValTStr fields they replace, so this stays binary-compatible with an already-deployed WinDevices.dll. A layout test asserts the total size (2672) and all 12 field offsets so a future edit cannot silently break that.
  • The UTF-8 contract is now documented on WD_DEVICE_INFO in WinDevicesAPI.h for C consumers.

Second bug fixed along the way

SafeStrCopy's truncation path was broken independently of the encoding issue. It clamped copyLen and passed that as the output size to WideCharToMultiByte, but Win32 does not truncate on a short buffer. It fails with ERROR_INSUFFICIENT_BUFFER and leaves the destination undefined. Even had it written, clamping on a byte boundary can split a multi-byte sequence. Non-ASCII strings are exactly the ones that approach the buffer limit, so the two bugs compounded.

Now the conversion goes through a temporary buffer, and truncation backs off to a UTF-8 character boundary. The destination always holds valid, NUL-terminated UTF-8.

Consistency

PtrToStringAnsi becomes PtrToStringUTF8 for the version build date and error-message strings. Both are ASCII today, so this is a consistency change, not a behaviour change.

Verification

Built the native library with MSVC (WinDevices.dll) and ran everything against it:

Suite Result
.NET tests (incl. 12 new) 57/57 pass
Native E2E (WinDevicesE2ETests) 23/26 pass

The 3 native E2E failures are hardware-gated and pre-existing. They assert "Please connect a USB flash drive" and require a specific JetFlash serial number, and no USB mass storage is attached to this machine. They do not touch the changed code path.

Baseline check: main fails 40 .NET tests with DllNotFoundException when the native DLL is not built. With the DLL present, both main and this branch are green, and this branch adds 12 passing tests.

New tests cover the exact reported string (Billboard-Gerät round-trips, and explicitly asserts it does not equal the Billboard-Gerät mojibake), NUL handling, missing terminator, empty and null buffers, malformed UTF-8 degrading rather than throwing, and the struct layout guard.

Known gap

The native truncation fix is verified by compilation and review but is not directly unit-tested. SafeStrCopy has internal linkage, and its truncation path cannot be driven through the public C API without a device whose name exceeds the buffer. Worth revisiting if the helper is ever made testable.

CI

windows-latest moved to the windows-2025-vs2026 image, which ships Visual Studio 2026. The workflows pin CMake to 3.29, and the "Visual Studio 18 2026" generator was only added in CMake 4.2, so CMake stopped recognising any Visual Studio on the runner and fell back to the NMake Makefiles generator. Configuration then failed before compiling anything. The two jobs that compile are now pinned to windows-2022. Note that windows-2025 is not an alternative, since it ships VS 2026 as well.

Not included

Two follow-ups from the same issue, deliberately left out to keep this reviewable and ABI-safe:

  1. Firmware revision. Cheap, since bcdDevice is already retrieved into HubConnectionInfo::_deviceDescriptor and simply never propagated. The reporter's second screenshot is USBDeview showing 30.98, which confirms it is bcdDevice (0x3098, BCD-formatted) rather than a SCSI INQUIRY revision.
  2. Populating the USB enumeration path. The screenshot shows Description, DeviceId, DevicePath, FriendlyName, Manufacturer and ProductName all empty. SetFriendlyName and SetDescription are only called from EnumerateByDeviceClass, never from EnumerateUsbDevices. This is a wider gap than the issue's "DeviceName would also be good" suggests.

Note that "Device Name: 40AY" in the USBDeview screenshot is not the friendly name. USBDeview's own Friendly Name field is empty there, and 40AY looks derived from the instance ID (USB\VID_17EF&PID_30A9\1S40AY0090E). Exposing the Instance ID as a proper field is probably the better answer than trying to replicate it.

Item 1 breaks the WD_DEVICE_INFO ABI and needs a version bump, so it belongs in its own PR.

The native C API writes every WD_DEVICE_INFO char[] field as UTF-8, but the
managed struct declared those fields as CharSet.Ansi/ByValTStr, so the CLR
decoded them using the system ANSI codepage. Every non-ASCII character was
corrupted: on a German system "Billboard-Gerät" arrived as "Billboard-Gerät"
(U+00E4 is UTF-8 C3 A4, misread as two CP1252 characters).

Marshal the fields as raw byte buffers and decode them explicitly as UTF-8 via
the new Utf8Buffer.ToStringZ helper. Byte buffers occupy exactly the same space
as the ByValTStr fields they replace, so WD_DEVICE_INFO's layout is unchanged
and the wrapper stays binary-compatible with an already-deployed WinDevices.dll.
A layout test asserts the struct size and every field offset to keep it that way.

Also fix SafeStrCopy's truncation path. It clamped the output length and passed
that to WideCharToMultiByte, but Win32 does not truncate on a short buffer - it
fails with ERROR_INSUFFICIENT_BUFFER and leaves the destination undefined. Now
the conversion goes through a temporary buffer and truncation backs off to a
UTF-8 character boundary, so a multi-byte sequence is never split.

PtrToStringAnsi -> PtrToStringUTF8 for the version build date and error message
strings, so the whole boundary decodes consistently. Both are ASCII today, so
this is a consistency change rather than a behaviour change.

Fixes #1 (encoding portion)
@TorinKS
TorinKS force-pushed the fix/utf8-string-marshaling branch from 80b4ccf to aaebaf4 Compare August 24, 2026 15:47
GitHub moved windows-latest onto the windows-2025-vs2026 image, which ships
Visual Studio 2026. The workflows pin CMake to 3.29, and the
"Visual Studio 18 2026" generator was only added in CMake 4.2, so CMake no
longer recognises any Visual Studio on the runner. It silently falls back to
the NMake Makefiles generator, and because nothing activates an MSVC developer
environment, configuration dies before compiling anything:

    -- Building for: NMake Makefiles
    CMake Error at CMakeLists.txt:57 (project):
      Running 'nmake' '-?' failed with: no such file or directory
    CMake Error: CMAKE_CXX_COMPILER not set, after EnableLanguage

Pin the two jobs that actually compile to windows-2022, which is still GA and
carries VS 2022. Note that windows-2025 is not an alternative - it ships VS 2026
as well; windows-2022 is the only label that still provides VS 2022.

The summary and increment-version jobs run PowerShell only and have no toolchain
dependency, so they stay on windows-latest.

This pins the runner image alongside the already-pinned CMake version. Pinning
only the tool while the image floats is what let the two drift apart. Bumping
CMake to >= 4.2 and staying on windows-latest is the alternative when VS 2022
images are eventually retired.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant