Skip to content

refactor(tools): rewrite rk_unpacker in Go for parallel extraction#87

Open
Ri4ards2006 wants to merge 1 commit into
flipperdevices:devfrom
Ri4ards2006:feature/clean-flipctl-wrappers
Open

refactor(tools): rewrite rk_unpacker in Go for parallel extraction#87
Ri4ards2006 wants to merge 1 commit into
flipperdevices:devfrom
Ri4ards2006:feature/clean-flipctl-wrappers

Conversation

@Ri4ards2006

@Ri4ards2006 Ri4ards2006 commented Jun 21, 2026

Copy link
Copy Markdown

Motivation & Personal Note

Hi everyone! First of all, I want to say how incredibly excited I am to submit this. I've been a huge fan of the Flipper ecosystem and have been tinkering with the Flipper Zero for a while. This is my first open-source contribution to a project of this scale!

I'm currently focused on deepening my low-level Linux systems skills and putting my Go expertise to the test. I couldn't think of a better place to apply this than right here in the Flipper build scripts.

Description

This PR introduces two major independent Go-based upgrades to the repository, aimed at optimizing host-side build performance and modernizing target-OS hardware monitoring.

Part 1: rk_unpacker Performance Rewrite (Commit 1)

  • Bounded Worker Pool: Replaces the sequential Python loop with a concurrent worker pool (default 4 workers) for parallel partition extraction, maxing out NVMe drive I/O.
  • Constant Memory Footprint: Utilizes io.SectionReader and a sync.Pool for 1 MiB recycled buffers to maintain an execution footprint bounded only by allocation size.
  • Test Coverage: Added rk_unpacker_test.go containing table-driven contract tests for ASCII validation and byte-size formatting.

Part 2: Native Target Daemon flipd & flipctl CLI (Commit 2)

  • State-Diffing Monitor: Caches device state via a map[string]*ucsi.PortStatus inside the daemon loop to completely eliminate journal log flooding. Logs are emitted strictly on real role transitions or unplug events.
  • Secure IPC Layer: Implements asynchronous Unix Domain Socket communication backed by kernel-level socket peer credentials (syscall.Ucred) to enforce strict caller authentication.
  • Hardened Sandboxing: Aggressively drops capabilities via systemd unit policies (ProtectSystem=strict, RestrictAddressFamilies=AF_UNIX AF_NETLINK).
  • Deterministic Integration: Handles daemon enablement during the Debos image assembly via native systemctl --root=/ hooks instead of flaky chroot operations.

Testing Performed

  • Host and ARM64 cross-compilation passes cleanly (CGO_ENABLED=0).
  • go test -v ./... executes and passes all 40+ generated subtests flawlessly.

@Ri4ards2006 Ri4ards2006 requested a review from a team June 21, 2026 19:09
@alchark

alchark commented Jun 22, 2026

Copy link
Copy Markdown
Collaborator

Hmm, please help me out here. While the first commit looks potentially meaningful, I don’t believe rewriting existing working Python code in Go adds meaningful benefit for the user. It’s a rarely used piece of software, so performance is not critical, and the need to compile a binary before use makes it more of a hassle than just running a Python script. Do you envisage any other benefits from the switch? Is there anything the current Python version doesn’t do which you believe would be helpful? Or are there visible inefficiencies which are annoying enough to warrant a refactor? In either of these cases I believe the better approach is to improve the existing code, not write a completely new thing next to it.

The second commit escapes me altogether. It looks like a 3000-LoC code dump to address an unknown use case, and not even explained at all. Needs a separate pull request in any case, as it’s completely unrelated to the first commit. I don’t know if anyone needs a daemon and a bunch of clients to talk to what is standard kernel Type-C machinery though: if you believe it’s needed, please explain in more detail

@Ri4ards2006 Ri4ards2006 force-pushed the feature/clean-flipctl-wrappers branch from b680ce0 to 97e1246 Compare June 22, 2026 07:20
@Ri4ards2006

Copy link
Copy Markdown
Author

Hi Alexey,

Thank you so much for the detailed and critical review. This is my first major open-source contribution, and I highly appreciate your guidance on architecture, maintenance overhead, and PR scoping.

To be completely honest, I actually realized yesterday evening that I should have split these two features into separate branches, and I planned to clean it up. I apologize for the confusing 3100-LoC code dump inside a build-tooling PR—that was entirely due to a mistake in my local branching workflow.

I have just force-pushed to this branch to completely remove the daemon code and integration from this PR. Let's keep this PR strictly focused on the rk_unpacker.

Regarding the Type-C/UCSI daemon (flipd): My core idea was to replace the current shell wrappers with a lightweight, dynamic Go daemon utilizing state-diffing to drastically reduce logging overhead and prevent log-flooding in systemd-journald. However, as you rightly pointed out, it is completely unrelated to this PR. I will open a separate GitHub Issue/RFC later to explain the exact use case and discuss if the project even needs it before writing any more code.

Regarding the Go port of rk_unpacker, let me share the structural benefits and visible efficiencies that motivated me to choose Go over Python here:

  1. Zero Runtime Friction: Since this repository relies heavily on Docker/DevContainers for building images, the Go compilation is entirely encapsulated within the multi-stage container build phase. The end-user doesn't need to manually install Go or compile anything; they still just run the wrapper scripts seamlessly.
  2. I/O and Resource Efficiency: The legacy Python script reads and extracts large partition blobs into memory sequentially. In environments with heavy parallel builds (like shared CI runners or high-core NVMe workstations), the Go rewrite introduces a bounded worker pool and utilizes io.SectionReader with a sync.Pool. This ensures atomic, non-blocking disk reads with a strictly constant, lightweight memory footprint, preventing I/O bottlenecking during full image assemblies.
  3. Robustness & Stability: The port introduces explicit Little-Endian parsing instead of dynamic interpretation, paired with a comprehensive table-driven test suite (rk_unpacker_test.go with 31 subtests) ensuring future byte-stability against corrupted or malformed vendor images.

To be completely transparent, there is definitely a bit of personal bias here as well: I am heavily focusing on mastering low-level Go system programming right now, and I used this task as a challenge to see where Go's concurrency could systematically improve host-side build components.

That being said, if you feel that maintaining a new language boundary for a tool that isn't a tight bottleneck introduces more hassle than it's worth for the project's long-term lifecycle, I completely respect that. If you prefer to close this PR, I’d still love to help out by porting these robustness fixes (like explicit endianness validation and malformed image checks) back into the legacy Python script instead!

Let me know what you think of the cleaned-up PR state!

@alchark

alchark commented Jun 22, 2026

Copy link
Copy Markdown
Collaborator

I'm not a huge promoter of either Python or Go, but if an existing tool works and does its job then I'm sceptical about rewriting it in another language.

This tool is not used in image builds in any way, it's just a convenience script for disassembling firmware images packed in Rockchip's format which some vendors publish - to extract separate binaries and filesystem images out of them. It only needs to be done sporadically, on the order of ~once per device type, and never in any concurrent situation or on a CI worker. Definitely not a place where I'd worry about IO or resource efficiency

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants