GH-45523: [R] Implement Utf8View type bindings by thisisnic · Pull Request #49712 · apache/arrow

thisisnic · 2026-04-11T18:15:28Z

Rationale for this change

No bindings for Utf8View type in the R package

What changes are included in this PR?

Implement bindings

Are these changes tested?

Yep

Are there any user-facing changes?

Yep, adding functionality.

AI Usage

Heavily used Codex/Claude here. I'm not confident of every line of code. I read things over, and iterated on it making sure that tests pass and nothing seemed wildly incorrect.

GitHub Issue: [R] An "cannot handle Array of type <utf8_view>" error occurs when reading ipc file include strings output by r-polars #45523

thisisnic · 2026-04-11T18:17:20Z

We should rebase once #49710 is merged as changes from that PR are in this branch as they were needed to make it work.

thisisnic · 2026-05-05T19:00:06Z

+      cpp11::stop(
+          "Cannot convert Dictionary Array of type `%s` to R: dictionary has "
+          "more levels than an R factor can represent",
+          dict_type.ToString().c_str());


We should add a test for this

thisisnic · 2026-05-06T20:49:12Z

+      return this->value_builder_->Append(
+          std::string_view(view_.bytes, static_cast<size_t>(view_.size)));


The old code passed (const char*, int32_t) which matches StringBuilder::Append but not StringViewBuilder::Append (which takes int64_t). Switching to std::string_view works for both builder types.

thisisnic · 2026-05-06T20:51:21Z

+template <typename T>
+class RPrimitiveConverter<T, enable_if_string_view<T>>
+    : public PrimitiveConverter<T, RConverter> {
+ public:
+  Status Extend(SEXP x, int64_t size, int64_t offset = 0) override {
+    RVectorType rtype = GetVectorType(x);
+    if (rtype != STRING) {
+      return Status::Invalid("Expecting a character vector");
+    }
+    return UnsafeAppendUtf8Strings(arrow::r::utf8_strings(x), size, offset);
+  }
+
+  void DelayedExtend(SEXP values, int64_t size, RTasks& tasks) override {
+    auto task = [this, values, size]() { return this->Extend(values, size); };
+    tasks.Append(false, std::move(task));
+  }
+
+ private:
+  Status UnsafeAppendUtf8Strings(const cpp11::strings& s, int64_t size, int64_t offset) {
+    RETURN_NOT_OK(this->primitive_builder_->Reserve(size - offset));
+    const SEXP* p_strings = reinterpret_cast<const SEXP*>(DATAPTR_RO(s)) + offset;
+
+    int64_t total_length = 0;
+    for (R_xlen_t i = offset; i < size; i++, ++p_strings) {
+      SEXP si = *p_strings;
+      total_length += si == NA_STRING ? 0 : LENGTH(si);
+    }
+    RETURN_NOT_OK(this->primitive_builder_->ReserveData(total_length));
+
+    p_strings = reinterpret_cast<const SEXP*>(DATAPTR_RO(s)) + offset;
+    for (R_xlen_t i = offset; i < size; i++, ++p_strings) {
+      SEXP si = *p_strings;
+      if (si == NA_STRING) {
+        this->primitive_builder_->UnsafeAppendNull();
+      } else {
+        this->primitive_builder_->UnsafeAppend(CHAR(si), LENGTH(si));
+      }
+    }
+
+    return Status::OK();
+  }
+};


I'm less confident about this entire block of code

thisisnic · 2026-05-06T20:52:43Z

A lot of .Rd updates as I used a newer roxygen2 version.

…ings

Copilot

Pull request overview

This PR adds R bindings and conversion support for Arrow utf8_view / string_view, addressing IPC/table conversion failures for data containing StringView columns.

Changes:

Adds string_view() R data type binding, export registration, and type tests.
Implements R↔Arrow conversion paths for StringView arrays and dictionary values.
Updates generated documentation and related converter support in C++/Python.

Reviewed changes

Copilot reviewed 13 out of 23 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
r/R/type.R	Adds `StringView` R6 type, constructor, and canonical type aliases.
r/R/arrowExports.R	Registers R wrapper for `StringView__initialize`.
r/NAMESPACE	Exports `string_view`.
r/src/datatype.cpp	Maps Arrow `STRING_VIEW` to R `StringView` and initializes `utf8_view`.
r/src/r_to_arrow.cpp	Adds R-to-Arrow StringView conversion and dictionary StringView value handling.
r/src/array_to_vector.cpp	Adds Arrow-to-R StringView and wider dictionary index conversion support.
r/src/arrowExports.cpp	Registers native StringView initialization entry point.
cpp/src/arrow/util/converter.h	Enables dictionary converters for `StringViewType`.
python/pyarrow/src/arrow/python/python_to_arrow.cc	Adjusts Python dictionary StringView append call.
r/tests/testthat/test-Array.R	Adds StringView array round-trip tests.
r/tests/testthat/test-Table.R	Adds table/dictionary StringView and wider index tests.
r/tests/testthat/test-data-type.R	Adds StringView data type and code round-trip tests.
r/DESCRIPTION	Updates roxygen metadata.
r/man/data-type.Rd	Adds `string_view()` documentation entry.
r/man/acero.Rd	Regenerated Acero documentation.
r/man/arrow-package.Rd	Regenerated package author documentation.
r/man/csv_convert_options.Rd	Regenerated CSV conversion docs.
r/man/csv_read_options.Rd	Regenerated CSV read docs.
r/man/CsvReadOptions.Rd	Regenerated CSV read options docs.
r/man/enums.Rd	Regenerated enum documentation.
r/man/JsonFileFormat.Rd	Regenerated JSON file format docs.
r/man/reexports.Rd	Regenerated reexports docs.
r/man/vctrs_extension_array.Rd	Regenerated vctrs extension docs.

Files not reviewed (10)

r/man/CsvReadOptions.Rd: Language not supported
r/man/JsonFileFormat.Rd: Language not supported
r/man/acero.Rd: Language not supported
r/man/arrow-package.Rd: Language not supported
r/man/csv_convert_options.Rd: Language not supported
r/man/csv_read_options.Rd: Language not supported
r/man/data-type.Rd: Language not supported
r/man/enums.Rd: Language not supported
r/man/reexports.Rd: Language not supported
r/man/vctrs_extension_array.Rd: Language not supported

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

+    int64_t total_length = 0;
+    for (R_xlen_t i = offset; i < size; i++, ++p_strings) {
+      SEXP si = *p_strings;
+      total_length += si == NA_STRING ? 0 : LENGTH(si);
+    }
+    RETURN_NOT_OK(this->primitive_builder_->ReserveData(total_length));
+
+    p_strings = reinterpret_cast<const SEXP*>(DATAPTR_RO(s)) + offset;
+    for (R_xlen_t i = offset; i < size; i++, ++p_strings) {
+      SEXP si = *p_strings;
+      if (si == NA_STRING) {
+        this->primitive_builder_->UnsafeAppendNull();
+      } else {


Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot

Pull request overview

Copilot reviewed 13 out of 23 changed files in this pull request and generated 1 comment.

Files not reviewed (10)

r/man/CsvReadOptions.Rd: Language not supported
r/man/JsonFileFormat.Rd: Language not supported
r/man/acero.Rd: Language not supported
r/man/arrow-package.Rd: Language not supported
r/man/csv_convert_options.Rd: Language not supported
r/man/csv_read_options.Rd: Language not supported
r/man/data-type.Rd: Language not supported
r/man/enums.Rd: Language not supported
r/man/reexports.Rd: Language not supported
r/man/vctrs_extension_array.Rd: Language not supported

+    int64_t total_length = 0;
+    for (R_xlen_t i = offset; i < size; i++, ++p_strings) {
+      SEXP si = *p_strings;
+      total_length += si == NA_STRING ? 0 : LENGTH(si);
+    }
+    RETURN_NOT_OK(this->primitive_builder_->ReserveData(total_length));
+
+    p_strings = reinterpret_cast<const SEXP*>(DATAPTR_RO(s)) + offset;
+    for (R_xlen_t i = offset; i < size; i++, ++p_strings) {
+      SEXP si = *p_strings;
+      if (si == NA_STRING) {
+        this->primitive_builder_->UnsafeAppendNull();
+      } else {


github-actions Bot added Component: R Component: C++ awaiting committer review Awaiting committer review labels Apr 11, 2026

thisisnic mentioned this pull request Apr 30, 2026

GH-39603: [R] Error: Cannot convert Dictionary Array of type dictionary<values=large_string, indices=uint32, ordered=0> to R #49710

Open

thisisnic force-pushed the GH-45523-ipc-polars branch 2 times, most recently from e8768d0 to bb711a7 Compare May 5, 2026 16:41

github-actions Bot added the Component: Python label May 5, 2026

thisisnic commented May 5, 2026

View reviewed changes

github-actions Bot added awaiting changes Awaiting changes and removed awaiting committer review Awaiting committer review labels May 5, 2026

thisisnic commented May 6, 2026

View reviewed changes

thisisnic added 9 commits May 28, 2026 10:36

Allow dictionaries from a wider range of types for indices

cd671bb

Implement string view

6e60b01

revert unwanted docs change

bb7468b

add context

fd83134

expand tests

bb684aa

Fix offset bug

f3123bd

auto-update description

24e6055

Fix DictionaryBuilder<StringViewType> Append signature in Python bind…

a9788ba

…ings

Fix DictionaryBuilder::Append for StringViewType in Python bindings

cf9ad10

Copilot AI review requested due to automatic review settings May 28, 2026 09:37

thisisnic force-pushed the GH-45523-ipc-polars branch from de59365 to cf9ad10 Compare May 28, 2026 09:37

Copilot started reviewing on behalf of thisisnic May 28, 2026 09:37 View session

github-actions Bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels May 28, 2026

Copilot AI reviewed May 28, 2026

View reviewed changes

Fix docs build issue

a74c9c1

Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>

Copilot AI review requested due to automatic review settings May 28, 2026 12:11

Copilot started reviewing on behalf of thisisnic May 28, 2026 12:11 View session

Copilot AI reviewed May 28, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GH-45523: [R] Implement Utf8View type bindings#49712

GH-45523: [R] Implement Utf8View type bindings#49712
thisisnic wants to merge 10 commits into
apache:mainfrom
thisisnic:GH-45523-ipc-polars

thisisnic commented Apr 11, 2026 •

edited

Loading

Uh oh!

thisisnic commented Apr 11, 2026

Uh oh!

thisisnic May 5, 2026

Uh oh!

thisisnic May 6, 2026

Uh oh!

thisisnic May 6, 2026

Uh oh!

thisisnic commented May 6, 2026

Uh oh!

Copilot AI left a comment

Uh oh!

Uh oh!

Copilot AI left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

		return this->value_builder_->Append(
		std::string_view(view_.bytes, static_cast<size_t>(view_.size)));

Conversation

thisisnic commented Apr 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Rationale for this change

What changes are included in this PR?

Are these changes tested?

Are there any user-facing changes?

AI Usage

Uh oh!

thisisnic commented Apr 11, 2026

Uh oh!

thisisnic May 5, 2026

Choose a reason for hiding this comment

Uh oh!

thisisnic May 6, 2026

Choose a reason for hiding this comment

Uh oh!

thisisnic May 6, 2026

Choose a reason for hiding this comment

Uh oh!

thisisnic commented May 6, 2026

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Reviewed changes

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull request overview

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

thisisnic commented Apr 11, 2026 •

edited

Loading