Support blob type visualization - #72
Conversation
gordonmurray
left a comment
gordonmurray
left a comment
There was a problem hiding this comment.
Thanks for implementing media detection and inline previews. This is a useful direction for issue #3, but I’m requesting changes before merge.
I need to define the binary/media response contract before this implementation can land. The viewer must not embed arbitrary-size images, audio, or video as base64 in every /rows response. That would make large binary datasets substantially slower and could worsen the existing large-payload problem. Browser-side loading="lazy" does not avoid transferring the base64 data already present in the JSON response.
The intended design should be:
- small media may be returned inline under a documented size limit;
- larger media returns metadata only, without a base64 payload;
- loading large media happens through an explicit, bounded/lazy request or user action;
- the API contract clearly distinguishes media metadata from inline content.
I’ll follow up with the exact size limit and endpoint contract. After that, please revise the implementation and add:
- API-level tests for small and oversized media values;
- compatibility tests for ordinary and fixed-size binary columns;
- the media response contract in docs/spec.md;
- the required [Unreleased] CHANGELOG.md entry referencing issue #3.
The preview UX is beneficial, but the current eager full-payload behavior is not safe to merge.
|
Hi @gordonmurray thanks for the feedback! On inline storage fetch vs extra access, I'm thinking whether we could
|
|
This eagerly embeds the complete blob as base64 in every /rows response. With up to 1,000 rows, large media columns can create huge responses, increase server memory/CPU usage, and freeze the browser. loading="lazy" does not help because the payload has already been transferred and parsed. Could we enforce a strict size limit or return media metadata with a separate, bounded media endpoint instead? This should be addressed before merging because it can make the viewer unusable for ordinary media datasets. |
"BM" and "ID3" are printable, so text that starts with them was detected as a bitmap or an MP3. A binary column with the text "BM25 scoring" returned a media object, and the page rendered a broken image. Both checks now read the structure behind the magic bytes. A bitmap must carry a known DIB header size. An ID3 tag must carry a valid version and a synchsafe size.
Every media value went into the rows response as base64. A page of 1000 rows with large images or video built a response of hundreds of megabytes, which used server memory and stopped the browser. Media of 64 KiB or less is still sent inline. Larger media returns the type and the size with no payload, and the cell shows what it holds. The worst case for a page is now near 4 MB. The new "inline" flag tells the two apart, so a later change can fetch a large value on demand.
A bare MPEG frame header is 11 bits, and UTF-16 text opens with the same two bytes. A note held in a binary column returned an audio object, and the page drew a player with nothing behind it. The "ftyp" marker has the same fault. It is four printable characters, four bytes into the value. An MP3 now needs an ID3 tag. An MP3 without a tag returns base64, as it did before media detection. A video now needs a box size in front of the marker. That size must be at least 16 bytes, no longer than the value, and a multiple of 4.
|
Thanks for this @dentiny, Rather than ask for changes, I pushed three commits to your branch. Two tighten the signature checks. Several signatures are printable characters, so ordinary text was detected as media: a column holding "BM25 scoring" came back as a bitmap and drew a broken image. Each check now reads the structure behind the magic bytes rather than the bytes alone. The third caps inline media at 64 KiB. A page of 1000 rows with large images built a response of hundreds of megabytes. Larger values now return their type and size with no payload, and the new "inline" flag lets a later Nothing needed from you here, the rest is up to me |
|
I took the "Closes #3" line out of the description and left the issue open. This covers small values inline, but larger media and Lance blob columns still need work, so the issue is the better place to track that. |
PR #72 added inline preview for media held in binary columns. The entry names it as a preview for small media, because a value above 64 KiB returns its type and size with no payload.
2 participants
Hi team, since lance file format specializes for multi-model support, in this PR I want to add inline visualization for blob-type data, including videos, images, audios, etc.
In simple words, the backend detects media formats from binary signatures and returns MIME metadata with a base64 payload. The frontend renders recognized values as inline images, audio, or video.
Example UI display
Followup items
AI usage disclaimer:
GPT-5.5 helped me make the code change, I deployed my own fork to k8s and verified it worked