Skip to content

try copy() instead of using shell cp - #61

Closed
crawld wants to merge 2 commits into
fsspec:mainfrom
crawld:main
Closed

try copy() instead of using shell cp#61
crawld wants to merge 2 commits into
fsspec:mainfrom
crawld:main

Conversation

@crawld

@crawld crawld commented Sep 28, 2025

Copy link
Copy Markdown

Problem

cp_file (and the move fallback) shells out to cp over an SSH exec channel. SFTP-only servers that deny exec (chrooted internal-sftp, transfer appliances) can't copy at all, and the remote must have a POSIX shell with cp.

Change

When the channel advertises the copy-data extension (supports_remote_copy, asyncssh >= 2.19 talking to e.g. OpenSSH >= 9.0), copy server-side over SFTP with channel.copy(..., remote_only=True). Everything else keeps the shell cp fallback, byte-for-byte as today.

Two guards matter:

  • supports_remote_copy gate — plain channel.copy() never raises SFTPOpUnsupported unless remote_only=True is passed; without the gate, servers lacking the extension would be silently "copied" through the client (download + re-upload). On asyncssh <= 2.18 copy() is only the client-side round-trip, so the gate (via getattr) also keeps old asyncssh on the shell path. No version pin change needed.
  • remote_only=True — belt and braces so asyncssh can never fall back to the client-side copy.

Tests

test_cp_file_remote_copy / test_cp_file_shell_fallback pin the gating with fake channels; the existing test_copy keeps covering the shell fallback end to end (mockssh has no copy-data).

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR updates the SSHFS copy implementation to prefer an SFTP server-side copy operation (when supported) before falling back to running cp over an SSH shell session.

Changes:

  • Attempt channel.copy() using the SFTP “copy-data” extension and fall back to shell cp if unsupported.
  • Keep existing shell-based cp execution path as a compatibility fallback.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread sshfs/spec.py Outdated
crawld and others added 2 commits August 6, 2026 12:27
Server-side copy over SFTP (asyncssh >= 2.19 with an OpenSSH >= 9.0
server) needs no shell access and keeps the data on the server, which
also unblocks copy/move on SFTP-only servers that deny exec. The
branch is gated on supports_remote_copy: without it asyncssh would
silently copy through the client (download + re-upload) on servers
lacking the extension, and asyncssh <= 2.18 has no server-side copy at
all. remote_only=True keeps that guarantee even if the gate is ever
bypassed. Everything else falls back to the shell cp as before.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A channel advertising supports_remote_copy must be used with
remote_only=True and no shell; a channel without it (or an asyncssh
without the attribute) must use the shell fallback. test_copy keeps
covering the fallback end to end against mockssh.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@shcheklein

Copy link
Copy Markdown
Collaborator

Thanks @crawld — the direction was right, and I've finished it up (rebased onto main, kept your authorship on the code change):

  • 53e1609 — your copy-data idea, now gated on supports_remote_copy and passing remote_only=True. Without those, channel.copy() never raises SFTPOpUnsupported — asyncssh silently copies through the client (download + re-upload) on servers lacking the extension, so the shell fallback was dead code and non-supporting servers would have regressed. asyncssh <= 2.18 has no server-side copy at all, which the getattr gate also handles — no pin bump needed.
  • 4d6116a — tests pinning both branches of the gate.

Details in the updated description.

@shcheklein

Copy link
Copy Markdown
Collaborator

Heads-up: GitHub doesn't fire workflow runs for maintainer pushes to a fork's default branch (this PR is from crawld:main), so CI can't run here — the reworked change moved to #74 with your commit authorship preserved. It will close this PR when it merges. Thanks again for kicking this off!

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated no new comments.

Suppressed comments (2)

sshfs/spec.py:274

  • _cp_file now always acquires an SFTP channel just to check supports_remote_copy, even when the server doesn’t support remote copy and the function immediately falls back to shell cp. This can unnecessarily consume/potentially create SFTP channels and add overhead on the common fallback path. Consider caching the capability after the first check so subsequent fallbacks can go straight to the shell path without touching the SFTP pool.
        async with self._pool.get() as channel:
            if getattr(channel, "supports_remote_copy", False):
                return await channel.copy(lpath, rpath, remote_only=True)

        cmd = f"cp {shlex.quote(lpath)} {shlex.quote(rpath)}"

tests/test_sshfs.py:272

  • The test comment says it covers the asyncssh < 2.19 case where supports_remote_copy doesn’t exist, but the fake Channel defines supports_remote_copy = False. This leaves the “missing attribute” behavior untested. Add a second fake channel without the attribute (or parametrize) so the getattr/default path is actually exercised.
def test_cp_file_shell_fallback(fs, monkeypatch):
    # Without copy-data support (or on asyncssh < 2.19, where the
    # attribute does not exist), the shell cp path is used.
    calls = {}

    class Channel:
        supports_remote_copy = False

@shcheklein

Copy link
Copy Markdown
Collaborator

Closing in favor of #74

@shcheklein shcheklein closed this Aug 6, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants