Bitvise SFTP Extensions ============================================================================== Copyright (C) 2024 by denis bider and Bitvise Limited. Last updated: 2024-04-12 URL: https://www.bitvise.com/files/specs/sftp-bitvise-extensions.txt THIS DOCUMENT AND THE INFORMATION HEREIN ARE PROVIDED ON AN "AS IS" BASIS. THE AUTHOR (OR AUTHORS) AND BITVISE LIMITED DISCLAIM ALL WARRANTIES, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTY THAT THE USE OF THE INFORMATION HEREIN WILL NOT INFRINGE ANY RIGHTS OR ANY IMPLIED WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. ------------------------------------------------------------------------------ 1. Introduction This documents the following SFTP protocol extensions as supported in Bitvise SSH Server, SSH Client and FlowSsh: - max-io@bitvise.com SSH_FXP_VERSION extension. - intended-size@bitvise.com SSH_FXP_VERSION advertisement and ATTRS extended attribute for SSH_FXP_OPEN. - no-buffering@bitvise.com SSH_FXP_VERSION advertisement and ATTRS extended attribute for SSH_FXP_OPEN. - check-file-blocks SSH_FXP_VERSION supported2 advertisement. 2. Mechanisms The SFTP protocol supports clients and servers to advertise support for extended functionality as part of SSH_FXP_INIT and SSH_FXP_VERSION. SFTP protocol versions 3 and higher also support the inclusion of arbitrary attribute name-value pairs in the ATTRS structure. The ATTRS structure is part of several SFTP requests and responses, including SSH_FXP_OPEN, SSH_FXP_MKDIR, SSH_FXP_SETSTAT, SSH_FXP_FSETSTAT, SSH_FXP_NAME, and SSH_FXP_ATTRS: SFTP version 3: https://tools.ietf.org/html/draft-ietf-secsh-filexfer-02 SFTP version 4: https://tools.ietf.org/html/draft-ietf-secsh-filexfer-04 SFTP version 6: https://tools.ietf.org/html/draft-ietf-secsh-filexfer-13 Extensions: https://tools.ietf.org/html/draft-ietf-secsh-filexfer-extensions-00 3. max-io@bitvise.com SFTP v6 allows servers to advertise support for large SSH_FXP_READ sizes using the "supported2" extension in SSH_FXP_VERSION. However, there is no provision for the server to communicate support for large SSH_FXP_WRITE requests, and the "supported2" extension is unsuitable with SFTP versions lower than 6. The following extension MAY be included by a server in its SSH_FXP_VERSION response when using SFTP versions 3 and higher: string extension_name = "max-io@bitvise.com" string extension_data = encoding of: uint32 max-read-size uint32 max-write-size In this case, extension_data is a string of length 8 bytes, where the string content encodes the maximum read and write sizes accepted by the server. Since this extension MAY be used independently of the extension "supported2", in particular when using SFTP versions 3-5, this extension duplicates the max-read-size field which is already present in the "supported2" extension. A server that sends both extensions SHOULD use the same max-read-size in both. By sending this extension, the server advertises that the client MAY send SSH_FXP_READ requests with 'len' field values up to max-read-size; and SSH_FXP_WRITE requests where the 'data' field contains up to max-write-size bytes. The server MUST accept SFTP packets large enough to accommodate the server's advertised max-write-size. A server SHOULD NOT advertise a max-read-size or max-write-size smaller than the minimum supported size suggested by the SFTP protocol (32,768 bytes). 4. intended-size@bitvise.com This is an extended attribute which MAY be sent by a client in SSH_FXP_OPEN. Servers and clients MUST ignore this attribute in any other context. A server which supports this attribute SHOULD indicate this in its SSH_FXP_VERSION by including the following extension fields: string extension_name = "intended-size@bitvise.com" string extension_data = "1" (string of length 1 containing byte 0x31) A client MUST accept any value of extension_data as indicating support for this attribute, including an empty extension_data. A client MAY attempt to use this attribute with servers which do not advertise it. To ensure compatibility, a client MAY restrict the use of this attribute only to servers which do advertise it. To use this attribute, a client includes the following as part of the ATTRS structure in SSH_FXP_OPEN for a file it intends to upload to the server: string extended_type = "intended-size@bitvise.com" string extended_data = encoding of: uint64 intended-size In this case, extended_data is a string of length 8 bytes, where the string content encodes the final upload file size intended by the client. The server can use this information to confirm a successful upload or to diagnose issues where a transfer did not complete as intended. When using SSH_FXF_TEXT_MODE, intended-size SHOULD reflect the final intended size of the file using the wire text-mode encoding (CR LF newlines). 5. no-buffering@bitvise.com This is an extended attribute which MAY be sent by a client in SSH_FXP_OPEN. Servers and clients MUST ignore this attribute in any other context. A server which supports this attribute MAY indicate this in its SSH_FXP_VERSION by including the following extension fields: string extension_name = "no-buffering@bitvise.com" string extension_data = encoding of: uint32 optimal-alignment In this case, extension_data is a string of length 4 bytes, where the string encodes the server's optimal offset and size alignment for unbuffered I/O. Common values for optimal-alignment include 512 or 4096. A server MUST accept SSH_FXP_READ and SSH_FXP_WRITE packets with data sizes up to the advertised optimal-alignment. To use this attribute, a client includes the following as part of the ATTRS structure in SSH_FXP_OPEN: string extended_type = "no-buffering@bitvise.com" string extended_data = "y" or "n" (a string of length 1 with this value) A client's inclusion of this attribute in SSH_FXP_OPEN indicates the client's preference about whether the server should open the file for unbuffered I/O: - If extended_data == "y", the client prefers unbuffered I/O. On Windows, this translates to the CreateFileW flag FILE_FLAG_NO_BUFFERING. - If extended_data == "n", the client prefers normal, OS-buffered I/O. If the server uses unbuffered I/O by default, this allows the client to disable it. When processing SSH_FXP_OPEN with no-buffering == "y", the server MUST apply any additional inferred flags that were not specified by the client which are necessary for unbuffered I/O, as long as the inferred flags do not conflict with flags the client did specify. In particular, on Windows, the server MUST infer SSH_FXF_BLOCK_READ, SSH_FXF_BLOCK_WRITE and SSH_FXF_BLOCK_DELETE, since files cannot be opened for unbuffered I/O with non-zero dwShareMode. The client's inclusion of this attribute is advisory and non-binding. The server MAY open the file with normal buffering even if the client requests unbuffered I/O, or unbuffered even if the client requests normal buffering. The purpose of this attribute is to provide the means for a client to request server functionality which involves performance and functionality tradeoffs. The purpose of this attribute is not to complicate client-server interaction. If the server opens the file for unbuffered I/O, the burden of meeting any I/O alignment requirements falls on the server. The client SHOULD send read and write requests aligned to server's optimal-alignment, but is not required to do so. The server MUST ensure that SSH_FXP_READ, SSH_FXP_WRITE and SSH_FXP_FSETSTAT perform consistently with a file that uses normal buffering. 6. check-file-blocks The SFTP v6 Extensions draft specifies the "check-file" extension. A server may advertise this as part of the "supported2" extension in SSH_FXP_VERSION. A server advertising "check-file" implies that clients may send extended SFTP requests "check-file-name" and/or "check-file-handle", defined in the draft. In practice, a variety of servers advertise "check-file" while supporting only partial functionality of "check-file-name" and "check-file-handle" requests. Some tested servers support only whole-file hashing, and some even disconnect if the file for which the client requested hashing is too large. As a result, clients cannot use the presence of the "check-file" advertisement to decide if the server supports block-by-block hashing of arbitrary sections of a file. This obstructs the use of hashing to detect differences in file content. Servers which support "check-file-name" and "check-file-handle", and also support block-by-block hashing, SHOULD advertise "check-file-blocks" as part of the supported2 extension in SSH_FXP_VERSION, in addition to "check-file". A server which advertises "check-file-blocks" MUST support all of the below: - Hashing a file block-by-block (i.e. not only the whole file). - Hashing a file block-by-block starting from an arbitrary start-offset. - Hashing using any block-size equal to or larger than 256 bytes. - Hashing an arbitrarily small, non-empty block at end of a file. For example: a server-side file contains 1025 bytes. The client sends the extended request "check-file-handle" with start-offset = 0, block-size = 256, length = 2048. The server MUST return 5 hash digests, where the first 4 hashes correspond to the first 1024 bytes, and the final hash is over the last byte. To protect against denial of service, servers SHOULD control the speed at which requested hashing is performed, rather than fail a well-formed request. ------------------------------------------------------------------------------