Creating file on AWS SFTP managed services always get "Cannot open file in mode: WRITE|TRUNCATE|CREATE|READ" (SSH_FX_OP_UNSUPPORTED)

[rama3i]

When using AWS SFTP managed services, sftp client always failed to create files. Create function (client.Create() ) always returned error with message "Cannot open file in mode: WRITE|TRUNCATE|CREATE|READ" (SSH_FX_OP_UNSUPPORTED).

This error only happened on AWS SFTP managed services. Regular SFTP server has no issue. The same endpoint were tested using sftp command, FileZilla and even .Net sftp library and all working normally. Only go sftp package were returned the error.

[eikenb]

Hey @rama3i, thanks for the report.

This one is going to be tricky to debug as I don't have access to an AWS account through work at the moment and AWS Transfer is pretty expensive w/ no free tier. So any additional information you can supply would be great. Thanks.

[rama3i]

Hi @eikenb ,

I am not really expert on neither Go nor sftp, but can you show me how to provide more info? Is there anything I could do to verbosely log everything to provide you any detail required?

[sugacube]

i'm a quite time constrained at the moment so difficult to dive deeply, but I had some thought about it. I could be totally wrong here but maybe worth a shot.

As aws sftp is s3 backed, maybe can't support read/write connection at the same time. maybe client.Create() should only open for writing not for reading ? @rama3i maybe you could try to change that in the code and try to see if that would work? (e.g. in client.go function Create() change os.O_RDWR into os.O_WRONLY) ?

I'm not sure how other sftp clients open for write - i tried looking around in the code of filezilla but could not immediately find relevant code. mabye @eikenb or someone else has looked around in the past in other clients maybe worth checking how this is done in other clients.

other thought I had was aws permissions, but that can't be the reason as apparently other clients work.

[sugacube]

just checked source code of winscp. winscp (client) appears afaict when uploading a file to open in mode write/create or write/create/truncate which seems to confirm my suspicion that removing the "READ" permission might solve the issue.

[eikenb]

Thanks for the investigation and thoughts @sugacube. I checked openssh's sftp client and it behaves similarly. Opening a new file with either uses write/create/truncate or write/create/append. No read anywhere in the upload call.

@rama3i you can test this idea out by replacing your client.Create(path) call with client.OpenFile(path, os.O_WRONLY|os.O_CREATE|os.O_TRUNC). This would replicate what winscp and openssh do opening the file for write only. If you get a chance to try this, please let me know if it helps.

[rama3i]

Thanks @sugacube @eikenb . The proposed solution works as expected. Using os.O_WRONLY|os.O_CREATE|os.O_TRUNC as flag I could create the file on AWS managed SFTP.

Why do this option matters? Will this solution works for regular SFTP? I actually think that the one causing error is the os.O_RDWR and os.O_TRUNC, and removing two of them leaving for only os.O_CREATE should works. Sadly that's not working. That's why I raise this issues.

[eikenb]

The problem here is the os.O_RDWR is used with the client.Create() call. That could be considered a bug... but API is designed to mimic the filesystem API and os.Create() opens files using O_RDWR (as well as O_CREATE and O_TRUNC).

IMO this really comes back around to an issue with treating remote SFTP sites the same as local filesystems. They are not and the actions are different, but that decision is in the past and how do we want to handle this. Just changing O_RDWR to O_WRONLY would fix the issue here but might break someone else code and it seems like many SFTP servers handle this just fine. Also it is completely within spec to open a new file with both read and write flags in this way (ie. AWS is out of spec here).

Short term I think you, @rama3i, are best off just using client.OpenFile() to create files for now. We could also document this fact about AWS.

Longer term we could think about adding some new client methods that are more oriented towards file transfers (eg. Put, Get, etc.) that would mimic Openssh's commands instead of filesystem commands. We could then guide people to one or the other based on needs.

[puellanivis]

os.Create() opens files using O_RDWR

Wait… what?! 👀

I always just assumed that it was opened in O_WRONLY because the O_CREATE and O_TRUNC flags passed in make its usefulness for reading neigh absent.

P.S.: Agree, AWS should be considered out-of-spec here, and 🤷‍♀️ not all workarounds should be made.

[eikenb]

@puellanivis ... 🤣 ... No. They replicated the os package directly, including the flags used in file operations. I think they wanted the sftp files to be API compatible with os files. I understand what they were going for.. but am unsure it works well in practice for all cases.

[puellanivis]

I mean, it makes sense why this repo duplicated what os.Create does, but I never would have thought it would have read permissions.

[sugacube]

To me (and you) it may be counter intuitive, but I justs saw it's crystal clear in the docs - it's repeated 3 times :

func (c *Client) Create(path string) (*File, error)
Create creates the named file mode 0666 (before umask), truncating it if it already exists. If successful, methods on the returned File can be used for I/O; the associated file descriptor has mode O_RDWR. If you need more control over the flags/mode used to open the file see client.OpenFile.

It's possible (but not very common) for an SFTP server to allow either read or write, but pretty unusual to offer both read and write but not at the same time. This must be a very AWS specific thing.

[puellanivis]

I am fairly sure this “feature” of Amazon is a result of quirks of how their s3 buckets work.

[rama3i]

I am fairly sure this “feature” of Amazon is a result of quirks of how their s3 buckets work.

Make sense. Object on S3 bucket allows write and read, but might not both. When we open, it actually download the file, not the actual file open like the one on the OS directly to the path (please pardon my lack of technical terms and knowledge). This might be happened for other SFTP that backed with Object Store, such as Open Stack Object Storage (Swift) and others. AFAIK object store only support Put and Get. So yeah, this might be the reason.

As this is not a common case, I think well documented workaround should be enough. At least developer know if Create fails on Object Store backed SFTP, they could use Open with os.O_WRONLY|os.O_CREATE|os.O_TRUNC flag

[puellanivis]

Solving the problem with documentation is not a bad idea. We can add a note under the godoc for Create that some backends like AWS SFTP do not support O_RDWR which is used by Create and provide the quick-workaround to switch to OpenFile(path, os.O_WRONLY|os.O_CREATE|os.O_TRUNC) in that case.

P.S.: trying to write some fallback code to try the OpenFile route if the Create fails with FX_OP_UNSUPPORTED creates an interesting situation where I can get the statusErr.Code but I cannot directly compare it against an available value without a type conversion. (statusErr.Code != sftp.ErrSshFxOpUnsupported (mismatched types uint32 and sftp.fxerr))

Perhaps the statusErr.Code should be typed to match the sftp.Err… values?

[rama3i]

Looking at the code, statusError.Code were taken from 32 bit of first data (CMIIW). Changing to int might cause some data lost. I think casting sftp.ssh_FX_OP_UNSUPPORTED to uint32 would be safer.

Trying to do the same as @puellanivis did.

func (c *Client) Create(path string) (*File, error) {
	file,err := c.open(path, flags(os.O_RDWR|os.O_CREATE|os.O_TRUNC))
	// AWS managed SFTP issues workaround. Need to check whether os.ORDWR is supported or not, if not (indicated by
	// error code ssh_FX_OP_UNSUPPORTED) try to reopen using os.O_WRONLY
	statusError, ok := err.(*StatusError)
	if err != nil && ok && statusError.Code == uint32(ssh_FX_OP_UNSUPPORTED){
		return c.open(path, flags(os.O_WRONLY|os.O_CREATE|os.O_TRUNC))
	}
	return file,err
}