AWS SDK breaking changes?

[jrista]

I started working on implementing this today in a project I am working on. I've had quite a bit of trouble, and finally started debugging through the node-sftp-s3 code. I am wondering if the AWS SDK may have changed significantly since the last time this module had a major update?

It looks like the existing code looks for /.dir objects on S3 to determine if an object is a directory. From what I can tell, the way S3 currently works, there are no .dir files. Instead, a directory is simply identified by a trailing / in the name of the object. I am not quite sure how extensive the use of /.dir is, however I have found a few use cases so far. Is there any chance this library could be updated to work with the current version of the AWS SDK?

[cmrigney]

Hi @jrista!

Since there are no "folders" in S3, this library emulates them by creating empty files ending in .dir. When there is a .dir there is a "folder".

Anyway, regarding your issue. I just published a new version to npm - v0.0.9 - which is the first release in about a year. Try that and see if it fixes your issue and ping me back.

[jrista]

Hmm...I am not sure that adding a .dir file is really necessary. With S3, object names that represent directories in S3 (child containers) end in a /. Other file names will usually end in either nothing, or a file extension. So I am not sure that using a .dir file is necessary...just checking that the object name ends in / should be sufficient.

But maybe I need to explain my use case more. I need to provide SFTP access for a large number of clients. Each client that connects to the SFTP server will need to have access only to their own area of the FTP server. From what I could tell, the way your library works, it automatically assumes a default path for each user, based on the username you specified when adding the user's public key.

The thing is, the SFTP client does not automatically create that directory in S3 if it does not exist. I went through and added a bunch of them manually in S3 directly, and I also pre-created various child directories for different kinds of files to be uploaded, but since they lack the necessary .dir meta files, the SFTP library doesn't seem to recognize them, and does not work with them.

Our ultimate goal is that our clients would simply use a simple sftp command to put objects directly to specific child directories, underneath their user-specific directory. They would not be able to create any directories themselves, we don't want them to do that, just upload files to them. We would then automatically kick off background processes by utilizing the events you have in the SFTP library (which, actually, we are not really sure how to do...you don't have any examples of how to hook into those events, so we are kind of guessing).

Do the .dir files contain any kind of meta data in them? I guess, for this to work, we will need to create .dir files in every one of the directories we create manually in S3, so that users can simply upload files directly to the right directories, and not have to create them themselves?

[jrista]

Ok, looks like the .dir files are just empty files. We can create one and manually put it up on S3 in each directory to get this working. I do think that the .dir files could be dropped in favor of just checking if the object names end in a /, if that is possible in a future update. Would make it easier to manage manual directory creation in S3 if necessary.

[cmrigney]

Yes @jrista, sorry for the delayed response. They are just empty files that indicate to the library that a "folder" exists there.

Just to clarify your request, are you asking that we change these metadata files from this abc/123/.dir to abc/123/? Still creating an empty file of course, but just emulating a "folder".

[jrista]

I am saying you don't need a file at all. All you need to do is check if the object name in S3 ends in a trailing slash. If the object name ends in a slash, then it is a "container", a directory. No meta files are required to identify if an object is a container or not.

[jrista]

@cmrigney Sorry, regarding your last question. If you were to CREATE a new folder, then yes, I believe what you stated is correct. I was thinking of the "read" a folder context, forgetting about the "write" a folder. If you want to just create a folder, I do believe all you need to do is include a trailing slash.

I could be wrong about that, however I am able to create empty folders within the AWS admin site for S3 itself, and I believe I've created empty folders in the past with the AWS node SDK as well. I can actually do a quick little test and try as well, as I've got some S3 code I'm working on here.