Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

the upload path is not reasonable when reconnect to hdfs/s3 #140

Open
zhenchuan opened this issue Sep 30, 2014 · 2 comments
Open

the upload path is not reasonable when reconnect to hdfs/s3 #140

zhenchuan opened this issue Sep 30, 2014 · 2 comments

Comments

@zhenchuan
Copy link

when using suro to upload file to s3/hdfs,i found something not reasonable.

if the suro sink can not connect the hdfs, localFile file will be saved as local instead;
when the hdfs was connected again, suro upload the saved files again...

but the uploadPath was constructed using current time .
i think it'll be better to use the file's last modified time to build the upload Path. so that the file i created will be update to the right date directory...

    private String makeUploadPath(File file) {
        return prefixFormatter.get() + file.getName();
    }

https://github.com/Netflix/suro/blob/master/suro-s3/src/main/java/com/netflix/suro/sink/remotefile/RemoteFileSink.java#L214

@metacret
Copy link
Contributor

metacret commented Oct 3, 2014

Right now, RemotePrefixFormatter is not having any parameter. We can change the interface RemotePrefixFormatter.get() as the RemotePrefixFormatter.get(File file) and then we can create another PrefixFormatter which will return the date of file's last modified.

Can you send PR?

@zhenchuan
Copy link
Author

ok,i'll make it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants