Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CloudObjectStorage failed to connect to private COS #48

Open
gilv opened this issue Dec 19, 2017 · 1 comment
Open

CloudObjectStorage failed to connect to private COS #48

gilv opened this issue Dec 19, 2017 · 1 comment

Comments

@gilv
Copy link

gilv commented Dec 19, 2017

I tried to use CloudObjectStorage and connect to my private object storage, but nothing worked.
For example, the following fails:
var credentialsGilCOS = scala.collection.mutable.HashMap[String, String](
"endPoint"->"http://s3-api.us-geo.objectstorage.softlayer.net",
"accessKey"->"AAA",
"secretKey"->"AHA")

var configurationNameGil = "service"
var cosGil = new CloudObjectStorage(sc, credentialsGilCOS, configurationNameGil, "softlayer_cos")
val url = cosGil.url("gvernik", "test1812/a2/data.parquet")
peopleDF.write.format("parquet").save(url)

The following works:
sc.hadoopConfiguration.set("fs.cos.gilv.access.key","AAA")
sc.hadoopConfiguration.set("fs.cos.gilv.secret.key","AHA")
sc.hadoopConfiguration.set("fs.cos.gilv.endpoint","https://s3-api.us-geo.objectstorage.softlayer.net")
sc.hadoopConfiguration.set("fs.cos.gilv.v2.signer.type","false")

peopleDF.write.format("parquet").save("cos://gvernik.gilv/test1812/a2/data.parquet")
@bassel-zeidan
Copy link
Collaborator

Hi @gilv ,
Strange! The class CloudObjectStorage has been used by a lot of users and it works fine. Here is an example:

https://apsx-dev.stage1.ng.bluemix.net/analytics/notebooks/4709da6c-371e-4b0c-b58e-86b7e5b33aa9/view?access_token=9daf7b648595142e634a4344d2b2b9619dbd83d51bdcf85e9e080efb23354e37

Maybe you are passing wrong configs. Currently i am on vacation but i would like to take a closer look when i am back. Could you please invite me to your project? or give me access to your notebook?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants