backups

[tcurdt]

I am trying to get access to the backups but are still failing. I am trying a GET

base = result['com.apple.mobileme']['com.apple.Dataclass.Account'].url
url = base + "/mbs/" + dsPrsID
https://p05-setup.icloud.com:443/mbs/64249351

But that's giving me even a 503. So that must be wrong.
Using a different host I get at least a 400.

https://p05-mobilebackup.icloud.com/mbs/64249351

[alexhulbert]

Hi! I stumbled upon this issue while searching "-mobilebackup.icloud.com/mbs" in google. I'm creating a full iCloud decryption program. It seems you're using an outdated url. Instead of p05-mobilebackup.icloud.com, try using "p15-mobilebackup.icloud.com" I must warn you though, once you get the response at that url, your going to to have to process it somehow. Its you're dsid (64249351 for you), a space, and then some hex. I need to make sense of this data. I know its kind of odd to ask you a question when its your issue, but since you've gotten this far, would you happen to know how this data is encrypted?

[jurriaan]

I don't think thats an outdated URI, i think p-15 stands for partition 15 (partitioning as in http://en.wikipedia.org/wiki/Partition_(database) )

On 15 nov. 2013, at 15:42, Taconut [email protected] wrote:

Hi! I stumbled upon this issue while searching "-mobilebackup.icloud.com/mbs" in google. I'm creating a full iCloud decryption program. It seems you're using an outdated url. Instead of p05-mobilebackup.icloud.com, try using "p15-mobilebackup.icloud.com" I must warn you though, once you get the info at that url, your going to to have to process it somehow. Its you're dsid (64249351 for you), a space, and then some hex. I need to make sense of this data. I know its kind of odd to ask you a question when its your issue, but since you've gotten this far, would you happen to know how this data is encrypted?

—
Reply to this email directly or view it on GitHub.

[tcurdt]

I also think it's the partition - might suggest the wrong base url though. It should return a list of backup id's AFAIK.

[alexhulbert]

Really? But what do you do with the response? Evidently, your supposed to get a list of backup Ids to use. I Just get random hex. Has anyone tried this recently? I must be doing something wrong, because I tried multiple apple IDs. If you want to review how I'm doing this, you can see my code here

Also, as for your problem, the get_account_settings will show you the correct url for making your request.

EDIT: I've noticed that different articles have different "P" values over time, I know they grow. So it might depend on when the backup was created. Currently, your backup should either have p15 or p16

EDIT 2: Just so you know, I'm using the mobilebackup in the url, and I'll try "setup" pretty soon

[tcurdt]

As for my problem - I am getting the "https://p05-setup.icloud.com:443/mbs/64249351" URL based from the get_account_settings :) ...but for me the request always fails with 503 or 400.
You are getting at least a hex result? What's your status code?

Once we have the list of backup ids we should get the file authentication tokens and with them get the URLs for the file chunks - with a big AFAIU.

[alexhulbert]

Oh, I remember the fix for your problem! I set up a man-in-the-middle attack on "Elcomsoft Phone Password Breaker" by intercepting the traffic over fiddler's proxy. I was then able to see that the headers are the same as the ones used for authorization, except you have to

1. make sure that protocol version is set to "1.7"
2. get a new mmeAuthToken (has to be a different one) from get_account_settings
3. use base64 like you did with get_account_settings for Authorization, but only with the new mmeAuthToken and type "X-MobileMe-AuthToken" before it instead of "Basic"

This is all a little confusing so I'll give you an example:

GET /mbs/DsPrsID HTTP/1.1
User-Agent: MobileBackup/5.1.1 (9B206; iPhone4,1)
Host: p05-mobilebackup.icloud.com
Accept: application/vnd.com.apple.mbs+protobuf
Accept-Language: en-US
X-Apple-Request-UUID: 4EFFF273-5611-479B-A945-04DA0A0F2C3A
X-Apple-MBS-Protocol-Version: 1.7
X-MMe-Client-Info: <iPhone4,1> <iPhone OS;5.1.1;9B206> <com.apple.AppleAccount/1.0 (com.apple.backupd/(null))>
Authorization: X-MobileMe-AuthToken base64(dsPrsID + ":" + mmeAuthToken (the one from get_accout_settings)

[alexhulbert]

The reason I'm doing all this is for a program I'm creating called "iCEW1ND" It's going to be able to backup all of your apps, data, settings, etc. without even booting up your phone (it just needs to enter DFU mode). It will also be able to restore all of that from a manual backup (using this program), from iTunes, or from iCloud.

[tcurdt]

Thanks for the pointer!

Same here. I have lost some SMS that are still stored in an iCloud backup. I could just not believe there is no easier way get them back. So I started digging :)

[alexhulbert]

OH MY GOD! You're profile looked familiar, so I clicked on it. Turns out that I was planning on using your "jDeb" package for backing up custom paths. What a coincidence! Actually, would you be willing to help me with my program? I'm about 1/3 done

[tcurdt]

Funny indeed :) Let me know if you run into issues with it.

[tcurdt]

Meh! Still cannot get it to work. Could you maybe attach a request log with the tokens/ids anonymized? I don't have a man in the middle setup atm.

[tcurdt]

As the header suggests the call should return a protobuffer.

[jurriaan]

I've pushed a working example to the repository
After you clone and bundle install you can run ruby icloud.rb
And when it lands at the pry prompt type c.process RubyiCloud::BackupRequest

The next step is to figure out the protocol buffers ;)

[alexhulbert]

Wow! It worked! What did you do differently? Upon looking at your code, it seems it works just like I'm doing it, yet it outputs
#<RubyiCloud::ProtocolBuffers::Backups psid="dsPrsID" udids=[the backup uuids] todo=false>

[alexhulbert]

I had no clue what protobuffers were, but now that I do, everything is making sense. Onward with my project! I'll let you know if I encounter any more problems along the way (I probably will) and you see my progress at github.com/Triforce1/iCEW1ND. Thanks!

EDIT: Its only been an hour and I'm stuck already :) I've been trying to get the udids from the output of icloud using protostuff. I have no clue how to approach this. Could somebody show me how to do this or at least point me in the right direction? I've got the encoded data as a byte array.

[jurriaan]

You have to reverse engineer the protocol buffer definitions using something like protoc --decode_raw.
Then you write a protocol buffer description and tweak it until it works.
I don't have time to finish the implementation at the moment, but you are welcome to do so ;)

[tcurdt]

Already had a first look with Charles for the man-in-the-middle. It supports protobuffer decoding.

[jurriaan]

I always use mitmproxy. It's open source and I think it also supports protobuffer decoding.
I've just pushed another change, you can now view information about a backup by using

c.process RubyiCloud::BackupRequest, "the backup UUID"

Hopefully it's useful ;)

[alexhulbert]

Sweet! I really appreciate all of this help, I should be done with this in no time.

[alexhulbert]

If you want to know more about how iCloud works, you can look at Vladmir Katalov's presentation at REcon 2013 on "Decrypting iCloud"

[alexhulbert]

I have good news! I've finished all of the Requests and I can now pull out 2 main things from the data: the keys to decrypt iCloud chunks and the file-name-to-chunk mapping data. I just need to retrieve two more things:

1. The chunk authentication tokens
   (p##-mobilebackup.icloud.com/mbs/(dsPrsID)/(backupudid)/(snapshotid)/getFiles)
2. The base url for downloading the files (p##-content.icloud.com/(dsPrsID)/authorizeGet)

These should be accessed using HTTP POST rather than GET, unlike the others. I'm trying to get a response, but its just returning null. I'm not sure what to do from here,

Another issue is decrypting these chunks once the've been downloaded. After looking over everything, the iCloud decryption process very closely resembles that of the iTunes method, where the manifest.mbdb in iCloud matches up to the file list and the keys are stored in a similar way.

Fortunately, there is an open-source iTunes decryptor written in Python which can be found here and here. Adapting this code may be the easiest way to decrypt iCloud backups into an easily-usable form. Do any of you have any idea how I should approach this? Sniffing and Disassembly won't help me much here...

[jurriaan]

Looking at your code i see you don't use the /mbs/$dsid$/$udid$/$id$/listFiles url. I haven't had time to investigate backups further, but don't you need the output of that? (because I just finished the proto description and decoding of listFiles.. Would be a shame if that was all in vain...).

Oh, and btw, I'm not the one behind the jDeb package ;)

[alexhulbert]

Oops forgot to push that...

[alexhulbert]

There, all of the (malfunctioning) code should be here.

Wait... the listFiles is in Proto?! I checked it with protoc and it said it couldn't parse it. If it's a protobuf, than our lives our going to be much easier :) I've been trying to parse hex manually all this time.

EDIT: I actually had the listFiles method before I pushed it. I GET it at line 182 in iCloud.java

[jurriaan]

It took me quite a while to figure it out, I used a hex editor to view the output and after a while I recognised a pattern.
Apple created their own format containing multiple protobuffer streams. The first few bytes are just a Varint (most protobuffer libraries have a function to read a varint from a data stream).
The varint is equal to the length of the next protobuffer data. So the output of listFiles looks like:
[ Varint Data Varint Data Varint Data .. etc ]

To get the data out of it you'll have to do something like this:

until io.eof? do
  length = Varint.decode(io)
  file = decode_file_protobuffer(io.read(length)) # decode the data
end

Edit: see the latest commit for the definitions of the data in the listFiles data. I have yet to figure out the meaning of everything..

Edit2: Info about varints

[alexhulbert]

I figured out the meaning of this file when I was reverse-engineering the program in IDA. After the chunks are downloaded and decrypted, their names are still just random hex strings. The purpose of "listFiles" is to tell you what the chunks' real names are. Its a mapping of real name to chunk name.

[alexhulbert]

This looks fairly easy, but how can the program differentiate between the varint and file? It seems like once I figue out that, I can use com.google.protobuf.CodedInputStream.readRawVarint32 to decode the first one into a length, and so on. Would it be something like this?

fileList = <LIST DATA>;
varintLength = <THE LENGTH OF A VARINT> ; // the actual length in bytes of a varint itself
files = {}; //This is an array that will contain each file as a protobuffer
offset = 0;
index = 0;
do {
    varint = CodedInputStream.readRawVarint32(fileList.bytes[ BETWEEN offset AND offset + varintLength ]);
    //"Varint" variable is the length of the protobuf after the varint; the decoded varint value
    offset += varintLength + 1;
    files[index] = Decode_Protobuf(fileList.bytes[ BETWEEN offset AND offset + varint ];
    offset += varint;
    index = index + 1;
} while (offset < fileList.length)
PARSE EACH ITEM IN files[] AND DO STUFF WITH IT;

[jurriaan]

The thing about variants is that they have a variable length (hence the name variant).
The CodedInputStream.readRawVarint32 method reads the varint and then stops.

You could try something like this:

CodedInputStream codedFileList = new CodedInputStream(fileListIOobject);
length = codedFileList.readRawVarint32();
files[index] = Decode_Protobuf(codedFileList.readRawBytes(length));

[tcurdt]

Wow - good progress here!

@jurriaan the jdeb reference was aimed at me ;)

[alexhulbert]

I got to where you are, @jurriaan. Accually, I got half the variable names the same :). The only problem is that every time I run "codedFileList.readRawVarint32();" it outputs a number (regardless of whether there should be some data there) and it only reads one byte. How am I supposed to know which are real varints and which are just part of the data? Here is what I was able to get
codedFileList.readRawVarint32() = 95, bytes read = 1
codedFileList.readRawVarint32() = 10, bytes read = 2
codedFileList.readRawVarint32() = 20, bytes read = 3
codedFileList.readRawVarint32() = 05, bytes read = 4
codedFileList.readRawVarint32() = 03, bytes read = 5
codedFileList.readRawVarint32() = 5430, bytes read = 7

That last one looked interesting so I tries decoding the next 5430 bytes as "File" in the protobuf... no results.
Right now I'm goint to try experimenting in a hex editor to see if there's a header of some sort that precedes the first varint.

[williamsjj]

This mean you're able to decrypt the files?

[alexhulbert]

Not yet. I think I know how, though.

[startcool]

Sounds awesome! I am always confusing the HTTP post data format so that I have no clue.

@williamsjj Have you got the files successfully with Icew1nd?

[williamsjj]

@startcool I'm doing my own thing in Python. But I successfully got the files using @TriForce1's instructions.

[startcool]

@williamsjj I was puzzled that why I cannot do that using the instructions. I cannot figure out that HTTP post data format is like key = value, the filename(encoded protobuf) is the value by getFiles or authorizeGet , but what's the key?

Can I get your source code to have a test? I have worked for AACS decrypt which may be useful to decrypt files, but I always have trouble in get files.

[williamsjj]

Don't use the instructions. I used @TriForce1's source code as the instructions.

[startcool]

@williamsjj In fact, I 've tested @TriForce1's source code ,but it's failed. Got the "bad request" error by getFiles!Have you done some fix?

[alexhulbert]

I'm working on it :/. I'll look at @jurriaan 's source code for the getFiles stuff.

[startcool]

@TriForce1 You also have failed with my account?In @jurriaan 's ruby code,we should figure out the pack ('H *') operation

[alexhulbert]

I'm in a school computer right now, so I can't test any code. I think "pack" converts a string to bytes. Packing with "H *" converts a hex string (like "1F1F2D") to its string equivalent. It just converts hex to a string, nothing fancy. I think that's what I'm doing in my code, but I'll double check that with some tests on both methods (the python and java ones) when I get home to make sure they produce the same output.

UPDATE: Skimming over my code, I don't believe I have any sort of code that's turning a literal string of hex values into bytes. I might have skipped it over, but I'll definitely look into this ASAP. But why did the code work for me?

[startcool]

That's what my doubts , you and @williamsjj are working well except me. Do you mind email your icloud Applie ID that you have tested successfully to me for a test?

[jurriaan]

So you are now reverse engineering my code? haha.. Thought you were almost done with it @TriForce1.. I don't have time to investigate it, but if you have some questions, feel free to ask!

[alexhulbert]

I'm just focused on that one issue @startcool is having. I've probably got a idiotic bug somewhere. I know it's going to end up to be something like last time :/. I make stupid mistakes in my code a lot. I'm actually half-way-ish through decrypting the files themselves. But I've got the GUI and other parts of the program bogging me down, so that's really bogging me down.

[startcool]

@TriForce1 hi,Do you get the bugs with my account?

[alexhulbert]

Yes I did: Same as you. You were talking about some sort of difference in the "hash" field? Is there a difference in the request for GetFiles between mine and Jurriaan's code? As in do they produce something different?

[startcool]

Yes.Print post data like this : @ body = "\ u0016 \ n \ u0014 \ u0000 \ u000EC \ x9B \ xB3 \ xAC \ u007FA \ xD0E \ u0018 & \ u0014 \ xBF2 \ xF5 \ xDC ^ 0 " in Jurriaan's code, his code as follow:

data = ProtocolBuffers::GetFilesRequest.new(hash: [file.chunkhash].pack('H*')).to_s
::ProtocolBuffers::Varint.encode(body_io, data.length)
body_io << data

the post data is body_io.string.I have checked the file.chunkhash is equal to ByteToHex(filename) in your code by listFile.

As I have doubts, the HTTP protocolpost data format is like key = value, the filename(formated protobuf) is the key by getFiles, but what's the key?

[alexhulbert]

In protobufs, the name/key doesn't matter. It's the value that does. They're distinguished from each other by the end part, like "= 1" or "= 3". I could call a field in my protobuf "chicken" and it would still work. So rather the structure should be "index: value", not "key = value". The name's just there so the programmer knows what a certain field is doing. For example, required bytes udid = 1; would still work if I changed it to required bytes chicken = 1;. It's that = 1 at the end that's important.

[startcool]

I mean the http protocol rather than google protobuff, like the form is submitted in html , the post data format of his post is key = value。For example, when I use Http Analyzer's Request Builder, I need to fill the post data's name and value. Follows：

@jurriaan should know the reason？

[jurriaan]

What are you trying to do? This is for submitting forms. You don't want to send raw protocol buffer data by hand.

[startcool]

We found the post data by getFiles is wrong in @TriForce1 's code,so I use Http Analyzer to debug it by hand.Your code worked well so that you should kown the error in @TriForce1 's code!

[startcool]

@jurriaan the post data's format between the submit form and getFiles What is the difference?

[jurriaan]

Which submit form? Forms use the mime type application/x-www-form-urlencoded, and getFiles uses some apple specific protobuf mime type and encodes the post data accordingly. HTTP post can contain anything, not just key value. Quoting wikipedia: 'As part of a POST request, an arbitrary amount of data of any type can be sent to the server in a request message body.'

[alexhulbert]

Sorry, but what on Earth is going on? I really have no clue what either of you are talking about :/. I've been fiddling around trying to get the decryption to work currently. So far, so good. I'm definitely going to pick up some harder projects once school lets out. These past two months have been pretty chaotic and I'm hoping everything will quiet down pretty soon. I'm really sorry I haven't been doing much lately.

[startcool]

@jurriaan Why don't you add 'authorizeGet ' to your code?I try to do that,but get failed. Maybe there is still something wrong in post data. I am a freshman in ruby so that have no clue to figure it out.

@TriForce1 Did you give up fixing the bug with my login?

[alexhulbert]

Don't worry. I'm not abandoning your bug.

[startcool]

@TriForce1 any news?

[alexhulbert]

You bet! Perfect timing, in fact! Work has been started back up again on the project as of earlier today. I'm working on making it so that everybody can run from the source right off the bat. I've done some testing, and there are a few changes I need to make to get everything running smoothly on all computers. I've got a hefty to-do list with this project, but @PythEch and I are going to plow through the whole thing and get it done. Right now, I'm not focusing on the GUI. Getting everybody to be able to download and decrypt backups is the top priority as of now.

[alexhulbert]

Lol. @startcool I'm getting the same error as you now. I wonder what happened? I'm going to try to pinpoint the problem. Hopefully, there's something I overlooked.

EDIT: It can't be the Authentication header. Since that would result in a 500.

[alexhulbert]

I fixed it! Sorry, it was my fault. I was messing around with newer versions of the iCloud Protocol and forgot to revert the changes. I'll push the working code within a couple minutes. It was a stupid error! Thank god.

[startcool]

@TriForce1 You got it!Well, it is just a small mistake regardless of those troubles caused. Thank you for your help.Then I will go working on backup decryption.

Yes, you're right. It is important to get these backups. In fact, I didn't use GUI.