backend: Add Apple iCloud Drive backend#7717
Conversation
|
Nice work :-) Some quick questions
Thank you |
not sure what client_id you are referring to? and no I havent run any integration test yet. |
|
fyi: so iCloud doesn't have functionality for updating/replacing files. At least not something I can find reverse engineering So if a file exists I move it to iCloud trash. And upload a new version of the file. |
|
@ncw how one handle folder invalidation? drive uses etags for files and folders. If a file or folder within a folder changes then the parent folders etag changes. This means I have to clear almost the whole tree from child to parent. Not sure how to handle that in general, what if a users change it locally but file/folder has changed remotely in between. What error should I raise on that? (if etags are not matching) cachedir doesn't seem to refresh after invalidation, see: rclone/backend/icloud/icloud.go Line 424 in 37816e9 When calling flushdir at the end of rmdir I see the call for FindLeaf but the cachedir entries aren't updated. |
This one at the top of the source! clientId = "e9f98057fb916de2bbd755ef280d7257146a76e5118f27ab2e9a3d065c20c17e"
Running the integration tests will guide you through what you need to do. You can start running them as soon as you've done
Seems OK!
Do you have to use the etags? They are optional for most cloud storage systems and rclone doesn't use them. This means that the last write wins which is what we want. If you can't ignore etags then refresh the etag and try again.
Not sure! |
etags are needed for file operations unfortunately, but I can just refresh it when it got stale. the clientid is a static that icloud.com uses. So I reuse it to mimic the browser. |
|
@ncw small question. iCloud Drive does not allow uploading empty files so check the size and raise ErrorCantUploadEmptyFiles when its 0 But when using macOS finder to copy a file, I get an partial file with zero length (while the actually file isn't empty) Not sure howto handle this. |
That sounds correct.
Are you using So you are seeing a zero length file in the finder? I'm not sure what you mean by a partial file? Can you reproduce this without the |
|
Its slow but finished alle interfaces and all integration test are passing 🎉 Only documentation left before its ready for review. |
Well done! You are running the full suite from
:-) |
|
@lostb1t not sure if it's too early for bug reports, but currently it looks like app folders aren't being represented as folders, but rather as files. |
bug reports are welcome. Will fix with the following pass, tnx |
|
@epetousis this has been fixed |
|
crap i broke my icloud. Too many trash items and now the endpoint is broken lol the icloud web api is a mess |
lostb1t
changed the title
|
Fix works great. Much appreciated! |
|
Hi, thanks. |
@cyayon Could you post a log from the command with the -vv flag (e.g. output from rclone -vv ls |
|
Hi, here is the verbose. |
|
do you maybe have advanced data protection enabled? |
|
Yes I have. |
|
yes its currently unsupported |
|
Ok, I understand. |
|
i hope to add support for it eventually.Its a different flow, need to dive in a bit to see what it entails |
|
Thanks !--Christophe YayonOn 20 Apr 2024, at 12:56, lostb1t ***@***.***> wrote:
i hope to add support for it eventually.Its a different flow, need to dive in a bit to see what it entails
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>
|
|
I just disabled the advanced protection and it do not change everything. No file or directory found ... |
try to delete the remote and add it again (reauthenticate) |
|
Great ! it works. |
|
Works for me on I had an issue with 2fa, when prompted none of my devices would show the code on screen. However I managed to get it working by going on my iPhone to I also struggled when looking for iCloud in the list of rclone remotes as it has not be inserted in the right alphabetical spot. Great work @lostb1t, cannot wait to see this landed on main. If you need some help to finish this work or just for some testing let me know. |
|
You gotta be kidding me. |
|
No - why should I 🤔 |
|
Wasn’t directed to you but Apple :) Seems they finally dropped their legacy auth after like 20 years. Yeah this requires a rewrite. i expect more reports will come in soon. |
😆 Let me know if I can help you with testing. |
|
Hi, |
|
This is such a shame... I was so happy to see this happening! ~ rclone --version
rclone v1.69.0-beta.8384.b9207e572
- os/version: linuxmint 22 (64 bit)
- os/kernel: 6.8.0-48-generic (x86_64)
- os/type: linux
- os/arch: amd64
- go/version: go1.23.2
- go/linking: static
- go/tags: none2024/11/04 16:13:14 DEBUG : iCloud Drive: config in: state="*all-set,1,false", result="AcQzsfBDzcjufalaXYjY2lc8L2e8YKNkg-lYJ5VgSMI2WoQ"
2024/11/04 16:13:14 DEBUG : iCloud Drive: config out: out=&{State:*all,2,false Option:<nil> OAuth:<nil> Error: Result:}, err=<nil>
2024/11/04 16:13:14 DEBUG : iCloud Drive: config in: state="*all,2,false", result=""
2024/11/04 16:13:14 DEBUG : iCloud Drive: config out: out=&{State:*all,3,false Option:<nil> OAuth:<nil> Error: Result:}, err=<nil>
2024/11/04 16:13:14 DEBUG : iCloud Drive: config in: state="*all,3,false", result=""
2024/11/04 16:13:14 DEBUG : iCloud Drive: config out: out=&{State:*all,4,false Option:<nil> OAuth:<nil> Error: Result:}, err=<nil>
2024/11/04 16:13:14 DEBUG : iCloud Drive: config in: state="*all,4,false", result=""
2024/11/04 16:13:14 DEBUG : iCloud Drive: config out: out=&{State:*all,5,false Option:<nil> OAuth:<nil> Error: Result:}, err=<nil>
2024/11/04 16:13:14 DEBUG : iCloud Drive: config in: state="*all,5,false", result=""
2024/11/04 16:13:14 DEBUG : iCloud Drive: config out: out=&{State:*all,6,false Option:<nil> OAuth:<nil> Error: Result:}, err=<nil>
2024/11/04 16:13:14 DEBUG : iCloud Drive: config in: state="*all,6,false", result=""
2024/11/04 16:13:14 DEBUG : iCloud Drive: config out: out=&{State:*all,7,false Option:<nil> OAuth:<nil> Error: Result:}, err=<nil>
2024/11/04 16:13:14 DEBUG : iCloud Drive: config in: state="*all,7,false", result=""
2024/11/04 16:13:14 DEBUG : iCloud Drive: config out: out=&{State:*all-advanced Option:false OAuth:<nil> Error: Result:}, err=<nil>
2024/11/04 16:13:14 DEBUG : iCloud Drive: config: reading config parameter "config_fs_advanced"
Edit advanced config?
y) Yes
n) No (default)
y/n> n
2024/11/04 16:13:16 DEBUG : iCloud Drive: config in: state="*all-advanced", result="false"
2024/11/04 16:13:16 DEBUG : iCloud Drive: config out: out=&{State:*postconfig Option:<nil> OAuth:<nil> Error: Result:}, err=<nil>
2024/11/04 16:13:16 DEBUG : iCloud Drive: config in: state="*postconfig", result=""
2024/11/04 16:13:16 DEBUG : iCloud Drive: config in: state="", result=""
2024/11/04 16:13:16 DEBUG : icloud: Authenticating as ***********
2024/11/04 16:13:17 DEBUG : iCloud Drive: config out: out=<nil>, err=HTTP error 400 (400 Bad Request) returned body: "{\"success\":false,\"error\":\"Invalid Session Token\"}"
2024/11/04 16:13:17 DEBUG : iCloud Drive: config out: out=<nil>, err=HTTP error 400 (400 Bad Request) returned body: "{\"success\":false,\"error\":\"Invalid Session Token\"}"
Error: HTTP error 400 (400 Bad Request) returned body: "{\"success\":false,\"error\":\"Invalid Session Token\"}"Advanced data protection is disabled and I'm using a brand new app password. I wouldn't know where to start to troubleshoot this issue, but I'm more than happy to help with testing or anything else needed to solve this last challenge! |
|
Hi, CRITICAL: Fatal error: HTTP error 400 (400 Bad Request) returned body: "{"success":false,"error":"Invalid Session Token"}" |
Yes same for me |
|
@LostB1 this is just about to be released into v1.69 However it sounds like it is broken :-( Shall I
Any other ideas? |
|
@ncw if possible i would leave it out. The change needed isn't a big one but im currently in the middle of moving houses. So currently not able to make the required changes. |
|
Hi,Since a few days I was able to refresh token. It seems Apple restored the legacy authentication.--Christophe YayonOn 14 Nov 2024, at 17:42, lostb1t ***@***.***> wrote:
@ncw if possible would leave it out.
The change needed isn't a bug one but im currently in the middle of moving houses. So currently not able to make the required changes.
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>
|
|
Hi, it works for me too! My bad, I was using the app password instead of regular password + 2FA. It worked at first try! |
|
Apple has done something and restored the legacy method.--Christophe YayonOn 14 Nov 2024, at 20:14, vintagefuture ***@***.***> wrote:
Hi,
it works for me too!
My bad, I was using the app password instead of regular password + 2FA. It worked at first try!
—Reply to this email directly, view it on GitHub, or unsubscribe.You are receiving this because you were mentioned.Message ID: ***@***.***>
|
|
I can confirm that logging in with the Apple ID and 2FA works. Logging in with app specific password does not. I think it would make sense to add a clear note in the config process. Thanks for implemetioning this feature @lostb1t! |
|
thats interesting. No way to be sure for how long/if it was just a fluke, but if it works then the release can proceed in my opinion. |
|
Ah good. @lostb1t can you send a PR explaining that in the docs please? |
|
Thank you @vintagefuture - I've merged that. |
|
Any chance to add an option to receive 2fa via sms? icloud-photos-downloader/icloud_photos_downloader#803 I don't wanna use my main apple id authenticated on backup systems so i've tried to use secondary apple id and it have no authorized systems to send 2fa except sms. |
Not planning on adding it myself. |
|
Hi, I got these when using a Apple ID (cn) Any progress? |
china right? not supported right now (china uses a different server) could you create an issue for this? will add it soon |
|
Thanks for the feature! When are we going to be able to use it? I wanted to use it in my fresh Fedora 41 install and the latest rclone doesn't include it yet. Thanks! |
You can grab the binary or package from beta.rclone.org. Testing on my MacBook and openSUSE server, works fine. |
|
Thanks @egigoka ! I was able to install and configure it perfectly. The only thing I'm facing is errors like this when I open Obsidian:
I used the |
I don't mount it directly, just use as backend for duplicati, so I don't know how to fix mounting problems. |
|
EDIT: OG: Maybe they got rid of legacy auth again or something. Thanks so much for this! |
20 participants
What is the purpose of this change?
Add support for a new backend, iCloud Drive
Was the change discussed in an issue or in the forum before?
#1778
Checklist
Notes
This feature branch is in a working state, feedback on finished implementations is appreciated. Ill keep it in draft untill everything is implemented and ill update the TODO accordingly
This PR is only for iCloud Drive. Ill continue work on Photos once this PR is merged.
Todo
Rclone Interfaces
Fs
Object
Mover
DirMove
Copier