Replies: 18 comments 30 replies
-
|
I don't use onedrive and I also don't know what it does internally. Borg has 2 places where it checks for unchanged / changed files:
Note that borg usually uses ctime and not mtime, because ctime is only controlled by the kernel (and thus more reliable) while mtime is controllable also by userspace applications and some apps are known to "roll back" mtime after they modified files. While the warning you have seen comes from the 2nd place, I guess you should try to make the 1st place work better for your case. If the files cache works as it should, borg will not read the file's contents in a 2nd+ backup if the file did not change. See that FAQ entry (and also the following 2) about how to debug/improve this: https://borgbackup.readthedocs.io/en/stable/faq.html#why-is-backup-slow-for-me |
Beta Was this translation helpful? Give feedback.
-
|
You are absolutely correct: the warning comes from the before and after stat. Based on your hint I could manually reproduce it: ctime changed (see below). But please consider: OneDrive (like others) is a local-cloud sync tool and you can't change the inner working, of course. Given that a lot of built-in disks (spinning, SSD, NVMe, ...) are smaller than some cloud storage spaces, you simply can't fill a local cache with all content from the cloud at the same time. You are bound to the cycle request file >> download file >> cache file locally >> borg process file >> free local cache (delete file; this is automatically and unavoidable done by OneDrive) >> move on to next file Freeing the local cache /deleting the locally cached file triggers a crime modification by OneDrive (again, no chance to prevent this). Of course, this is a simplification, because the file gets deleted sometime after downloading other files, but before the local disks fills up. But it's the base process. Cloud storage spaces are ubiquitous. But they are not a backup! There are many reasons that could lead to a temporarily or permanent loss of your data or access to your data. I really would appreciate to use borg as the missing and freely available backup tool. So I hope that maybe a simple configuration, maybe a command line switch, can solve the describes situation without backing up every file each time borg runs. As you might already have guessed: I am a total newbie to borg :-) Here is a stat comparison. Steps done:
Here's the outcome:
The only thing that changed is the line starting with "Change". |
Beta Was this translation helpful? Give feedback.
-
|
You can add The first backup will be slow, but the 2nd then might work faster. Guess that is a bug in onedrive, I don't see a reason why it changes the ctime if the file did not change. Hmm, maybe it attaches some "onedrive metadata" to the file and that is what changes the ctime? |
Beta Was this translation helpful? Give feedback.
-
|
I would assume OneDrive is working correct in it's own definition and changes some metadata. I will try to test your options on a small folder. |
Beta Was this translation helpful? Give feedback.
-
|
It took me quite a while to get back to this. Today I started a backup of the OneDrive folder, which contains some bigger files - from raw photos with approx. 25 MB up to video files of 1 or 2 GB. The output of the running command still shows "file changed while we backed it up" for some files, although I used a files-cache option, which should be very relaxed:
|
Beta Was this translation helpful? Give feedback.
-
|
"file changed WHILE we backed it up" is the 2nd case (see #8676 (comment)) and has nothing to do with the files cache. |
Beta Was this translation helpful? Give feedback.
-
|
Ah, understood. I further understand, that this won't cause a backup (with an amount of data transferred to the remote backup location) of the file during the next run. From already transferred data - it's the initial backup - I estimate that I have to wait 3 more days for the first backup to finish. |
Beta Was this translation helpful? Give feedback.
-
|
Just try it, I would suspect that the "changed while backupped" files might be backed up again in next backup. Update: actually it SHOULD be backed up again, as the current backup might be inconsistent. The reason why it emits the warning is because if a file changes while we backup it, its backup could be incomplete or not internally consistent. That's the reason why fs snapshots exist for modern filesystems. |
Beta Was this translation helpful? Give feedback.
-
|
Played a little with options, but still it seems that I won't succeed with Borg Backup (which is a pitty). Regardless of the --files-cache option chosen; I still got "file changed while we backed it up" for almost all files. I guess it might stem from the fact, that OneDrive files just get downloaded on demand. My internal drive in the Mac is smaller than OneDrive's size. So all files in the OneDrive folder exist only as placeholders. The file contents get downloaded on accessing the file - I guess the file changes after borg accesses it doing the first stat. It's a pity because there IS a real need to backup OneDrive. Remember: all your access to OneDrive (same for iCloud, Google, ...) depends on the company behind it and there ARE reports of access denial due to "violating" usage terms - some maybe justified, some definitely inadmissible. Inadmissible denial to rightfully access your own data could wreck a life. That's a reason why one should backup. So, question: can you think of measures to make borg work in such an environment (files loaded on demand) without backing up the same file every time? |
Beta Was this translation helpful? Give feedback.
-
|
I did a quick experiment with OneDrive (I just remembered I made a MS account some years ago when I wanted to play with some beta or so): The first 2 results are the same, which is good: just doing a stat does not change the stat results. But the 3rd result is different (after just reading the pdf):
The main difference appears to be that in the first output, the file might be a placeholder or symbolic reference (with no blocks allocated and special flags), while in the second output it's a regular file with actual content (2256 blocks allocated). This is likely showing the difference between a cloud-synced file placeholder and the actual downloaded file after it was accessed. The problem here is that the ctime changed, which tells that the file was changed (this includes file data AND metadata, file data wasn't changed in this case). Every state change (file present locally vs. file only present in the cloud) causes a ctime change. |
Beta Was this translation helpful? Give feedback.
-
|
So, OneDrive changing ctime can be worked around by using the non-default But, as ctime changes if a cloud-only file gets downloaded when reading it, borg will complain that the file changed while it was backed up. It looks like these complaints can be ignored because the file did not really change. |
Beta Was this translation helpful? Give feedback.
-
|
Another issue might be that borg calls Luckily, it looks like just the But, there is not only file data, but also file metadata and I suspect that some fs calls related to reading the xattrs and ACLs might trigger the download of the file. To avoid that, borg has For OneDrive, giving these flags was NOT necessary, but for other cloud storages they might be necessary. |
Beta Was this translation helpful? Give feedback.
-
|
Update: originally, ctime/mtime was mixed up - I now edited and corrected my posts above. Hint: use |
Beta Was this translation helpful? Give feedback.
-
SolutionTested on macOS 15.5, borg 1.4.0, OneDrive Note:
I had only a few files and enough local space. Guess the ctime issue of OneDrive together with not having enough space could lead to 2 runs not being enough to stabilize and even that it never stabilizes, because due to the ctime changes, the file never gets memorized in the files cache. In that case, doing separate borg runs for data sets that still fit on the local disks might solve this issue. |
Beta Was this translation helpful? Give feedback.
-
|
Thanx a lot. I will give it a try. Meanwhile I tried to move the OneDrive folder to a local USB disk with enough space. But OneDrive refuses to be put on removable disks, even if they are APFS formatted as some articles about OneDrive and USB disk state. |
Beta Was this translation helpful? Give feedback.
-
|
It seems your solution will only stabilize if there is enough free space on the OS drive to store the whole cloud storage locally - which might not be the default. Why the OS drive? In a standard macOS setup the user's home folders (~/) are under /Users. With the latest macOS Updates, Apple forced cloud providers, like iCloud Drive, Dropbox, or OneDrive, to put their cloud storage cache under ~/Library/CloudStorage. Although some providers might implement this enforcement slower than others and an old installed Cloud Storage Provider version might still behave different, it might eventually be this way. Moving the cloud storage cache, using symlinks and such, is currently (July 2025) not meant to be user configurable nor supported. Thus said, I am not sure whether the following Microsoft blog post will offer a solution for the OneDrive Cloud Storage Provider: OneDrive for Mac Now Supports Removable Drives Why not cache all cloud storage locally by default? Given that most people (well, this is a, maybe educated, guess) will buy MacBooks or Mac computers with a 256GB or 512GB drive, due to Apple prices, and buying Microsoft Office gives you 1TB alone - not to mention iCloud Drive - and the fact that you'll have plenty of data only locally (covered by Time Machine backups), it seems there won't be enough space to cache the whole cloud storage. Example: I myself have a 256GB OS disk in my Mac mini, 60 GB of it free space, about 600GB in my OneDrive and around 200 GB in my iCloud Drive - and I already have two external drives to store local data. Some data is not relevant for borg backups, like the iPhone Backups in iCloud, and some data is (intentionally) redundant locally, in iCloud and OneDrive (e.g. photos). But long before 1/10 of my OneDrive data gets cached locally during a borg create run the first files will bee freed up again, and the files get never memorized correctly in the files cache. An idea: downloading file contents from the internet into the local cloud storage cache should be the slowest part in a borg create run. Or maybe you, Thomas, will come up with a bright idea?! I really hope that you can extend the wonderful Borg Backup to cloud storage on macOS (yes, alas...a double feature: cloud storage and the cloud storage provider thing on macOS - sorry ;-) - and I confess: plump attempt to influence ;-). Kind regards! |
Beta Was this translation helpful? Give feedback.
-
Did you see that? |
Beta Was this translation helpful? Give feedback.
-
|
The problem with your hash value idea is that one would have to compute the hash of the current file to compare it against the hash value in the cache to determine whether the file is unchanged. But that computation is not fast: it would require (downloading and) reading the complete file. If borg is configured to work without files cache, it would also (download and) read and chunk the full file, compute chunk hashes and check if chunks with these hashes are already in the repo. So, what you suggested with the hash idea is not faster than what borg does without the files cache. |
Beta Was this translation helpful? Give feedback.
-
root cause of the problemThe root cause of the OneDrive issue is that OneDrive is telling (via a ctime change) that "this file has changed" every time the inner state of the file changes (present locally vs. present only in the cloud), although its content and its metadata was not changed by the user. Without that, borg would work as expected and very efficiently. So, guess you need to bring this up at Microsoft and see what they say. |
Beta Was this translation helpful? Give feedback.
-
Yes, but there aren't hardly any small data sets that are small enough or I have to break up things - like each photo folder in a separate run.
I know. What I meant is: downloading and reading the files will be done each and every time anyways (in the current situation, because borg assumes the file has changed and must be backed up), so doing a hash computation would be a not-too-much overhead.
Mmmh, at work I was responsible for a MS Premier Support contract and had a lot of communication with them. It is already very exhausting if there is an error in Microsoft's software, but at least they act. Talking about wishes you would like them to implement, you'll almost any time get: "This is not an option" ;-) |
Beta Was this translation helpful? Give feedback.
-
|
Guess there could be a hack for borg to make OneDrive work even while it has this "ctime bug": In Change that to: @cowegner can you try that? Borg uses But, for simple use cases, mtime might also work. |
Beta Was this translation helpful? Give feedback.
-
|
Correct, you must not point borg to the OneDrive "folder" if it is just a symlink. But you can give /Users/x/OneDrive/something if that is a toplevel folder in your OneDrive. |
Beta Was this translation helpful? Give feedback.
-
|
Here the summary of the 2nd run (both runs with the changed code line). You'll see from "This archive" and "All archives", as well as from the chunks, that the 2nd run doubled the archive size, meaning all files were backed up again. Run time were only slightly less than the first run. The bad compression rate is due to almost only already compressed JPEG files. ` Chunk index: 28200 57849 |
Beta Was this translation helpful? Give feedback.
-
|
The stats output is as expected - you need to look at the "deduplicated size" column of "this archive" to see that the dedup worked (similar for "unique chunks" value). Do a 3rd run with same data and same --files-cache option value and check the debug output for the files cache. If the files cache works as expected, it should be very fast (and shouldn't download files from OneDrive). If not, the log output will tell you what's going wrong. |
Beta Was this translation helpful? Give feedback.
-
|
Should I set special debug options or output to logfile? |
Beta Was this translation helpful? Give feedback.
-
You can use that, but I would remove --progress as it doesn't mix well with the other logging. You can redirect the log output on stderr by appending: |
Beta Was this translation helpful? Give feedback.
-
|
it's working :-) ` This archive: 20.02 GB 19.96 GB 1.82 kB Chunk index: 28229 73716 What does it mean, because I currently use the slightly modified version of archive.py? |
Beta Was this translation helpful? Give feedback.
-
|
Sure. The "bug" in OneDrive only affects the ctime, it does not change the mtime. If borg uses the mtime for "did the file change while it was backed up" detection, it works as it should and is not affected by the OneDrive issue. It's not perfect for general use, as it would not notice a metadata change, only content changes. |
Beta Was this translation helpful? Give feedback.
-
Beta Was this translation helpful? Give feedback.
-
|
Thanx again...a lot. |
Beta Was this translation helpful? Give feedback.
-
|
https://github.com/borgbackup/borg/pull/8959/files this will be in 1.4.2. |
Beta Was this translation helpful? Give feedback.
-
|
BTW: I seem totally lost in formatting these :-( I played a little with OneDrive files, requesting different folders, freeing space up again...just what OneDrive does... I guess it's a thing to note: After OneDrive downloads and frees up files on the local drive, the next borg create run might take a considerable amount of time again, though nothing has changed and no additional data gets into the backup! |
Beta Was this translation helpful? Give feedback.
-
|
Make sure you run the patched borg code and check the debug output for the files cache operations. |
Beta Was this translation helpful? Give feedback.
-
|
Also, make sure to run at least 3 backup runs with the patched code for each folder you look at timing wise. |
Beta Was this translation helpful? Give feedback.
-
|
@cowegner did it improve after some runs using the patched code? did you check the debug output for the files cache operations? |
Beta Was this translation helpful? Give feedback.
-
|
#8959 is a still open PR and corresponds to the code patch I gave above. I will release 2.0.0b19 soon and could merge (and forward port to master) that PR depending on your feedback @cowegner. |
Beta Was this translation helpful? Give feedback.
-
|
Had a few days off, will answer tomorrow. |
Beta Was this translation helpful? Give feedback.
-
|
IMHO it's working "bit-wise". Below are summary lines from the Nth run. Please note, that I've done a couple of runs before i changed the log naming scheme to "borg-Nth run.log". One could see from the deduplicated bytes, that nothing gets backed up, what is already there. I am a little surprised about the time it takes and the fluctuation in duration, though. But this is not related to borg, more to internet connection, OneDrive, etc. ==> borg-3.log <==
Duration: 3 hours 8 minutes 35.64 seconds
Number of files: 16792
Utilization of max. archive size: 0%
------------------------------------------------------------------------------
Original size Compressed size Deduplicated size
This archive: 36.63 GB 36.51 GB 1.56 MB
All archives: 239.79 GB 239.04 GB 36.28 GB
Unique chunks Total chunks
Chunk index: 39671 232015
------------------------------------------------------------------------------
==> borg-4.log <==
Duration: 5 hours 28 minutes 28.35 seconds
Number of files: 16793
Utilization of max. archive size: 0%
------------------------------------------------------------------------------
Original size Compressed size Deduplicated size
This archive: 36.62 GB 36.51 GB 1.48 MB
All archives: 276.41 GB 275.54 GB 36.28 GB
Unique chunks Total chunks
Chunk index: 39721 260910
------------------------------------------------------------------------------Thanx again @ThomasWaldmann |
Beta Was this translation helpful? Give feedback.
-
|
One can see ("deduplicated size") that the deduplication is working. But, it is way too slow, so likely the files cache is not working as expected. Either it is triggering download from onecloud again and again or it is reading/chunking/hashing everything again and again or both. Can you post the files cache debug lines? |
Beta Was this translation helpful? Give feedback.
-
|
Sure. Do you mean lines like these: Reading files cache ...
FILES-CACHE-LOAD: starting...
Verified integrity of /Users/Me/.cache/borg/b7634cd4124a3223bd3b5d4bf84ba58da50e39941fce8383ad22a3db1333850a/files
FILES-CACHE-LOAD: finished, 2 entries loaded.
security: read previous location '/Volumes/NVMe/_borg_/OneDrive-Type.de'
security: read manifest timestamp '2025-07-21T08:07:20.662869'
security: determined newest manifest timestamp as 2025-07-21T08:07:20.662869
security: repository checks ok, allowing access
Processing files ...
UNKNOWN: no file metadata in cache for: b'/Users/Me/Library/CloudStorage/OneDrive-Pers\xc3\xb6nlich/.849C9593-D756-4E56-8D6E-42412F2A707B'
A /Users/Me/Library/CloudStorage/OneDrive-Persönlich/.849C9593-D756-4E56-8D6E-42412F2A707B
/Users/Me/Library/CloudStorage/OneDrive-Persönlich/.849C9593-D756-4E56-8D6E-42412F2A707B: file changed while we backed it up
C /Users/Me/Library/CloudStorage/OneDrive-Persönlich/.849C9593-D756-4E56-8D6E-42412F2A707B
UNKNOWN: no file metadata in cache for: b'/Users/Me/Library/CloudStorage/OneDrive-Pers\xc3\xb6nlich/Me (von Share)/some-file'
A /Users/Me/Library/CloudStorage/OneDrive-Persönlich/Me (von Share)/some-file
/Users/Me/Library/CloudStorage/OneDrive-Persönlich/Me (von Share)/some-file: file changed while we backed it up
C /Users/Me/Library/CloudStorage/OneDrive-Persönlich/Me (von Share)/some-fileThese line are repeating, with some "d" lines for directories in between. |
Beta Was this translation helpful? Give feedback.
-
|
Yes, especially the lines printed directly before the "A", "M" or "C" status lines. Is it always UNKNOWN? Or is it always "C"? We need some more examples here. |
Beta Was this translation helpful? Give feedback.
-
|
Also make sure you still have the patch in place, see #8676 (comment) . |
Beta Was this translation helpful? Give feedback.
-
|
In case we don't get OneDrive behaving, this might be alternatives:
|
Beta Was this translation helpful? Give feedback.
-
|
Yes, the patch is still in place. What's left out in the following is just repeated stuff. This snippet is a complete picture of the run. Command:
using builtin fallback logging configuration
Enabling debug topic borg.debug.files_cache
33 self tests completed in 0.03 seconds
Verified integrity of /Volumes/NVMe/_borg_/OneDrive.de/index.491
TAM-verified manifest
security: read previous location '/Volumes/NVMe/_borg_/OneDrive.de'
security: read manifest timestamp '2025-07-21T08:07:20.662869'
security: determined newest manifest timestamp as 2025-07-21T08:07:20.662869
security: repository checks ok, allowing access
Creating archive at "/Volumes/NVMe/_borg_/OneDrive.de::Backup-20250721153639"
Verified integrity of /Users/Me/.cache/borg/b7634cd4124a3223bd3b5d4bf84ba58da50e39941fce8383ad22a3db1333850a/chunks
Reading files cache ...
FILES-CACHE-LOAD: starting...
Verified integrity of /Users/Me/.cache/borg/b7634cd4124a3223bd3b5d4bf84ba58da50e39941fce8383ad22a3db1333850a/files
FILES-CACHE-LOAD: finished, 2 entries loaded.
security: read previous location '/Volumes/NVMe/_borg_/OneDrive.de'
security: read manifest timestamp '2025-07-21T08:07:20.662869'
security: determined newest manifest timestamp as 2025-07-21T08:07:20.662869
security: repository checks ok, allowing access
Processing files ...
UNKNOWN: no file metadata in cache for: b'/Users/Me/Library/CloudStorage/OneDrive-Pers\xc3\xb6nlich/.849C9593-D756-4E56-8D6E-42412F2A707B'
A /Users/Me/Library/CloudStorage/OneDrive-Persönlich/.849C9593-D756-4E56-8D6E-42412F2A707B
/Users/Me/Library/CloudStorage/OneDrive-Persönlich/.849C9593-D756-4E56-8D6E-42412F2A707B: file changed while we backed it up
C /Users/Me/Library/CloudStorage/OneDrive-Persönlich/.849C9593-D756-4E56-8D6E-42412F2A707B
UNKNOWN: no file metadata in cache for: b'/Users/Me/Library/CloudStorage/OneDrive-Pers\xc3\xb6nlich/Me (von Share)/SomePath/File'
A /Users/Me/Library/CloudStorage/OneDrive-Persönlich/Me (von Share)/SomePath/File
/Users/Me/Library/CloudStorage/OneDrive-Persönlich/Me (von Share)/SomePath/File: file changed while we backed it up
C /Users/Me/Library/CloudStorage/OneDrive-Persönlich/Me (von Share)/SomePath/File
UNKNOWN: no file metadata in cache for: b'/Users/Me/Library/CloudStorage/OneDrive-Pers\xc3\xb6nlich/Me (von Share)/SomePath/File'
A /Users/Me/Library/CloudStorage/OneDrive-Persönlich/Me (von Share)/SomePath/File
/Users/Me/Library/CloudStorage/OneDrive-Persönlich/Me (von Share)/SomePath/File: file changed while we backed it up
C /Users/Me/Library/CloudStorage/OneDrive-Persönlich/Me (von Share)/SomePath/File
d /Users/Me/Library/CloudStorage/OneDrive-Persönlich/Me (von Share)/SomePath/Path
d /Users/Me/Library/CloudStorage/OneDrive-Persönlich/Me (von Share)/SomePath
UNKNOWN: no file metadata in cache for: b'/Users/Me/Library/CloudStorage/OneDrive-Pers\xc3\xb6nlich/Me (von Share)/SomePath/File'
A /Users/Me/Library/CloudStorage/OneDrive-Persönlich/Me (von Share)/SomePath/File
/Users/Me/Library/CloudStorage/OneDrive-Persönlich/Me (von Share)/SomePath/File: file changed while we backed it up
C /Users/Me/Library/CloudStorage/OneDrive-Persönlich/Me (von Share)/SomePath/File
d /Users/Me/Library/CloudStorage/OneDrive-Persönlich/Me (von Share)/SomePath/Path
d /Users/Me/Library/CloudStorage/OneDrive-Persönlich/Me (von Share)/SomePath
UNKNOWN: no file metadata in cache for: b'/Users/Me/Library/CloudStorage/OneDrive-Pers\xc3\xb6nlich/Me (von Share)/SomePath/File'
A /Users/Me/Library/CloudStorage/OneDrive-Persönlich/Me (von Share)/SomePath/File
/Users/Me/Library/CloudStorage/OneDrive-Persönlich/Me (von Share)/SomePath/File: file changed while we backed it up
C /Users/Me/Library/CloudStorage/OneDrive-Persönlich/Me (von Share)/SomePath/File
d /Users/Me/Library/CloudStorage/OneDrive-Persönlich/Me (von Share)/SomePath/Path
d /Users/Me/Library/CloudStorage/OneDrive-Persönlich/Me (von Share)/SomePath
<snip>
...
<snap>
UNKNOWN: no file metadata in cache for: b'/Users/Me/Library/CloudStorage/OneDrive-Pers\xc3\xb6nlich/Me (von Share)/SomePath/File'
A /Users/Me/Library/CloudStorage/OneDrive-Persönlich/Me (von Share)/SomePath/File
/Users/Me/Library/CloudStorage/OneDrive-Persönlich/Me (von Share)/SomePath/File: file changed while we backed it up
C /Users/Me/Library/CloudStorage/OneDrive-Persönlich/Me (von Share)/SomePath/File
UNKNOWN: no file metadata in cache for: b'/Users/Me/Library/CloudStorage/OneDrive-Pers\xc3\xb6nlich/Me (von Share)/SomePath/File'
A /Users/Me/Library/CloudStorage/OneDrive-Persönlich/Me (von Share)/SomePath/File
/Users/Me/Library/CloudStorage/OneDrive-Persönlich/Me (von Share)/SomePath/File: file changed while we backed it up
C /Users/Me/Library/CloudStorage/OneDrive-Persönlich/Me (von Share)/SomePath/File
UNKNOWN: no file metadata in cache for: b'/Users/Me/Library/CloudStorage/OneDrive-Pers\xc3\xb6nlich/Me (von Share)/SomePath/File'
A /Users/Me/Library/CloudStorage/OneDrive-Persönlich/Me (von Share)/SomePath/File
/Users/Me/Library/CloudStorage/OneDrive-Persönlich/Me (von Share)/SomePath/File: file changed while we backed it up
C /Users/Me/Library/CloudStorage/OneDrive-Persönlich/Me (von Share)/SomePath/File
UNKNOWN: no file metadata in cache for: b'/Users/Me/Library/CloudStorage/OneDrive-Pers\xc3\xb6nlich/Me (von Share)/SomePath/File'
A /Users/Me/Library/CloudStorage/OneDrive-Persönlich/Me (von Share)/SomePath/File
/Users/Me/Library/CloudStorage/OneDrive-Persönlich/Me (von Share)/SomePath/File: file changed while we backed it up
C /Users/Me/Library/CloudStorage/OneDrive-Persönlich/Me (von Share)/SomePath/File
d /Users/Me/Library/CloudStorage/OneDrive-Persönlich/Me (von Share)/SomePath/Path
d /Users/Me/Library/CloudStorage/OneDrive-Persönlich/Me (von Share)/SomePath
UNKNOWN: no file metadata in cache for: b'/Users/Me/Library/CloudStorage/OneDrive-Pers\xc3\xb6nlich/Me (von Share)/SomePath/File'
A /Users/Me/Library/CloudStorage/OneDrive-Persönlich/Me (von Share)/SomePath/File
/Users/Me/Library/CloudStorage/OneDrive-Persönlich/Me (von Share)/SomePath/File: file changed while we backed it up
C /Users/Me/Library/CloudStorage/OneDrive-Persönlich/Me (von Share)/SomePath/File
UNKNOWN: no file metadata in cache for: b'/Users/Me/Library/CloudStorage/OneDrive-Pers\xc3\xb6nlich/Me (von Share)/SomePath/File'
A /Users/Me/Library/CloudStorage/OneDrive-Persönlich/Me (von Share)/SomePath/File
/Users/Me/Library/CloudStorage/OneDrive-Persönlich/Me (von Share)/SomePath/File: file changed while we backed it up
C /Users/Me/Library/CloudStorage/OneDrive-Persönlich/Me (von Share)/SomePath/File
d /Users/Me/Library/CloudStorage/OneDrive-Persönlich/Me (von Share)/SomePath/Path
d /Users/Me/Library/CloudStorage/OneDrive-Persönlich/Me (von Share)/SomePath
UNKNOWN: no file metadata in cache for: b'/Users/Me/Library/CloudStorage/OneDrive-Pers\xc3\xb6nlich/Me (von Share)/SomePath/File'
A /Users/Me/Library/CloudStorage/OneDrive-Persönlich/Me (von Share)/SomePath/File
/Users/Me/Library/CloudStorage/OneDrive-Persönlich/Me (von Share)/SomePath/File: file changed while we backed it up
C /Users/Me/Library/CloudStorage/OneDrive-Persönlich/Me (von Share)/SomePath/File
UNKNOWN: no file metadata in cache for: b'/Users/Me/Library/CloudStorage/OneDrive-Pers\xc3\xb6nlich/Me (von Share)/SomePath/File'
A /Users/Me/Library/CloudStorage/OneDrive-Persönlich/Me (von Share)/SomePath/File
/Users/Me/Library/CloudStorage/OneDrive-Persönlich/Me (von Share)/SomePath/File: file changed while we backed it up
C /Users/Me/Library/CloudStorage/OneDrive-Persönlich/Me (von Share)/SomePath
UNKNOWN: no file metadata in cache for: b'/Users/Me/Library/CloudStorage/OneDrive-Pers\xc3\xb6nlich/Me (von Share)/SomePath/File'
A /Users/Me/Library/CloudStorage/OneDrive-Persönlich/Me (von Share)/SomePath/File
/Users/Me/Library/CloudStorage/OneDrive-Persönlich/Me (von Share)/SomePath/File: file changed while we backed it up
C /Users/Me/Library/CloudStorage/OneDrive-Persönlich/Me (von Share)/SomePath/File
d /Users/Me/Library/CloudStorage/OneDrive-Persönlich/Me (von Share)/SomePath
UNKNOWN: no file metadata in cache for: b'/Users/Me/Library/CloudStorage/OneDrive-Pers\xc3\xb6nlich/Icon\r'
A /Users/Me/Library/CloudStorage/OneDrive-Persönlich/Icon
/Users/Me/Library/CloudStorage/OneDrive-Persönlich/Icon
: file changed while we backed it up
C /Users/Me/Library/CloudStorage/OneDrive-Persönlich/Icon
d /Users/Me/Library/CloudStorage/OneDrive-Persönlich
check_free_space: required bytes 555756708, free bytes 1468274491392
Storage quota: 36.28 GB out of 1.00 TB used.
security: saving state for b7634cd4124a3223bd3b5d4bf84ba58da50e39941fce8383ad22a3db1333850a to /Users/Me/.config/borg/security/b7634cd4124a3223bd3b5d4bf84ba58da50e39941fce8383ad22a3db1333850a
security: current location /Volumes/NVMe/_borg_/OneDrive.de
security: key type 3
security: manifest timestamp 2025-07-21T19:05:37.203411
FILES-CACHE-SAVE: starting...
FILES-CACHE-KILL: removed all old entries with age >= TTL [20]
FILES-CACHE-KILL: removed all current entries with newest cmtime 9223372036854775807
FILES-CACHE-SAVE: finished, 2 remaining entries saved.
Verified integrity of /Volumes/NVMe/_borg_/OneDrive.de/index.551
Archive authentication DISABLED.
TAM-verified archive
Archive authentication DISABLED.
TAM-verified archive
Archive authentication DISABLED.
TAM-verified archive
Archive authentication DISABLED.
TAM-verified archive
Archive authentication DISABLED.
TAM-verified archive
Archive authentication DISABLED.
TAM-verified archive
Archive authentication DISABLED.
TAM-verified archive
Archive authentication DISABLED.
TAM-verified archive
Archive authentication DISABLED.
TAM-verified archive
------------------------------------------------------------------------------
Repository: /Volumes/NVMe/_borg_/OneDrive.de
Archive name: Backup-20250721153639
Archive fingerprint: 27eaa2487cc994fd56208d6ea624c913372b655eb7763f2189f70bbddd94f67f
Time (start): Mon, 2025-07-21 15:37:08
Time (end): Mon, 2025-07-21 21:05:37
Duration: 5 hours 28 minutes 28.35 seconds
Number of files: 16793
Utilization of max. archive size: 0%
------------------------------------------------------------------------------
Original size Compressed size Deduplicated size
This archive: 36.62 GB 36.51 GB 1.48 MB
All archives: 276.41 GB 275.54 GB 36.28 GB
Unique chunks Total chunks
Chunk index: 39721 260910
------------------------------------------------------------------------------
rcs: [1] |
Beta Was this translation helpful? Give feedback.
-
|
The problem is the "C" status for all the files it processes. I thought we had determined that when using the mtime (with that patch, instead of the default ctime) for changed-while-backuped it should not determine it as changed? It won't put files into the files cache that have changed while they were backed up and that is why it is so slow. |
Beta Was this translation helpful? Give feedback.
-
|
I did some own experiments with OneDrive again and I think this OneDrive issue has a working workaround in 1.4-maint branch code since recently (a cli option equivalent to the src patch provided earlier). Ignore the file called "latest", it is the file with the most recent timestamp and borg never will cache that file (see FAQ for details). Also the .DS_store should be ignored, it is macOS specific. So, we are just looking at file1, file2, file3 below. |
Beta Was this translation helpful? Give feedback.
-
|
Thanks a lot. I will test when the code is available via homebrew on macOS. |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
First and foremost I hope that I have not overlooked obvious posts related to this question. I tried various keywords here and on Google, but couldn't find an answer.
Using borg to backup files from my OneDrive folder to another cloud storage shows "file changed while we backed it up" for each and every file and files will be fully processed again the next time. So one ends with a full (as in: full gigabytes) backup every time.
Background:
I have OneDrive configured on my macOS. It is syncing my OneDrive folder and doing files-on-demand, as the disk in my Mac is smaller than my OneDrive space.
OneDrive, being a sync not a backup solution, needs proper backup. So I tried borg to backup the Onedrive folder.
First, I was interested, whether borg would backup files-on-demand at all, for almost all of the files are links to OneDrive, rather than physical on my computer. OneDrive has to download them on access to make them available to borg backup. borg triggers the download correctly and then backs up the file.
Second, OneDrive should download the file, currently processed by borg and "delete" it (leave only the OneDrive link to it) soon afterwards to prevent my local disk from filling. OneDrive handles this, so my disk never got full.
Everything got properly backup from OneDrive to the other cloud destination. But the borg log file showed "file changed while we backed it up" for each and every file. Starting a second backup sees each file as changed and will backup it again.
Any idea or recommendation how to solve this?
Beta Was this translation helpful? Give feedback.
All reactions