Drive-Backup backup type doesn't appear to be working

[dyslecix]

Windows 10 Pro
Python 3.11 installed
Installed drive-backup with pipx

First, thank you for the excellent script!

Installation worked no problem. Running a basic backup works. But unless I'm misunderstanding intended behavior, it appears that no matter which type of backup I choose [complete|update|increment], it just overwrites the last file.

Help file says "Default is 'complete'.

So I ran

backup backup -d I:\Google-Drive-Backup

This runs a backup to the correct destination.

But subsequent backups just overwrite the existing backup using the same destination folder, not creating a new one.

So I tried

dbackup backup -d I:\Google-Drive-Backup -t complete

then I tried

dbackup backup -d I:\Google-Drive-Backup -t increment

then I tried

dbackup backup -d I:\Google-Drive-Backup -t complete --source "NPC"

Between backup tests I changed information in the Google drive backups, and made sure it was saved. So my expectation of intended behavior is that it will not touch the existing backup in my destination, but add another with the new information ... and anew file name? I'm guessing.

I can run this manually and I have this automated in a batch file that runs in Windows task Scheduler.

But it just continues to overwrite the existing backup in the destination folder. Is this a bug or am I doing somethng wrong?

[dunkmann00]

Hello @dyslecix! Thanks for trying out Drive Backup and for providing a detailed issue.

So I'm guessing when you have reran the backups they were all today (11/5)? If this is the case, then that is why it keeps overwriting the old backup. If you waited until tomorrow and reran it, it would create a new folder to store the backup in and the backup behavior would be what you are expecting (depending which type you choose).

Having said that, I think this is a bug. The way I was using it, I was running backups weekly, so I never had this problem and just never really thought about the scenario of running multiple backups each day. But it shouldn't be overwriting old backups if you are running a complete or incremental backup.

I think the simplest way to fix this would be for Drive Backup to add the time along with the date to the backup folder that it creates (i.e. Google Drive Backup 11-5-24 10:36:12 or something like that). This way, even if it is ran multiple times on the same day, it will create a new folder to store the backup in each time. Would this behavior be acceptable for how you would expect it to work?

[dyslecix]

Thanks for the quick response dunkmann00.

I'm not sure that I understand your intended solution. How would I get it to append the date and time to the folder? Or are you suggesting that I add it on each run? Is there a way to automate that? Because in the longer help file dbackup backup -h I don't see how I could add the option to ensure that it adds the date and time to each backup folder.

If it is not possible to automate the date & time, then I woudl just go with backing it up once a day.

However, it turns out that what it is doing is not overwriting the file, it is just not adding the new file. This is for backups run on the same day. Once there is a backup for that day, it appears that it won't make another, whether my script options / parameters are the same or different.
So if I run

dbackup backup -d I:\Google-Drive-Backup -t complete --source "NPC"

once in a day, it will create that backup.

But it won't create subsequent backups, despite the script clearly running and saying it completed 100%

Example ... the first run today, for one of the documents, I had one doc with the followinng info added

Then later this morning I changed that to

However, I have run probably a dozen backups since then with default, then different options / parameters, and every single backup shows that same original document showing the "11/5/2024 Backup test Increment 8.11 AM"

So yes, I think it might be a bug. At this point if I get one backup a day, where it actually backs up the cahnges, I'd be happy with that.

Thanks

[dunkmann00]

Sorry if I was confusing, but my intended solution was more of me thinking out loud of how I could improve Drive Backup to better handle if someone runs it multiple times a day. I will open a PR for this, nothing you would have to do other than give it a try when the new version is released.

As for the problem of that spreadsheet you show not updating, I'm unclear what happened. When you say you changed it, do you mean you changed the file on your Google Drive or in the already downloaded backup?

[dyslecix]

As for the problem of that spreadsheet you show not updating, I'm unclear what happened. When you say you changed it, do you mean you changed the file on your Google Drive or in the already downloaded backup?

I meant that I changed the file on Google Drive. I did that several times and ensured the changes were saved in Google Drive. Then I reran the script, multiple times, different options / parameters as mentioned above like

dbackup backup -d I:\Google-Drive-Backup (the default setting is mentioend to be complete)
tried
dbackup backup -d I:\Google-Drive-Backup -t complete
tried
dbackup backup -d I:\Google-Drive-Backup -t increment
tried
dbackup backup -d I:\Google-Drive-Backup -t complete --source "NPC"

Drive backup ran each time, a CMD window opened, and it gave the completed message. But each time when I checked the downloaded backup, it wasn't changed. So what appears to have happened is that it downloaded the first backup on 11-5-2024, named it like Filename Date (example: Google Drive Backup 11-5-2024), and then on subsequent runs it runs but does nothing. As in it doesn't change the contents in the backup location. I ran it again today, and it created a new folder Google Drive Backup 11-6-2024, but it's the same as the contents of Google Drive Backup 11-5-2024. Multiple files have changed on Google drive in the location it is pointing to, yet the downloads (or apparent downloads) are identical to the first one it downloaded. There are 3 files in the Google drive folder it is downloading. Multiple people have used the documents (*.gsheets) and saved them, so the contents on the drive have changed several times over the last 24 hours.

Drive backup Install location (Win 10 PC) is

C:\Users\INTEL9900\pipx\venvs\drive-backup

Log file from the last run

The log is appending the documents with 'Aready downloaded current version'. But it's not the current version in the backup location. It's a version from yesterday morning.

[dunkmann00]

Okay so I setup what I believe is a similar scenario to the one you have to see if I could duplicate this bug. Unfortunately, I cannot. When I run it initially everything downloads correctly. After I make changes to the Google Sheet on my Google Drive and then run the backup again, it downloads the new version correctly.

What Drive Backup looks for to determine if it has the newest version already downloaded is the modified time of the local file and the file on Google Drive. If the local file's modified time is older than the file on Google Drive, it will download the file from Google Drive. Otherwise it knows it has the more recent file so it doesn't need to download anything.

I'm wondering if this could be a time zone issue? From doing some checking around that it looks like Google Drive uses UTC. So that shouldn't be it but I'm not 100% sure given that you are having an issue. Is everyone who has made changes to the file you are having issues with in the same time zone?

[dunkmann00]

Or maybe something else is touching your local files and changing the modified time.

I guess something to look at before you run a backup would be to see if the modified date of the local file matches what you think it should be (i.e. what it was when you download it). And then if it's modified time has been changed, check if it's date is more recent than the last time your Google Sheet shows it was last modified according to Google Drive.

[dyslecix]

All people accessing the files on Google drive are in the same time zone.

Drive Backup ran today, Nov 8th, automatically (by batch file and task manager) at 8 PM PST, my time.

It did not download the latest version of the files.

Before Drive backup ran, the main local file in the destination folder, that is being acessed daily on Google drive, was in a local folder structure on my PC: Google Drive Backup 11-7-2024 > NPC, and the file Properties looked like this:

So that is before Drive Backup ran at 8 PM.

Then, when drive backup ran, it didn't download the latest version of the file. As a matter of fact it's stuck downloading the same file version from changes made Nov 5th. So after Drive backup ran tonight at 8 PM PST, it created a new folder: Google Drive Backup 11-8-2024 > NPC and the main file that was accessed mutlipe times over the past 3 days, has properties that look like this now:

even though it gave me a version of the file from Nov 5th. On Google Drive, that file's properties look like this:

And that correclty identifies the file, as it was accessed today Nov 8th, multiple times, by multiple people.

So the local file has a modified date that is older than that of the file on Google Drive, meaning that intended behavior would be to download the newer file from the drive. But it just gave me the same file with the modifications from Nov 5th.

So in those two folders:
Google Drive Backup 11-7-2024 > NPC
Google Drive Backup 11-8-2024 > NPC

Is exactly the same file, modified Nov 5th, 3 days ago. There have been multiple modifications since then.

[dunkmann00]

Hmm, this is certainly confusing. Although from the screenshots you provided it looks like the local file is from November 7, not November 5 like you say. So I'm not sure why that is like that.

Is it possible you have two copies of the file? Maybe one is not in your NPC folder and that is the one getting modified while the NPC folder one is still unchanged?

If you delete all of your backups and run it again does it then download the new/current version?

Sorry for the hassle you are having. And I appreciate you helping trying to get to the bottom of this. I am not trying to doubt what you are saying by the way, just trying to figure out why it is doing what it is doing.

Thanks.

[dyslecix]

OK, thanks for hanging in with this. Never for a moment crossed my mind that you doubted my results. Your comment about duplication of files changed my troubleshooting approach.

And I believe I've solved it. I'm going to have to test this more with .bat files and Task Scheduler, but I think I've now got it figured.

One of the reasons I went to your drive backup was that I absolutely despise Google drive backup and sync for Windows. It is a convoluted contradictory mess of a program. However I still had it installed, and there was duplication of some files locally, even though they were in separate destinations.

1. So the first step was to uninstall Google drive backup and sync for Windows.

2. I went to your drive-backup local install folder (C:\Users\INTEL9900\pipx\venvs\drive-backup) opended a CMD windows there from Win File Explorer address bar and just ran the default

dbackup backup

It ran a backup to that install folder destination

And guess what? It gave me the most recent version of the files.

Then I modified my .bat file to

@echo off dbackup backup -d I:\Google-Drive-Backup exit

and that gave me the most recent version of the files in destination:
I:\Google-Drive-Backup\Google Drive Backup 11-9-2024\My Drive\NPC

EDIT here: Also, notice in the above path that there is now another folder from Google Drive called My Drive between Google Drive Backup 11-9-2024 and NPC, so removing the specification of --source NPC may also have had Something to do with it??

What I am going to do now is set it as a task in task scheduler for 8 PM every day to run to that destination. I'll let it run as a task for a couple days and check in with it to see if it is giving me the correct files. It notifies me at 8PM of the successful run anyways which, thank you, is a bonus.

I highly suspect that the problems were related to the installation of Google drive backup and sync for Windows. However, if the task scheduler / .bat backups fail and stop giving me the currrent version, then I would just move to having the backups go to the default location at:
C:\Users\INTEL9900\pipx\venvs\drive-backup

But either way, I believe this will be solved. Just give it a few days to test.
Thanks

[dyslecix]

Well unfortunately it didn't work.

The automated backup that ran today (Sun Nov 10)

@echo off dbackup backup -d I:\Google-Drive-Backup exit

gave me a backup sheet from Nov 7th. It missed 3 days of updates.

When I went to the default install location C:\Users\INTEL9900\pipx\venvs\drive-backup
and manually ran

dbackup backup

It ran and completed, but what it gave me was also the backup from Nov 7th.

So now I'm back to having no idea why it is doing this

[dunkmann00]

Ok so based on what you said when it did work the other day, I think that just had to do with the fact you backed up everything in My Drive. Since it was the first time you did that, Drive Backup didn't have a previous version of any of your files as far as it was concerned, so everything needed to be downloaded (it doesn't know the things in NPC were the same as My Drive\NPC).

As for why it doesn't download the correct versions after it has already downloaded something is unclear. However, after having looked through the code to refresh my memory and also testing it myself, the behavior is for it to check the modified times of both the local file and the file on Google Drive and then choose whichever one is newer. So my best guess at this point is that for some reason the modified time of the local file is getting changed and appears to be newer than the file on Google Drive.

It would be nice to know with some certainty if that is in fact what is happening. Mostly just to try to determine is this something that is a common problem or something unique. But I think either way, it does raise the question, "Is Drive Backup's approach to determining which file is the most recent/up-to-date the best approach?" It is simple and fast to check, which is a plus, but it isn't the only thing that can change the modified time on local files. Perhaps there are a not insignificant number of reasons that the modified time could get changed even if the file isn't actually "modified".

[dunkmann00]

I think perhaps, short of you installing Drive Backup via git clone and poking around the code, a good way to try to determine if it does have something to do with the modified time would be to carry out a little test.

1. Run a backup manually like you have done into a new directory just for this test.
2. At this point, check to make sure it downloaded the current/newest version of the file.
3. If it did, record the modified time of the file.
4. In a day or two after you know there have been changes to the sheet, recheck the modified time of the original downloaded file from step 1. It should be the same as when it was first downloaded. If not, then something changed it. If it has changed, record the new time.
5. Manually run the backup again into the test directory. If the time recorded in step 4 was changed, it would not be surprising if the backup does not work correctly. However, if the time is the same, I would expect it to work.

If you do give this a try, please let me know what you find.

Thanks.

[dyslecix]

What I'm going to try now is a workaround.

I'll run drive backup in a batch file from task Scheduler at 6 PM

Then at 6:10 PM Task Scheduler runs a second batch file that copies the contents of the first folder (source) to a second folder and deletes the source files

@echo off robocopy I:\Google-Drive-Backups I:\Google-Drive-Backups2 /E /MOV

The /MOV switch, from Microsoft "Moves files, and deletes them from the source after they're copied." This woudl mean that when drive backup runs a second time (second and consecutive days) the target folder will be empty and there will be no need to check Modified times. Tested this and it worked on tests. Now just let it run from task Scheduler and see if it works at the Task time today. It's not elegant, but it might just be a workaround.

[dyslecix]

Day 1 ... both tasks ran and it was successful. I'll let you know what happens tomorrow