For years, I backed up my laptop every night to a cloud provider without problems. But a few days ago, I was unable to restore a file because my account was “under maintenance”. Time to look for a new service. With a backup script that I can understand.
You have a lot of data on your laptop. We all do. And it is fragile. Here is a sad tale of Ph.D. students losing years of research data when the IT staff at their university deleted files instead of backing them up! So, what to do?
You can back up onto an external disk. I lived in the San Francisco Bay Area long enough to realize that the external disk will be burned to a crisp by the fire caused by the Big One. Cloud backup is the only way.
There are any number of vendors for cloud backup. Pay a modest fee, install some software, and backups happen every night. If you need to restore a trashed file, you visit a website and download the file. If, heaven forbid, your entire disk was trashed, you install some software to restore it. Some vendors even arrange for shipping and receiving disks of files for customers with slow networks.
I had such a vendor for many years. And it saved my bacon a number of times. Every few months, I restored a file that I accidentally lost, and twice I restored all files after crashed disks. It was well worth the modest fee.
A few days ago, it happened again. I did something stupid, overwrote a file, and said, oops, gotta restore it from cloud backup. Except...
The account is paid up. Why would it be “under maintenance”? I never found out because my ticket remains unresolved as I write this article. Clearly idrive.com has gone bad.
I tried running their restoration script. It failed with a similar error message.
Now I got pretty nervous because the backup and restoration scripts are a mess of Perl (I kid you not), all redirecting to a proprietary utility program.
Fortunately, I was able to restore the particuiar file from other sources. But a backup vendor who can't deliver a backup at a moment's notice is a non-starter. So I looked for alternatives.
There are any number of vendors for cloud backup. Being burnt with a situation where a vendor denies you access to your own files, what are alternatives?
Users of Microsoft/Apple services have what I imagine to be backup with a trusted vendor (i.e. Microsoft/Apple) that isn't going to flake out on them. If that's good enough for you, use their service and stop reading.
Still here? One good way is for your data to end up on a third party site like Amazon S3. That's what utilities such as rclone, Borg, restic promise. And I am sure they do a fantastic job. Except for me to know that I can restore a file, or heaven forbid, all my files, I have to learn how to operate their particular restoration process.
Before there was cloud backup, I used rsync to back up onto an external disk. It was simple and fast.
rsync -abvz --delete-after --backup-dir=backup-$(date -Idate) "${ROOT_DIR}" /path/to/external/disk
Nothing particular is needed for restoring. The files are on the external disk, with changed or deleted files in the daily backup directories. It is gloriously transparent.
But you can't just rsync to Amazon S3. They don't give you a file system. There are ways to impose a file system on S3 buckets, but that's another thing to learn and potentially get wrong.
I stumbled upon https://rsync.net, which provides something more useful: a file system in the cloud. My one-line backup script from years past works. Except I can take out --backup-dir because they provide ZFS snapshots. Restoring is equally trivial.
I don't normally write about commercial services, but backup is such an essential function for an author that I am making an exception.
I very much hope that rsync.net will give me many years of trouble-free service at an excellent price, but I have learned the hard way that vendors can go bad over time. Now I added a few more lines to my backup script: restoring and verifying ten random files.
# The backup
rsync -av --delete-after "${ROOT_DIR}" ${BACKUP_ACCOUNT}: > "${LOGFILE}"
# Testing restore of ten files
for RANDOMFILE in $(find "${ROOT_DIR}" -type f | shuf -n 10) ; do
echo "Testing restore of ${RANDOMFILE}" >> "${LOGFILE}"
RESTOREDFILE=$(mktemp)
rsync -asvq ${BACKUP_ACCOUNT}:"${RANDOMFILE:1}" "${RESTOREDFILE}"
# -s handles filenames with spaces
# :1 removes initial /
diff "${RANDOMFILE}" "${RESTOREDFILE}" >> "${LOGFILE}"
rm "${RESTOREDFILE}"
done
The remainder of the script emails me the log file. When I wake up, I have a glance at the email. At the end, I see the names of ten random files, so far with no diff message, and marvel how much forgotten junk is in my hard disk.
Comments powered by Talkyard.