Perhaps rclone is slow because of the USB drive on Raspberry Pi
If I don’t use the “–size-only”, I’m guessing it takes forever to verify the files. Is it really the fault of the Ethernet? It’s 1000 Mbps.
Oh and rclone was still running when I woke up, or after I went to the bathroom and crap. I think because I added “-L” to the rclone command. Apparently that slows it to a crawl as well.
Either the Raspberry Pi’s USB sucks, or that drive does. I’ll find out soon, as I’m going to order a NAS drive soon, and put it in a USB 3.0 hard drive dock. It’ll be connected to the Pi of course. I’ll run rclone without the “–size-only” setting, and with “-L” on the Pi itself, to backup the USB drive.
Now I could switch their roles, make the slower drive the backup drive. But it has time on it, so it’s going to die sooner. I’d rather wait until it dies, and get another NAS drive.
I killed rclone and ran it manually, without “-L”.
Or maybe NFS is slow. I could use rsync and SSH.
I added the command to my backup.sh file when I got the drive, probably after finding out it’s super slow without “–size-only”. That’s just a guess, I don’t recall.
Oh and when running rclone, even without “-L”, the load on the Pi is high. Perhaps I should just buy a LinkStation. But would it actually be better? Kind of doubt it. It doesn’t cost enough. $210 for a 6 TB NAS, is kind of cheap. If you look at the enclosures, the cheapest one is around $130. Which probably isn’t better then the Pi either. You still need a drive, around $130 for a 6 TB NAS drive.
I tried searching for high load, and just found some idiot who was using NTFS on the Pi. I wouldn’t recommend NTFS for any Linux computer. I actually had to copy data, and format one drive at a time, I forget how many were NTFS. Probably when I decided to never use Windows again, not in a VM, or bare metal.
Found something on Reddit. Not sure what the CPU % is, just the load. I think six or higher. I’d have to run SSH to get the exact CPU %. And rclone isn’t running right now. They said their load is 5.0+. They might be having the same issue.
Could also be the drive on my desktop, it is a slower drive. SMR or whatever. Or does that only affect writing?
dd if=/dev/zero of=testy bs=1G count=1 1+0 records in 1+0 records out 1073741824 bytes (1.1 GB, 1.0 GiB) copied, 10.9259 s, 98.3 MB/s
Why should I use “async” for NFS? That is with sync. Something I just read, said async can cause corrupted data, if the server crashed. I get over 90 MB/s, so no need to change it to async.
Perhaps I should use SSH for rclone, it’ll probably fly, and I can verify the data correctly.
And it’s already using SFTP. So the Pi is just super slow at verifying files. Too bad you can’t run the verification on the NFS share. Or something. That is run md5 or whatever on the desktop, instead of the Pi. Perhaps there’s a better command to use that is faster.
Ok, I think I can check the files with “–download” instead. Going to give that a try, let’s see how long it takes. Going to comment out the command in my backup.sh file, so if it takes a long ass time, it won’t be running two times. That only works with rclone check. Lame. You have to run it all by itself. Too lazy to mess with it.
Hmm, how should I backup the drive to the new backup drive? And verify the files are good? If it uses md5 or something, it’ll take ages.
So the Pi is too slow to verify files other then checking the size?
I’m guessing I tried using NFS, instead of SFTP, but it was probably too slow, or didn’t work. At least, md5 or whatever, would run on my desktop instead.
Good idea, use the check command, but with the NFS share path, instead of SFTP.
Not sure automating check to run would do any good. You’d need to output it to a text file everyday. Might as well manually run it, just copy the command to a sh file. And run it once a day. Or week, or month. Or when you remember.
Hmm, how will that work with two NFS shares? You can’t run it on the Pi, it’s too fucking slow. Would probably take days, or weeks. That means, I have to share the backup drive on the Pi.
And using it with “–size-only” isn’t a good idea, it’s not syncing changed files. Might if the size doesn’t match maybe. I stopped the check, it was trying to check stuff that is excluded. Running again with that directory excluded. But probably going to stop it again, and see how long it takes to run it without “–size-only”, after I comment out the command in my backup script.
“–checkers 1” perhaps that’ll fix it. If you let it do 8 checks at a time, you can barely if at all access Channels DVR. The Raspberry Pi really needs a better CPU. Using rclone without checking the files, is kind of pointless. That is if files change, but the file size doesn’t.
Checking files still takes forever, even doing one at a time.
There we go, use rsync, not rclone. I should have excluded the .cache folder though. But, I guess I’ll have a 52 GB cache now, it’s for the AUR I think.
Now to setup a rsync daemon, it might be faster then SSH. Except, that isn’t secure. Is NFS secure? And it might be fast enough over SSH.
It can get 50 MB/s using rsync over SSH. The speed over NFS is 98 MB/s. Not with rsync, rsync probably barely works over NFS. Just do the above dd test.
Don’t set “upload only = true”, just set “read only = false”. If you don’t, look in the log, it’ll say it’s read only.
pid file = /var/run/rsyncd.pid lock file = /var/run/rsync.lock log file = /var/log/rsync.log port = 12000 uid = 0 gid = 0 [backup] path = /path/to/backup comment = backup read only = false timeout = 300
If you don’t set uid and gid to 0(root), rsync -a won’t work. I want my permissions. Even though I don’t actually need the permissions for what I’m backing up.
Also, running rsync like that, seems like a horrible idea. Hopefully the firewall works. For IPv6, some stuff has to be open, or IPv6 doesn’t work. Hmm, is there a setting to only listen on the IPv4 address? Just in case. Looks like address = does the trick.
Nice, don’t need a LinkStation now. Don’t use rclone with a Raspberry Pi, it sucks. I can also backup the entire drive, including my outdated laptop backup.
If using Debian, after creating rsyncd.conf, just run systemctl enable –now rsync. Or maybe I should say Raspberry Pi OS, which is based on Debian. Is there any advantage to using a 64bit Raspberry Pi OS? And can you convert an existing 32 bit to it? Cause I’m not reinstalling it. Well, the speed to writing to the USB drive is fast enough, and Channels DVR works fine. So I’ll leave it as is.
Nice, the Pi OS is outdated, it’s based on Buster, one release behind.
Upgrading to Bullseye. Edit /etc/apt/sources.list, replace buster with bullseye. Then run apt update and apt full-upgrade. I looked it up, but I guess I was right, you still have to edit sources.list. Might have said to run apt autoclean or something after updating. You obviously have to reboot.
rsync is going to be sad, when I stop it again, or it stops itself. Will it keep trying, so when the server comes back up, it’ll resume itself?
That article said to run rpi-update as well, I was going to but it said I shouldn’t, unless a Raspberry Pi engineer tells me to. So I typed “n”. Doesn’t matter if it won’t boot, I got a backup, but without the rsync config. That’s easy to redo, I posted the config above.