Talk:Backing Up via Rsync

From WebOS Internals
Revision as of 09:07, 13 October 2009 by Hopspitfire (talk | contribs) (Response.)
Jump to: navigation, search

Rsync Daemon

Why turn on the rsync daemon at all? This could easily be accomplished simply by running

rsync -HrlptgoDPvvS --force --delete --del --stats -e ssh root@IPADDRESS:/ /media/pre-backup/

Also, since -P includes --progress, there's no reason to call it a second time.


We don't have a root password set up on the Pre, it's using sudo therefore that's why ssh won't work. Rsyncd.conf gives the rsync daemon root privileges so it can mirror the device. I'm be open to suggestions, but I'm trying to make this fit in with the next steps guide and limit the number of steps.

Thanks for pointing out the redundancy on --progress.


Ahhh yes. I keep forgetting that not everyone set up keys for root access.


You dont need to be root to use the rsync over ssh to backup, the pervious command will work if you are using the username and password set up in the optware setup.

rsync -HrlptgoDPvvS --force --delete --del --stats -e username@IPADDRESS:/ /media/pre-backup/

I also dont think you need the "ssh" in the command the new versions of rsync use it automatically. I have writen up another tutorial of how to back up the pre without setting up the rsync daemon. check it out here. I have not done a restore yet so I still have to look into that but I see no reason why it cant be done over ssh as long as the partition is r/w and the user has sudo configured.


The problem with running 'rsync -avz' (in your article) is that the files in your backup directory won't get removed when they're not present on the Pre, only overwritten. Have you tested your command above with a normal user account, and did you run into any permission issues? -hopspitfire 16:07, 13 October 2009 (UTC)


I've added these to my .profile on the Pre:

exclude1="--exclude=/proc --exclude=/sys --exclude=/tmp/webdiskcache"
exclude2="--exclude=/media/internal/AUDIO --exclude=/media/internal/Place_iTunes_Uses"
syncopt1="-az --partial --progress --stats --delete-after --bwlimit=300 --numeric-ids"
syncpath=" / me@$IPADDRESS:~/PreRSync/ "
alias Sync='d=`date +%Y%m%d.%H%M%S` ; sudo rsync --rsync-path=sudo\ rsync $syncopt1 -b --backup-dir=~/PreRSync.bak/$d $syncpath $exclude1 $exclude2'

On the server in the /etc/sudoers I've added something like this (not sure this is the most secure thing to do):

me     ALL=(ALL) ALL, NOPASSWD: /usr/bin/rsync

Now I can simply login to the Pre and run Sync, it prompts for a password at the server but hostkeys could be used to avoid that. The addition of the dated backup options means that I get a copy of any removed or overwritten files at the server. I use something called cwrsync which I believe is the cygwin rsync command without installing all of cygwin, might be handy and simple enough for windows users. --Dolio 07:02, 1 October 2009 (UTC)

Nice additions and thanks for sharing, feel free to add them to the article. I'll have to try these myself. -hopspitfire 07:05, 1 October 2009 (UTC)

Pre Rebooting Issue

If you don't exclude some of the directories then the Pre reboots when the backup process touches them. -hmagoo

Can you specify which files/directories prompt the reboot? -hopspitfire 02:57, 20 August 2009 (UTC)
I could narrow it down to proc and/or sys, that's narrow enough for me to exclude both, any progress on restore testing? I'm not running an emulator. -hmagoo
I tested the restore process and it works. Can you run the backup process and send the output to a file (rsync ... > /media/internal/rsync.log) and post it? -hopspitfire 00:49, 24 August 2009 (UTC)
rsync-outputs.tar.gz this was the console and log output from running a backup. I had already a backup in place in the destination but excluded /dev, /sys/ and /proc initially, ran it this time without those exclusions to test it out (again, as I saw this reboot the very first time I tried this method). rebooted right after this line in the console, similar in the log.
rsync: read errors mapping "/sys/devices/platform/lcd-controller/ctrl_reg_dump" (in root): No data available (61)


I ran the backup again excluding /sys and everything completed, only errors I got in the console were:
rsync: send_files failed to open "/proc/sys/kernel/sched_nr_migrate" (in root): Permission denied (13)
rsync: send_files failed to open "/proc/sys/net/ipv4/route/flush" (in root): Permission denied (13)
rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1524) [generator=3.0.5]


Thanks for attaching your log. /sys and /proc don't need to be included in the backup, but /dev does (to initialize devices on boot). After testing the backup and restore, the easiest way is just mkdir the /sys and /proc directories and exlude them in the backup. Does the pre still restart when you excluded those directories?? -hopspitfire 21:23, 26 August 2009 (UTC)

Shouldn't /dev exist fine after a restore and before rsync backup restore? I guess if you made some strange volume changes, but still. --NetWhiz 21:59, 26 August 2009 (UTC)

I was using /dev as an example of a directory that _shouldn't_ be excluded in the backup process, so /dev will exist after a backup prior to a restore (otherwise the Pre won't boot because it can't initialize the mapper devices for storage). -hopspitfire 22:37, 26 August 2009 (UTC)

You mean it SHOULD be excluded b/c you will not need to restore it. --NetWhiz 22:54, 26 August 2009 (UTC)

(failed=reboot).. failed with no excludes. successful with /dev/,/proc,/sys excluded. successful with /sys excluded. but with those three error messages.--Hmagoo 23:52, 26 August 2009 (UTC)

@NetWhiz: /dev needs to be populated on the actual filesystem (before devfs/udev get loaded) for a *nix system to boot.
@Hmagoo: Thanks, I went ahead and fixed the lines in the article.
-hopspitfire 00:12, 27 August 2009 (UTC)

I understand, BUT when you do a restore with WebOS Doctor it will already be there. Are we not talking about the same thing or are we just cross talking? --NetWhiz 01:11, 27 August 2009 (UTC)

OH, I'm talking about a restore using rsync ;) (in this guide, after the intial webOS Doctor). I still don't know if we should be overwriting the entire system (with the rsync backup, including /dev). Any ideas on this? My reasoning for doing a full system overwrite is version compatibility. -hopspitfire 01:16, 27 August 2009 (UTC)
LOL! I would say NOT to overwrite /dev b/c it should already be setup correctly and there is nothing anyone should be doing in there with any mod anyway that I have seen. Trying to overwrite some of the virtual devices can be a BAD thing as some have seen (crashes, freezes, etc.) Just my thoughts. --NetWhiz 01:58, 27 August 2009 (UTC)