Backing Up via Rsync

From WebOS Internals

Jump to: navigation, search

Don't try and restore Palm system or database files using this method. The restore itself will appear to work, but will usually stuff up your Palm Profile in the process.

Contents

Synopsis

The rsync remote-update protocol allows rsync to transfer just the differences between two sets of files across the network link, using an efficient checksum-search algorithm.

Some of the additional features of rsync are:

  • support for copying links, devices, owners, groups and permissions
  • exclude and exclude-from options similar to GNU tar
  • a CVS exclude mode for ignoring the same files that CVS would ignore
  • can use any transparent remote shell, including rsh or ssh
  • does not require root privileges
  • pipelining of file transfers to minimize latency costs
  • support for anonymous or authenticated rsync servers (ideal for mirroring)

Background & Purpose

Using rsync as a means to make full backups of the Pre, both for disaster recovery and convenience. I have been using Rsync as a means to backup all my Gentoo and FreeBSD servers, and even a 4TB volume on our NetApp. I was able to restore every system simply by using rsync to push the files to a new disk/array. Using a special string of commands, rsync backups are nearly as depth-oriented as 'dd', while still being much more convenient.

Requirements

  • 1 Unix-based computer (with access to the same WIFI or usbnet subnet as the Pre)
  • On your Palm Pre, you need to have access to linux and have completed the next steps after accessing linux.

The Unix-based computer will be consistently referred to as the "host".

The rsync binary is part of Palm's original package, so it is possible to make a wrap-around gui-based ipk for users who do not want access to linux.

Backup Procedure

Notice

The backup method has been tested, but there are no guarantees explicitly or implicitly given for data replication and retention. These tools have worked exceptionally well for me, but they might not work for you.

Out of respect for Sprint, I will only document how to use rsync over WIFI or usbnet. I strongly disadvise against using rsync over EVDO (even through SSH). This process is bandwidth intensive and we don't want Sprint complaining to Palm.

Quick Overview

1. We will set up the rsync daemon on the Palm Pre and make it acccessible by the host.

2. The host will be set up to download the Pre's files.

3. The transfer will take place.

4. The daemon on the Pre will be killed to prevent anyone else from gaining access.

Setting up the Pre

1. Access your Pre via WIFI or usbnet.

2. Edit /etc/rsyncd.conf and make sure you fill in the host IP address (your computer)

pid file = /var/run/rsyncd.pid
uid = root
gid = root
hosts allow = FILL_YOUR_HOST_IP_HERE
hosts deny = *
use chroot = no

[root]
 path = /


3. Start the rsync daemon and allow the rsync daemon to communicate with the host. Make sure you fill in the host IP address (your computer):

 export IPADDRESS='''enter the IP Address of your host'''
 rsync --daemon && /usr/sbin/iptables -I INPUT 1 -p TCP --dport 873 -s $IPADDRESS -j ACCEPT

Setting up the host

1. Make sure you are the root user.

 sudo -i

or

 su -l

2. On the host, you will need to create a directory where the Palm Pre will back up the files. My example will be /media/pre-backup

 mkdir /media/pre-backup

Running the transfer

On the host

1. If you want to exclude any directories you are already backing up (or simply don't want to include), add --exclude=/path/to/your/dir after "-stats" in the command (in Step 3). You will need to do this for every directory you wish to NOT include in the transfer.

2. 1st time: If this is the first time running the backup, make sure to create the /sys and /proc directories (we're excluding them in the backup, but the directory needs to be there for the filesystems to mount correctly). Replace /media/pre-backup with the backup directory.

mkdir /media/pre-backup/sys /media/pre-backup/proc

3. Fill your IP Address in the command below, along with your excludes (if you have any), and begin the transfer from the host computer. Make sure to set the correct backup directory (if you deviated from /media/pre-backup), otherwise you could overwrite data on your host machine.

 export IPADDRESS='''enter the IP Address of your pre'''
 rsync -HrlptgoDPvvS --force --delete --del --stats --exclude=/sys --exclude=/proc rsync://root@$IPADDRESS/root/ /media/pre-backup/

3a. Every time you run the transfer, it will retain the exact same file-structure of the Pre, only deleting files that don't exist and only sending files that have changed.

Disabling the rsync daemon on the Pre

1. You need to disable the rsync daemon on the Pre to prevent anyone from gaining access to your files. To kill the rsync daemon, you will need to find the Process ID first, then kill the process.

 ps auxw | grep rsync 
 root     11567  0.0  0.0  10480   624 ?        Ss   09:18   0:00 rsync --daemon
 root     11576  0.0  0.0   6124   640 pts/1    S+   09:18   0:00 grep rsync (IGNORE THIS ONE)

Then kill the process:

 kill 11567

Restore Procedure

Quick Overview

1. Set up the host's rsync daemon

2. Gain access to the Pre's Linux.

3. Transfer the backed-up files to the pre and pray it works.

Setting up the host

1. Edit /etc/rsyncd.conf (or /usr/local/etc/rsyncd.conf for FreeBSD). Check your documentation. Be sure to fill in the Pre's IP Address, and set the correct directory for the Pre backup (my example will be /media/pre-backup).

pid file = /var/run/rsyncd.pid
uid = root
gid = root
hosts allow = FILL_YOUR_PALM_PRE_IP_HERE
hosts deny = *
use chroot = no

[pre-backup]
 path = /media/pre-backup/


2. Start the rsync daemon (you can use an init script, I'm using the command below just for simplicity. Check your documentation)

 rsync --daemon

Setting up the Pre and restoring

A. For the scope of this guide, the requirements above still apply for the Palm Pre: you need to have access to linux.

1. Access the Pre via WIFI or usbnet.

2. Turn off the services while we restore all the files.

 /sbin/initctl list | grep running | grep -v novacomd | grep -v pmklogd | grep -v pmsyslogd | awk '{ print $2 }' | while read serviceName ; do /sbin/initctl stop $serviceName ; done

Note from user: killing all services except the ones listed above will kill your wifi connection. I was unable to attempt a restore without leaving all the services running, as I couldn't figure out the magic combination of supporting services that needed to be grep -v'd in order to allow the wifi network to keep running.

3. Fill your IP Address in the command below, and start the transfer. This command will take everything from your Host system and overwrite EVERYTHING on the Pre. Exclude the cryptofs as this will be restored via its target "/media/internal/.palm", the other excluded directories should not or cannot be overwritten.

 export IPADDRESS='''enter the IP Address of your host'''
 cd /
 mount -o remount,rw /
 mount -o remount,rw /boot
 mount -o remount,rw /media/internal
 rsync -HrlptgoDvvS --force --delete --del --stats --exclude=/dev --exclude=/proc --exclude=/sys --exclude=/media/internal/.palm rsync://root@$IPADDRESS/pre-backup/ /

Note: you should restore /media/cryptofs and NOT /media/internal/.palm, because encrypted filesystem uses a device-dependent encryption key

Note from user: This process would work up to a point and then silently fail. I think it was the novaterm connection that would bail on me. I'm wondering if it would actually restore successfully if the rsync command would be proceeded by a `nohup`.

3a. If you have any "Out of Space" issues, try removing the --delete --del and replacing it with --delete-before

3b. It might also help to use a temporary directory for the copy with --temp-dir=/media/internal/tmp, before settings this option, create the directory mkdir /media/internal/tmp.

4. Reboot immediately and pray it works.

 reboot

Development/Improvement Ideas

  • I need more people to test the restore process and get back to me with suggestions.
    • The process does not really work if you restore to a different device: the PalmDatabase.db3 and PalmAccountDatabase.db3 contain device dependent informations, that will cause the "You are no longer signed to your Palm Account" message, reboot and data wipe.
    • It's almost pointless to completely backup the device anyway, as only /var and /media/internal really contain user data.
  • It would be awesome if the Palm Pre supported NFS, otherwise we're stuck to setting up /etc/rsyncd.conf on the Pre (for backup) and the Host (for restore).
  • A gui wrapper is possible because the Palm Pre comes shipped with rsync. Also, there needs to be an easy Windows server program to make this mainstream, only thing to use at the moment that's Windows based is cygwin. Otherwise we're stuck with Unix.
  • It's possible to create a cron job to run the backup every evening over WIFI (especially if you dock you're phone overnight). The cron job would be run from the Host computer. It's also recommended to set up iptables for allowing port 873 only for the host on the interface eth0 or usb0.
  • Would be nice to be able to have it run at a low priority and not kill any ssh connections. Have tried renicing the rsync daemons but has had no affect.

Contributor(s)

  • hopspitfire
  • NetWhiz
  • klktrk
Personal tools
Google AdSense