en:navody:rsync:start

Rsync

For common user

  • This variant is suitable for storing the files of one user, assuming that there is no need to keep information about the files owner
  • It is possible access to data storage from different machines - for example, back up one system but do recover files on multiple machines (with different bond between user account and user identifiers uid/gid).
IPv4 addresses of access servers (hosts) for each data center are:

Ostrava ssh.du4.cesnet.cz
Jihlava ssh.du5.cesnet.cz

IPv6 addresses of access servers (hosts) for each data center are:

Ostrava ssh6.du4.cesnet.cz

Services using IPv6 addresses are currently in testing mode.

Command line client for rsync (Linux, Windows/cygwin)

In a GNU/Linux is rsync installation available directly from the package manager (in most distributions).

We don't recommend use the option -c (–checksum) for offline data (already migrated data to tape). If you use it, data weill be transfer back to disk at every file for comparison. This would result in a long delay.
In the case of a larger number of offline files we recommend to use the option -W.

Example of use rsync for backup directory tree folder to account user server host into directory path (rsync format):

rsync -av folder user@host:path

Example of restore directory tree folder from account user server host into working directory.

rsync -av user@host:path/folder .

GUI rsync client Grsync (Windows, Linux, OS X)

The guide is written for Grsync v.0.6.3 running on OS Windows XP, Grsync v.0.9.2 running on OS Linux. Other versions of Grsync or/and running on another operating system may act or look a little bit different.

Graphical interface of Grsync in Windows:
Grsync ve Windows

Graphical interface in GNU/Linux / Xorg:
Grsync v Linuxu

Description of graphical interface

  1. … you can define server setting here and save it
  2. … source location (rsync format, i.e. user@host:path)
  3. … destination location (rsync format)
  4. … caution: prompt for server password or SSH keys may appears in the command window from which the program was run
  5. … if you move mouse over the checkbox then it displays information about appropriate parameter for line rsync.

For administrators

  • This option of use the service rsync is suitable for storing synchronized backup file system with data ownership and access rights, where such data is necessary for files to keep.
  • E.g. back up file server which is used to by multiple users and when restore of files from a backup happens then have to be set ownership and permissions same as the original files.
Term of use for this variant:
  • during the use of service over certain directory tree should be still calling rsync with te same parameters setting --numeric-ids and --fake-super
  • set of users in the source system have to be fixed to assign numerical identifiers accounts (uid/gid).
  • to data storage should be accessed only from one source system or multiple systems where they have the same numeric identifiers accounts (uid/gid).
  • we don't recommend usage GUI client grsync in Windows in this variant - in tests were appeared incorrect mapping users uid/gid.

If you break these conditions it will very likely leads to inconsistent data about files ownership.

Meaning of the parameters
We don't recommend using the option -c (–checksum) for offline data (already migrated data to tape). Its use would result in a transfer of data back to disk every file for comparison. This would result in a long delay.
In the case of a larger number of offline files we recommend to use the option -W.

--numeric-ids

  • when you are using this parameter the server process doesn't apply for mapping uid/gid values ​​of the user/group name but it will use directly numerical values ​​(uid/gid) of the source system.
  • therefore the values uid/gid mustn't be changed and the parameter must be constantly use in one directory tree or not use at all.

--rsync-path

  • this parameter allows you to pass parameters to the server process - this option is used to pass parameters --fake-super:
    --rsync-path="rsync --fake-super"

--fake-super

  • this parameter enables server-side emulation of superuser mode and then rsync stores the data about ownership and access rights to the extended attributes (xattrs)

--super

  • this parameter forces root behaviour - changes in ownership and rights to files. In the tests were needed only for restore the files to the client with Windows OS. Caution: when you are sending files on non-root accountthen you can't use this parameter (right choice int this case is --fake-super in --rsync-path).

Command line rsync client (Linux, Windows/cygwin)

Linux

Example of use rsync for backup directory tree folder to account user server host into directory path:

rsync -av --numeric-ids --rsync-path="rsync --fake-super" folder user@host:path

Example of restore directory tree folder from account user server host into working directory.

rsync -av --numeric-ids --rsync-path="rsync --fake-super" user@host:path/folder .
Windows/Cygwin

Example of back up and restore directory of user Adminstrator in Windows:

rsync -av --numeric-ids --rsync-path="rsync --fake-super" /cygdrive/c/Documents\ and\ Settings/Administrator user@host:path

I you are doing restore in Windows you may need additional parameter --super for proper recovery of information about files ownership and access rights:

rsync -av --numeric-ids --super --rsync-path="rsync --fake-super" user@host:path/ /cygdrive/c/Documents\ and\ Settings/restore

Notation of source and destination

Notation for remote location

user@server:path

where:

  • user … username of the account which is used to access data storage.
  • server … domain name of data storage server.
  • path … relative path (relate to the user's home directory) in the file system data storage.
IPv4 addresses of access servers (hosts) for each data center are:

Ostrava ssh.du4.cesnet.cz
Jihlava ssh.du5.cesnet.cz

IPv6 addresses of access servers (hosts) for each data center are:

Ostrava ssh6.du4.cesnet.cz

Services using IPv6 addresses are currently in testing mode.

Example:

user@ssh.du4.cesnet.cz:VO_storage-tape_tape/zalohy/1205-kveten
Character "/" (respectively "\" in Windows outside Cygwin) in the end of definition source location has in rsync special purpose:
  • … if you use the character at the end, than it works with the whole directory (folder), i.e. directory will appear directly in the target area.
  • … if you don't use the character at the end, than it works only with the contents of a directory (folder), i.e. in the target area will appear only objects contained in the directory, not the directory itself.

File integrity check

To check the integrity of your files after transfer you can use Hasher app, which is described in the following guide.

References

Website of the project rsync: http://rsync.samba.org/
Website of the project Grsync: http://www.opbyte.it/grsync/
Website of the project Grsync for Windows: http://grsync-win.sourceforge.net/
Website of the project Grsync for MacOS X: http://grsync-mac.tuxfamily.org/?lang=en

NAS Synology

The Network-attached storage (NAS) from Synology is implicitly communicating via Rsync process which is not running on the server side due to security reasons. It is possible to do backups from NAS Synology via Rsync by switching the interface from GUI to CLI and using standard SSH / rsync via script which will be called by cron. Please note that the shell in NAS Synology is not bash, sh, tcsh, csh … but ash.

   
#!/bin/ash

LOG_FILE=/var/log/backup_`date '+%F'`.log
LOG_FILE_ERR=/var/log/backup_errors_`date '+%F'`.log

echo "`date '+%b %d %T'` ==== Start of transmission ====" >> $LOG_FILE

rsync -a --exclude="[#]recycle/" /volume1/backup service_account@ssh.duX.cesnet.cz:~/VO_nazev-tape_tape/ 1>> $LOG_FILE 2>> $LOG_FILE_ERR

RETURN_CODE=`echo $?`
echo "`date '+%b %d %T'`==== End of transmission ====" >> $LOG_FILE

# Back up logs as well
rsync -a /var/log/backup* service_account@ssh.duX.cesnet.cz:~/VO_nazev-tape_tape/

Last modified:: 11.11.2019 22:34