Home directories and migration policies on the HSMs

This document describes directory (folder) structures on CESNET hierarchical storage facilities.

A user home directory contains subdirectories that correspond to virtual organizations (VO) the user is a member of, and migration policies. These subdirectories are used to store data. They are symbolic links (shortcut folders) pointing to the directory structure of the VO.

The home directory isn't designed to store your data. Its just for storing SSH keys and other configuration files, and for creating links to the real data. The home directory has soft quotas 50 MiB/500 files ¹⁾ and hard 5 GiB/5000 files ²⁾.

Example for user "user" with membership in two VOs (eg. VO_compass and VO_storage), one of them uses two different migration policies for du4, du5, …:

/home/user:
|-- .bash_history
|-- .screenrc
|-- .ssh
|  \-- authorized_keys
|-- VO_compass-tape_tape -> /exports/tape_tape/backup/VO_compass/home/user
|-- VO_compass-tape_tape-archive -> /exports/tape_tape/archive/VO_compass/home/user
|-- VO_compass-disk_only -> /exports/disk_only/backup/VO_compass/home/user

This structure we will automatically prepare to the user whenever he becomes a member of VO. Users can of course create a different structure using symbolic links. However, the links created by us, we do not recommended to delete.

We are preparing an Accounting system. This system will show occupied space and quota settings. Accounting system is coming soon.

Migration policy determines what happens to your data when you run out of space on the disks, where data that no one else is using will be moved etc. Read the page on HSM (Hierarchical Storage Management) if you need information about the principles of data migration.

Migration policy agreed administrator of virtual organization with administrators of data storage CESNET. Virtual organizations often use only one or a few policies.

Currently available are the following migration policies.

1. tape - data on SATA disks and copy on a single tape (suitable for second or third backups)
2. tape_tape - data on SATA disks and copy on two different tapes (for standard data and first backups)
   1. In Brno and Jihlava data storage are data on the discs, one copy is on MAID and the second copy is on tape (this is suitable for common data and first backup).
   2. In Ostrava data storage are data on the discs (Ostrava data storage does not have MAID discs) and on the tape. If you do not use your data, they are migrated from discs to the tapes. The data are in the end up in two copies on the tapes.
3. disk_only - data is only for disks (RAID), and never moved nor backed up (for fast-changing data, e.g., backups you delete in several days)
   1. In Jihlava data storage (du5) are data on the discs array. On du5 we handle the redundancy by using snapshots, which are kept for several days.

Example:
If the user "user" (member in "storage") wants to use specific migration policy "tape_tape" (data on two different tapes) than he uploads data into the directory "VO_storage-tape_tape" in his home directory.

Below is the list of migration policies and their quotas contained in the virtual organization „storage“ which each registered user is automatically a member.

1. tape_tape-archive - policy is suitable for storing livelier data. It is not recommended to store backups.
   • Soft quota 500 MiB/400 000 files ³⁾ and hard quota 500 MiB/500 000 files ⁴⁾.
2. tape_tape - it is suitable to use this policy primarily for storing backups. Policy called "backup" allows to store data maximally for 12 months. Older data could be deleted, see Terms of Service.
   • Soft quota 2 TiB/90 000 files ⁵⁾ and hard 2,5 TiB/100 000 files ⁶⁾.