en:navody:home-migrace-plzen:start

Home directories and migration policies on the HSMs

This document describes directory (folder) structures on CESNET hierarchical storage facilities.

Home directory

A user home directory contains subdirectories that correspond to virtual organizations (VO) the user is a member of, and migration policies. These subdirectories are used to store data. They are symbolic links (shortcut folders) pointing to the directory structure of the VO.

The home directory isn't designed to store your data. Its just for storing SSH keys and other configuration files, and for creating links to the real data. The home directory has soft quotas 50 MiB/500 files 1) and hard 5 GiB/5000 files 2).

Example for user “user” with membership in two VOs (eg. VO_compass and VO_storage), one of them uses two different migration policies for du2, du3:

/home/user:
|-- .bash_history
|-- .screenrc
|-- .ssh
|  \-- authorized_keys
|-- VO_compass-tape_tape -> /exports/tape_tape/VO_compass/home/user
|-- VO_compass-disk_only -> /exports/disk_only/VO_compass/home/user
\-- VO_storage-cache_tape -> /exports/cache_tape/VO_storage/home/user

Example for user “user” with membership in two VOs (eg. VO_compass and VO_storage), one of them uses two different migration policies for du4, du5, …:

/home/user:
|-- .bash_history
|-- .screenrc
|-- .ssh
|  \-- authorized_keys
|-- VO_compass-tape_tape -> /exports/tape_tape/backup/VO_compass/home/user
|-- VO_compass-tape_tape-archive -> /exports/tape_tape/archive/VO_compass/home/user
|-- VO_compass-disk_only -> /exports/disk_only/backup/VO_compass/home/user
Please pay attention, that only some of the symlink names in case of du4 contain the word “archive”, e.g. VO_compass-tape_tape-archive. That is related to the change in the Terms of Service. All data on new Data Storage facilities, starting from du4, are treated as a backups and archives - data with permanent value. Backups can be stored only for particular period specified in the Terms of Service. All directories without designation “archive” are considered as backup.

This structure we will automatically prepare to the user whenever he becomes a member of VO. Users can of course create a different structure using symbolic links. However, the links created by us, we do not recommended to delete.

How do I get information about my quota and occupied space?

We are preparing an Accounting system. This system will show occupied space and quota settings. Accounting system is coming soon.

Migration policies

Migration policy determines what happens to your data when you run out of space on the disks, where data that no one else is using will be moved etc. Read the page on HSM (Hierarchical Storage Management) if you need information about the principles of data migration.

Migration policy agreed administrator of virtual organization with administrators of data storage CESNET. Virtual organizations often use only one or a few policies.

Currently available are the following migration policies.

  1. tape - data on SATA disks and copy on a single tape (suitable for second or third backups)
  2. tape_tape - data on SATA disks and copy on two different tapes (for standard data and first backups)
  3. disk_only - data is only for disks (RAID), and never moved nor backed up (for fast-changing data, e.g., backups you delete in several days)
Minimal size o migrable file is 64 kiB, i.e. files smaller than 64 kiB are not migrated to the tapes and remain on a disc tier.

Example:
If the user “user” (member in “storage”) wants to use specific migration policy “tape_tape” (data on two different tapes) than he uploads data into the directory “VO_storage-tape_tape” in his home directory.

Virtual organization "storage"

Below is the list of migration policies and their quotas contained in the virtual organization „storage“ which each registered user is automatically a member.

  1. tape_tape-archive - policy is suitable for storing livelier data. It is not recommended to store backups.
    • Soft quota 500 MiB/400 000 files 3) and hard quota 500 MiB/500 000 files 4).
  2. tape_tape-backup - it is suitable to use this policy primarily for storing backups. Policy called “backup” allows to store data maximally for 12 months. Older data could be deleted, see Terms of Service.
    • Soft quota 2 TiB/90 000 files 5) and hard 2,5 TiB/100 000 files 6).
1)
Soft quota is a limit that can be exceeded temporarily. If exceeded, the user will be informed.
2) , 4) , 6)
Hard quota is a limit that can not be exceeded.
3) , 5)
Soft quota is a limit that can be exceeded temporarily. If exceeding happens, the user will be informed.
Last modified: 16.07.2019 13:09