Home directories and migration policies on the HSMs

This document describes directory (folder) structures on CESNET hierarchical storage facilities.

Home directory

A user home directory contains subdirectories that correspond to virtual organizations (VO) the user is a member of, and migration policies. These subdirectories are used to store data. They are symbolic links (shortcut folders) pointing to the directory structure of the VO.

The home directory isn't designed to store your data. Its just for storing SSH keys and other configuration files, and for creating links to the real data. The home directory has soft quotas 50 MiB/500 files 1) and hard 5 GiB/5000 files 2).

Example for user “user” with membership in two VOs (eg. VO_compass and VO_storage), one of them uses two different migration policies for du4, du5, …:

|-- .bash_history
|-- .screenrc
|-- .ssh
|  \-- authorized_keys
|-- VO_compass-tape_tape -> /exports/tape_tape/backup/VO_compass/home/user
|-- VO_compass-tape_tape-archive -> /exports/tape_tape/archive/VO_compass/home/user
|-- VO_compass-disk_only -> /exports/disk_only/backup/VO_compass/home/user
Please pay attention, that only some of the symlink names in case of du4 contain the word “archive”, e.g. VO_compass-tape_tape-archive. That is related to the change in the Terms of Service. All data on new Data Storage facilities, starting from du4, are treated as a backups and archives - data with permanent value. Backups can be stored only for particular period specified in the Terms of Service. All directories without designation “archive” are considered as backup.

This structure we will automatically prepare to the user whenever he becomes a member of VO. Users can of course create a different structure using symbolic links. However, the links created by us, we do not recommended to delete.

How do I get information about my quota and occupied space?

We are preparing an Accounting system. This system will show occupied space and quota settings. Accounting system is coming soon.

Migration policies

Migration policy determines what happens to your data when you run out of space on the disks, where data that no one else is using will be moved etc. Read the page on HSM (Hierarchical Storage Management) if you need information about the principles of data migration.

Migration policy agreed administrator of virtual organization with administrators of data storage CESNET. Virtual organizations often use only one or a few policies.

Currently available are the following migration policies.

  1. tape - data on SATA disks and copy on a single tape (suitable for second or third backups)
  2. tape_tape - data on SATA disks and copy on two different tapes (for standard data and first backups)
    1. In Brno and Jihlava data storage are data on the discs, one copy is on MAID and the second copy is on tape (this is suitable for common data and first backup).
    2. In Ostrava data storage are data on the discs (Ostrava data storage does not have MAID discs) and on the tape. If you do not use your data, they are migrated from discs to the tapes. The data are in the end up in two copies on the tapes.
  3. disk_only - data is only for disks (RAID), and never moved nor backed up (for fast-changing data, e.g., backups you delete in several days)
    1. In Jihlava data storage (du5) are data on the discs array. On du5 we handle the redundancy by using snapshots, which are kept for several days.
Minimal size o migrable file is 64 kiB, i.e. files smaller than 64 kiB are not migrated to the tapes and remain on a disc tier.

If the user “user” (member in “storage”) wants to use specific migration policy “tape_tape” (data on two different tapes) than he uploads data into the directory “VO_storage-tape_tape” in his home directory.

Virtual organization "storage"

Below is the list of migration policies and their quotas contained in the virtual organization „storage“ which each registered user is automatically a member.

  1. tape_tape-archive - policy is suitable for storing livelier data. It is not recommended to store backups.
    • Soft quota 500 MiB/400 000 files 3) and hard quota 500 MiB/500 000 files 4).
  2. tape_tape - it is suitable to use this policy primarily for storing backups. Policy called “backup” allows to store data maximally for 12 months. Older data could be deleted, see Terms of Service.
    • Soft quota 2 TiB/90 000 files 5) and hard 2,5 TiB/100 000 files 6).
Soft quota is a limit that can be exceeded temporarily. If exceeded, the user will be informed.
2) , 4) , 6)
Hard quota is a limit that can not be exceeded.
3) , 5)
Soft quota is a limit that can be exceeded temporarily. If exceeding happens, the user will be informed.
Last modified: 23.03.2021 17:13