Ceph RBD (RADOS Block Device) provides users with a network block device that looks like a local disk on the system where it is connected. The block device is fully managed by the user. An user can create a file system there and use it according to his needs.
The following instructions are valid for the CENTOS / RHEL distribution. Instructions for UBUNTU / DEBIAN are at the end of this section .
First, install the release.asc key for the Ceph repository.
sudo rpm --import 'https://download.ceph.com/keys/release.asc'
In the directory /etc/yum.repos.d/ create a text file ceph.repo and fill in the record for Ceph instruments.
Some packages from the Ceph repository also require third-party libraries for proper functioning, so add the EPEL repository.
sudo yum install -y epel-release
sudo dnf install -y epel-release
sudo yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
Finally, install the basic tools for Ceph which also include RBD support.
sudo yum install ceph-common
Use the credentials which you received from the system administrator to configure and connect the RBD. These are the following:
In the directory /etc/ceph/ create the text file ceph.conf with the following content.
In the case of the Jihlava data storage with a code designation CL1:
[global] fsid = 19f6785a-70e1-45e8-a23a-5cff0c39aa54 mon initial members = mon001-cl1-aba-jihl1,mon002-cl1-aba-jihl1,mon003-cl1-aba-jihl1 mon host = [v2:18.104.22.168:3300,v1:22.214.171.124:6789],[v2:126.96.36.199:3300,v1:188.8.131.52:6789],[v2:184.108.40.206:3300,v1:220.127.116.11:6789] auth_client_required = cephx
In the case of the Pilsen data storage with a code designation CL2:
[global] fsid = 3ea58563-c8b9-4e63-84b0-a504a5c71f76 mon_initial_members = mon001-cl2-aba-plz1,mon005-cl2-aba-plz1,mon007-cl2-aba-plz1 mon_host = [v2:18.104.22.168:3300/0,v1:22.214.171.124:6789/0],[v2:126.96.36.199:3300/0,v1:188.8.131.52:6789/0],[v2:184.108.40.206:3300/0,v1:220.127.116.11:6789/0] auth_client_required = cephx
In the case of the Ostrava data storage with a code designation CL3:
[global] fsid = b16aa2d2-fbe7-4f35-bc2f-3de29100e958 mon_initial_members = mon001-cl3,mon002-cl3,mon003-cl3 mon_host = [v2:18.104.22.168:3300/0,v1:22.214.171.124:6789/0],[v2:126.96.36.199:3300/0,v1:188.8.131.52:6789/0],[v2:184.108.40.206:3300/0,v1:220.127.116.11:6789/0] auth_client_required = cephx
Further in the directory /etc/ceph/ create the text file ceph.keyring. Then save in that file the keyring, see the example below.
[client.rbd_user] key = sdsaetdfrterp+sfsdM3iKY5teisfsdXoZ5==
We strongly recommend using –exclusive option while mapping the RBD image. That option will prevent mapping an image on multiple devices or multiple times locally. This multiple mapping can cause data corruption! So please be aware that if you foresee any risk of multiple mapping then use “–exclusive” option.
On the other hand do not use –exclusive option in case you need to mount the RBD image on multiple machines, e.g. clustered file system.
Now RBD mapping can be performed (rbd_user is a string originating from the keyring, after stripping the string client..
sudo rbd --id rbd_user --exclusive device map name_pool/name_image
sudo rbd -c /home/username/ceph/ceph.conf -k /home/username/ceph/username.keyring --id rbd_user device map name_pool/name_image
Then check the connection in kernel messages.
Now check the status of RBD.
sudo rbd device list | grep "name_image"
The next step is to encrypt the mapped image. Use cryptsetup-luks for encryption
sudo yum install cryptsetup-luks
Then it encrypts the device.
sudo cryptsetup -s 512 luksFormat --type luks2 /dev/rbdX
Finally, check the settings.
sudo cryptsetup luksDump /dev/rbdX
In order to perform further actions on an encrypted device, it must be decrypted first.
sudo cryptsetup luksOpen /dev/rbdX luks_rbdX
We recommend using XFS instead of EXT4 for larger images or those they will need to be enlarged to more than 200TB over time, because EXT4 has a limit on the number of inodes.
Now create file system on the device, here is an example xfs.
sudo mkfs.xfs /dev/mapper/luks_rbdX
Once the file system is ready, we can mount the device in a pre-created folder in /mnt/.
sudo mount /dev/mapper/luks_rbdX /mnt/rbd
Unmount the volume.
sudo umount /mnt/rbd/
sudo cryptsetup luksClose /dev/mapper/luks_rbdX
sudo rbd --id rbd_user device unmap /dev/rbdX/
Example for 8GB:
echo 8388608 > /sys/block/rbd0/queue/read_ahead_kb
Example for 512MB:
echo 524288 > /sys/block/rbd0/queue/read_ahead_kb
To apply changes you have to unmap image and map it again.
Settings for automatic RBD connection, including LUKS encryption and mount filesystems. + proper disconnection (in reverse order) when the machine is switched off in a controlled manner.
systemctl start mnt-rbd_luks_pool.mount
- If the dependencies of the systemd units are correct, it performs an RBD map, unlocks LUKS and mounts all the automatic fs dependent on the rbdmap that the specified .mount unit needs (⇒ mounts both images in the described configuration).
systemctl stop rbdmap.service
systemctl stop ceph-rbdmap.service)
- this command should execute if the dependencies are set correctly
close i RBD unmap.
when resizing an encrypted image, you need to follow the order and the main one is the line with cryptsetup –verbose resize image_name
rbd resize rbd_pool_name/image_name --size 200T cryptsetup --verbose resize image_name mount /storage/rbd/image_name xfs_growfs /dev/mapper/image_name