Ceph RBD (RADOS Block Device) provides users with a network block device that looks like a local disk on the system where it is connected. The block device is fully managed by the user. An user can create a file system there and use it according to his needs.
The following instructions are valid for the CENTOS / RHEL distribution. Instructions for UBUNTU / DEBIAN are at the end of this section .
First, install the release.asc key for the Ceph repository.
sudo rpm --import 'https://download.ceph.com/keys/release.asc'
In the directory /etc/yum.repos.d/ create a text file ceph.repo and fill in the record for Ceph instruments. For CentOS 9, we modify the baseurl line from “rpm-nautilus/el7” to “/rpm-reef/el9/”.
Some packages from the Ceph repository also require third-party libraries for proper functioning, so add the EPEL repository.
CentOS 9
sudo dnf install -y epel-release
RedHat 9
sudo yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-9.noarch.rpm
Finally, install the basic tools for Ceph which also include RBD support.
sudo yum install ceph-common
Use the credentials which you received from the system administrator to configure and connect the RBD. These are the following:
In the directory /etc/ceph/ create the text file ceph.conf with the following content.
In the case of the Jihlava data storage with a code designation CL1:
[global] fsid = 19f6785a-70e1-45e8-a23a-5cff0c39aa54 mon initial members = mon001-cl1-aba-jihl1,mon002-cl1-aba-jihl1,mon003-cl1-aba-jihl1 mon host = [v2:78.128.244.33:3300,v1:78.128.244.33:6789],[v2:78.128.244.37:3300,v1:78.128.244.37:6789],[v2:78.128.244.41:3300,v1:78.128.244.41:6789] auth_client_required = cephx
In the case of the Pilsen data storage with a code designation CL2:
[global] fsid = 3ea58563-c8b9-4e63-84b0-a504a5c71f76 mon_initial_members = mon001-cl2-aba-plz1,mon005-cl2-aba-plz1,mon007-cl2-aba-plz1 mon_host = [v2:78.128.244.65:3300/0,v1:78.128.244.65:6789/0],[v2:78.128.244.69:3300/0,v1:78.128.244.69:6789/0],[v2:78.128.244.71:3300/0,v1:78.128.244.71:6789/0] auth_client_required = cephx
In the case of the Ostrava data storage with a code designation CL3:
[global] fsid = b16aa2d2-fbe7-4f35-bc2f-3de29100e958 mon_initial_members = mon001-cl3,mon002-cl3,mon003-cl3 mon_host = [v2:78.128.244.240:3300/0,v1:78.128.244.240:6789/0],[v2:78.128.244.241:3300/0,v1:78.128.244.241:6789/0],[v2:78.128.244.242:3300/0,v1:78.128.244.242:6789/0] auth_client_required = cephx
In the case of the Brno data storage with a code designation CL4:
[global] fsid = c4ad8c6f-7ef3-4b0e-873c-b16b00b5aac4 mon_initial_members = mon001-cl4,mon002-cl4,mon003-cl4,mon004-cl4,mon005-cl4 mon_host = [v2:78.128.245.29:3300/0,v1:78.128.245.29:6789/0] [v2:78.128.245.30:3300/0,v1:78.128.245.30:6789/0] [v2:78.128.245.31:3300/0,v1:78.128.245.31:6789/0] auth_client_required = cephx
In the case of the Prague data storage with a code designation CL5:
[global] fsid = c581dace-40ff-4519-878b-c0ffeec0ffee mon_initial_members = mon001-cl5,mon002-cl5,mon003-cl5,mon004-cl5,mon005-cl5 mon_host = [v2:78.128.245.157:3300/0,v1:78.128.245.157:6789/0] [v2:78.128.245.158:3300/0,v1:78.128.245.158:6789/0] [v2:78.128.245.159:3300/0,v1:78.128.245.159:6789/0] auth_client_required = cephx
Further in the directory /etc/ceph/ create the text file ceph.keyring. Then save in that file the keyring, see the example below.
[client.rbd_user] key = sdsaetdfrterp+sfsdM3iKY5teisfsdXoZ5==
We strongly recommend using –exclusive option while mapping the RBD image. That option will prevent mapping an image on multiple devices or multiple times locally. This multiple mapping can cause data corruption! So please be aware that if you foresee any risk of multiple mapping then use “–exclusive” option.
On the other hand do not use –exclusive option in case you need to mount the RBD image on multiple machines, e.g. clustered file system.
Now RBD mapping can be performed (rbd_user is a string originating from the keyring, after stripping the string client..
sudo rbd --id rbd_user --exclusive device map name_pool/name_image
sudo rbd -c /home/username/ceph/ceph.conf -k /home/username/ceph/username.keyring --id rbd_user device map name_pool/name_image
Then check the connection in kernel messages.
dmesg
Now check the status of RBD.
sudo rbd device list | grep "name_image"
The next step is to encrypt the mapped image. Use cryptsetup-luks for encryption
sudo yum install cryptsetup-luks
Then it encrypts the device.
sudo cryptsetup -s 512 luksFormat --type luks2 /dev/rbdX
Finally, check the settings.
sudo cryptsetup luksDump /dev/rbdX
In order to perform further actions on an encrypted device, it must be decrypted first.
sudo cryptsetup luksOpen /dev/rbdX luks_rbdX
We recommend using XFS instead of EXT4 for larger images or those they will need to be enlarged to more than 200TB over time, because EXT4 has a limit on the number of inodes.
Now create file system on the device, here is an example xfs.
sudo mkfs.xfs /dev/mapper/luks_rbdX
Once the file system is ready, we can mount the device in a pre-created folder in /mnt/.
sudo mount /dev/mapper/luks_rbdX /mnt/rbd
Unmount the volume.
sudo umount /mnt/rbd/
Volume lock.
sudo cryptsetup luksClose /dev/mapper/luks_rbdX
Volume unmapping.
sudo rbd --id rbd_user device unmap /dev/rbdX/
Example for 8GB:
echo 8388608 > /sys/block/rbd0/queue/read_ahead_kb
Example for 512MB:
echo 524288 > /sys/block/rbd0/queue/read_ahead_kb
To apply changes you have to unmap image and map it again.
The approach described above is not persistent (won't survive reboot). To do it persistent you have to add following line into “/etc/udev/rules.d/50-read-ahead-kb.rules” file.
# Setting specific kernel parameters for a subset of block devices (Ceph RBD)
KERNEL=="rbd[0-9]*", ENV{DEVTYPE}=="disk", ACTION=="add|change", ATTR{bdi/read_ahead_kb}="524288"
Settings for automatic RBD connection, including LUKS encryption and mount filesystems. + proper disconnection (in reverse order) when the machine is switched off in a controlled manner.
systemctl start mnt-rbd_luks.mount
- If the dependencies of the systemd units are correct, it performs an RBD map, unlocks LUKS and mounts all the automatic fs dependent on the rbdmap that the specified .mount unit needs (⇒ mounts both images in the described configuration).
systemctl stop rbdmap.service
(resp. systemctl stop ceph-rbdmap.service)
- this command should execute if the dependencies are set correctly umount, LUKS close i RBD unmap.
When resizing an encrypted image, you need to follow the order and use the right resizing tool.
rbd -c ceph_conf -k ceph_keyring --id ceph_user resize rbd_pool_name/image_name --size 200T rbd -c ceph_conf -k ceph_keyring --id ceph_user device map rbd_pool_name/image_name cryptsetup open --key-file luks_key_file /dev/rbd/rbd_pool_name/image_name rbd_luks cryptsetup resize --key-file luks_key_file --verbose rbd_luks mount /dev/mapper/rbd_luks /mnt/mount_point xfs_growfs /mnt/mount_point
Problem Description: Unable to map RBD and the connection times out.
Solution:
Most likely, your firewall is blocking the initiation of communication to the internet. It is necessary to allow the range according to the given cluster on ports 3300 and the range of ports 6789-7300.
1. Set “related/established” on the firewall.
2. Configure the firewall for the cluster range clX and allow ports 3300/tcp and the range of ports 6789-7300/tcp.
cl1 - 78.128.244.32/27 cl2 - 78.128.244.64/26 cl3 - 78.128.244.128/25 cl4 - 78.128.245.0/25 cl5 - 78.128.245.128/25
3. Activate jumbo frames (support for large frames). These must be correctly configured throughout the entire path up to the CESNET backbone network, i.e., they must be set on all your active network elements towards the CESNET network, as well as on the server where you are trying to connect the RBD image. We recommend setting 9000 bytes on the server; for active network elements, it depends on several factors, which you should discuss with your network administrator.