LUG - Connecting and Configuring Ceph RBD Using a Linux Client

Ceph RBD (RADOS Block Device) offers users a network block device that appears as a local disk on the system where it is connected. The block device is entirely managed by the user, who can create a file system on it and use it according to his needs.

Setting Up the RBD Client (Linux)

CentOS Setup

First, install the release.asc key for the Ceph repository.

sudo rpm --import 'https://download.ceph.com/keys/release.asc'

In the directory /etc/yum.repos.d/ create a text file ceph.repo and fill in the record for Ceph instruments.

[ceph]
    name=Ceph packages for $basearch
    baseurl=https://download.ceph.com/<contact-admin-for-current-version>/el7/$basearch
    enabled=1
    priority=2
    gpgcheck=1
    gpgkey=https://download.ceph.com/keys/release.asc

Some packages from the Ceph repository also require third-party libraries for proper functioning, so add the EPEL repository.

CentOS 7

sudo yum install -y epel-release

CentOS 8

sudo dnf install -y epel-release

RedHat 7

sudo yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm

Finally, install the basic tools for Ceph which also include RBD support.

CentOS 7

sudo yum install ceph-common

On CentOS 8

sudo dnf install ceph-common

Ubuntu/Debian Setup

Ubuntu/Ceph includes all necessary packages natively, so you can just run following command:

sudo apt install ceph

RBD Configuration and Mapping

To configure and connect to RBD, use the credentials provided by the system administrator. The necessary details are as follows:

• pool name: rbd_vo_poolname
• image name: vo_name_username
• keyring: [client.rbd_user] key = key_hash ==

In the directory /etc/ceph/ create a text file ceph.conf with the following content.

LUG Data Storage

[global]
    fsid = 3d8d9fd1-418a-4b13-b3d6-a437c9294ebf
    mon_host = [v2:195.113.231.135:3300/0,v1:195.113.231.135:6789/0] [v2:195.113.231.136:3300/0,v1:195.113.231.136:6789/0] [v2:195.113.231.137:3300/0,v1:195.113.231.137:6789/0]
    auth_client_required = cephx 

Next, in the /etc/ceph/ directory, create a text file ceph.keyring. Then, save the keyring in that file, as shown in the example below.

[client.rbd_user]
        key = sdsaetdfrterp+sfsdM3iKY5teisfsdXoZ5==

Now RBD mapping can be performed (rbd_user is a string originating from the keyring, after stripping the string client.

sudo rbd --id rbd_user --exclusive device map name_pool/name_image

sudo rbd -c /home/username/ceph/ceph.conf -k /home/username/ceph/username.keyring --id rbd_user --exclusive device map name_pool/name_image

Next, check the connection in kernel messages.

dmesg

Now, check the RBD status.

sudo rbd device list | grep "name_image"

Encrypting and Creating a Filesystem

The next step is to encrypt the mapped image use cryptsetup-luks.

sudo yum install cryptsetup-luks

Then, it encrypts the device.

sudo cryptsetup -s 512 luksFormat --type luks2 /dev/rbdX

Finally, check the settings.

sudo cryptsetup luksDump /dev/rbdX

To perform further actions on an encrypted device, it must be decrypted first.

sudo cryptsetup luksOpen /dev/rbdX luks_rbdX

Now, create file system on the device, here is an example xfs.

sudo mkfs.xfs -K /dev/mapper/luks_rbdX

Once the file system is ready, we can mount the device to a pre-created folder in /mnt/.

sudo mount /dev/mapper/luks_rbdX /mnt/rbd

Ending the Work with RBD

Unmount the volume.

sudo umount /mnt/rbd/

Close the encrypted volume.

sudo cryptsetup luksClose /dev/mapper/luks_rbdX

Volume unmapping.

sudo rbd --id rbd_user device unmap /dev/rbdX/

echo 8388608 > /sys/block/rbd0/queue/read_ahead_kb

echo 524288 > /sys/block/rbd0/queue/read_ahead_kb

You can configure specific kernel parameters for a subset of block devices (Ceph RBD)

KERNEL=="rbd[0-9]*", ENV{DEVTYPE}=="disk", ACTION=="add|change", ATTR{bdi/read_ahead_kb}="524288"

Permanently Mapping RBD

Configuration for automatic RBD connection, including LUKS encryption and filesystem mounting, along with proper disconnection (in reverse order) when the machine is switched off in a controlled manner.

RBD Image

Edit the configuration file located at /etc/ceph/rbdmap by adding the following lines.

# RbdDevice             Parameters
#poolname/imagename     id=client,keyring=/etc/ceph/ceph.client.keyring
pool_name/image_name id=rbd_user,keyring=/etc/ceph/ceph.keyring

LUKS

Edit configuration file located at /etc/crypttab by adding the following lines.

rbd_luks_pool /dev/rbd/pool_name/image_name  /etc/ceph/luks.keyfile luks,_netdev

where /etc/ceph/luks.keyfile is a LUKS key.

fstab file

Edit the configuration file located at /etc/fstab by adding the following lines.

/dev/mapper/rbd_luks_pool /mnt/rbd_luks_pool btrfs defaults,noatime,auto,_netdev 0 0

Systemd Unit

Edit the configuration file located at /etc/systemd/system/systemd-cryptsetup@rbd_luks_pool.service.d/10-deps.conf by adding the following lines.

[Unit]
    After=rbdmap.service
    Requires=rbdmap.service
    Before=mnt-rbd_luks_pool.mount

Manual Connection

If the systemd unit dependencies are correctly configured, it will performs the RBD mapping, unlock LUKS and mount all fs dependent on the rbdmap, as specified in the .mount unit needs (⇒ this will mount both images as described in the configuration).

systemctl start mnt-rbd_luks_pool.mount

Manual Disconnection

This command should execute correctly if the dependencies are set up properly, umount, close LUKS and unmap RBD.

systemctl stop rbdmap.service
(alternatively `systemctl stop ceph-rbdmap.service`)

Image Resize

When resizing an encrypted image, you need to follow the correct order, with the key step being the command cryptsetup --verbose resize image_name.

rbd resize rbd_pool_name/image_name --size 200T

cryptsetup --verbose resize image_name

mount /storage/rbd/image_name

xfs_growfs /dev/mapper/image_name

Snapshots of RBD image

A snapshot is a read-only logical copy of an image at a particular point in time: a checkpoint. One of the advanced features of Ceph block devices is that you can create snapshots of images to retain point-in-time state history. Ceph also supports snapshot layering, which allows you to clone images (e.g., a VM image) quickly and easily. Ceph block device snapshots are managed using the rbd command and multiple higher level interfaces, including QEMU, libvirt, OpenStack and CloudStack.

A In case the RBD image is NOT shared across more machines (see the case B), but there is even minimal risk of mapping your RBD and mounting your filesystem on multiple machines we recommend you to use --exclusive option. In this case, you have to create RBD snapshot on the unmapped RBD image always. You should also make sure that your image is not being mapped on any other device. You can use the command # rbd –id client_name status rbd_pool_name/rbd_image_name. In case that image is not mapped on any machine you should get the following output Watchers: none.

B In case you don’t use the option --exclusive it is sufficient to use fsfreeze command. After the snapshot is done you can enable I/O via fsfreeze with the option -unfreeze. More detail can be found on the man page of fsfreeze(8).

Snapshot basics

The following procedures demonstrate how to create, list, and remove snapshots using the rbd command.

Create snapshot

To create a snapshot with rbd, specify the snap create option, the pool name and the image name.

rbd snap create {pool-name}/{image-name}@{snap-name}

For example:

rbd snap create rbd/foo@snapname

List snapshots

To list snapshots of an image, specify the pool name and the image name.

rbd snap ls {pool-name}/{image-name}

For example:

rbd snap ls rbd/foo

Rollback snapshot

To rollback to a snapshot with rbd, specify the snap rollback option, the pool name, the image name and the snap name.

rbd snap rollback {pool-name}/{image-name}@{snap-name}

For example:

rbd snap rollback rbd/foo@snapname

Snapshot protection

For safe snapshots handling we strongly recommend assigning Protected to each snapshot you create. Protected attribute ensures that you can’t accidentally delete the existing snapshot.

rbd snap protect {pool-name}/{image-name}@{snapshot-name}

Example:

rbd snap protect rbd/foo@snapname

Unprotect the snapshots

rbd snap unprotect {pool-name}/{image-name}@{snapshot-name}

Example:

rbd snap unprotect rbd/foo@snapname

Delete snapshot

To delete a snapshot with rbd, specify the snap rm subcommand, the pool name, the image name and the snap name.

rbd snap rm {pool-name}/{image-name}@{snap-name}

For example:

rbd snap rm rbd/foo@snapname

Mapping and mounting the snapshot

Snapshot is possible to map and mount in read-only mode:

rbd map {pool-name}/{image-name}@{snap-name}
mount /dev/rbd/{pool-name}/{image-name}@{snap-name} /mnt/{mountpoint} -o ro

Example:

rbd map rbd/foo@snapname
mount /dev/rbd/rbd-pool/foo@snapname /mnt/snap -o ro

Be aware: some filesystems require the use of additional options for the mount command — e.g. XFS:

mount /dev/rbd/rbd-pool/foo@snapname /mnt/snap -o ro,norecovery