Table of Contents

NFS

CESNET provides a large a storage that can be accessed via NFSv4 protocol.

These storages can be also accessed through your personal computer. This guide contains instructions how to do it on machines running Linux OS. Unfortunately for MS Windows isn't NFSv4 client which would be free, there's only commercial NFS Maestro.

To ensure security of data is required authenticated connection via Kerberos. When you use proper options of mount command you can get encrypted communication.

What do we need?

To access the NFSv4 servers that CESNET provides you'll need the following things:

Hostname that can be transleted. You can vetify it with:

198.51.100.10   myclient.domain.com myclient
IPv4 addresses of access servers (hosts) for each data center are:
\ Ostrava nfs.du4.cesnet.cz
Jihlava nfs.du5.cesnet.cz
IPv6 addresses of access servers (hosts) for each data center are:

Ostrava nfs6.du4.cesnet.cz

Services using IPv6 addresses are currently in testing mode.

Mount point directory

A shared space should be mapped to any directory. Sdílený prostor je třeba mapovat do nějakého adresáře. We recommend to follow the convention established in MetaCentrum, i.e. subdirectory in /storage:

  mkdir -p /storage/cesnet-du4

Kerberos ticket for mounting directory

After you will attach mount point directory you need ticket. Ticket can get only root and he can get it one of two choices:

Depending on used way, the client's tools configuration is a little bit different.

Use user ticket with expiration period

As root is need to make a ticket with command

kinit user@EINFRA

where user is your login. Verify validity of ticket is described on this site. Disadvantage of this process is a limited ticket validity which means also limited time of mount device. After expiration of ticket is impossible to enter to device. Maximum period for this method of mounting is 8 days (maximum period of validity of renewable ticket).

This method needs to enter password and that is why is not possible to automatize making of ticket, f.e. by cron.

Use data saved in keytab /etc/krb5.keytab

If you get a special keytab for service indentities, you can skip this section.

Long-lasting keytab for users can be get by following method:

ssh -o PubkeyAuthentication=no -o GSSAPIAuthentication=no USER@SERVER "remctl kdccesnet.ics.muni.cz accounts nfskeytab" > krb5.keytab

where USER is your login used in EINFRA and SERVER depends, where you wish to connect (e.g. for Ostrava data storage “ssh.du4.cesnet.cz”).

Is necessary so a command ssh was authenticate by password, not by key or any other way. Without enter the password, command remctl will not work.

Place the obtain file krb5.keytab to /etc/krb5.keytab.

If file /etc/krb5.keytab already exist, you have to “merge” these files. Process is different for different Kerberos libraries versions and is described here..

The file owner should be root and permission for read and write should have only root, so:

 # ls -l /etc/krb5.keytab 
-rw------- 1 root root 57 Nov  2 09:48 /etc/krb5.keytab
Keytab file contains principal in form nfs/USER@EINFRA (or nfs/SERVICE_USER@EINFRA-SERVICES for special service identities) and a password. To anybody, who can read this file, is allowed to be like this saved account and so mount the NFS volume. In some special cases can be advantage that the file can be read by other, service accounts. In this case is more advantageous copy the file and give permission for read and write only to adequate service account. See more details bellow.
Because the password for convenient account is saved in keytab, this method doesn't ask enter the password and so can be used for automatic mounting of volumes.

Client's NFS tools

We recommend to install tools from our repositories. These packages already contains patches for right function of some attributes. Debian9 / Ubuntu18.04 (i386/amd64)

apt update
apt install nfs-common libnfsidmap2

Debian6 / Ubuntu12.04 (i386/amd64)

wget ftp://homeproj.cesnet.cz/apt/du/du.list -O /etc/apt/sources.list.d/cesnet-du.list
wget ftp://homeproj.cesnet.cz/apt/du/APT-GPG-KEY-du -O - | apt-key add -
apt-get update
apt-get install nfs-common libnfsidmap2

openSUSE12 (only x86_64)

zypper addrepo ftp://homeproj.cesnet.cz/rpm/du-opensuse/stable/x86_64/
zypper install --no-recommends nfs-client libnfsidmap

CentOS6 (only x86_64)

wget https://du.cesnet.cz/_media/cs/navody/nfs/cesnet-du.repo -O /etc/yum.repos.d/cesnet-du.repo
yum install nfs-utils nfs-utils-lib


For another OS we recommend have installed nfs-utils version 1.1.0 and higher.

At the same time you will need running rpcbind or (in older versions) portmap. It should be installed in dependencies of nfs-utils. If it's not, use same-named package (portmap, rpcbind).

Setup nfs-utils for NFSv4

After editing a configuration is necessary restart proper NFS services.
In case that you are mounting NFS with usual ticket, it is necessary have ticket as root before service start (nfs-common on Debian and Ubuntu version < 12.04, service gssd and Ubuntu > 12.04).

After correct setup nfs-utils should be possible to mount volume as root. You still have to do next steps.

Use user ticket with expiration period

Debian/Ubuntu

Setting nfs-utils is in file /etc/default/nfs-common. Set the values like this:

NEED_STATD=yes
STATDOPTS=
NEED_IDMAPD=yes
NEED_GSSD=yes

Next you need to add a line in /etc/default/nfs-common (for Debian and Ubuntu less than 12.04):

RPCGSSDOPTS="-n"

For Ubuntu 12.04 and newer is change in /etc/init/gssd.conf: Original line:

exec rpc.gssd

replace by:

exec rpc.gssd -n  
  

After these changes is necessary to restart service (if it has run - nfs-common/service gssd).

openSUSE

Setting of nfs-utils is in file /etc/sysconfig/nfs. Set up at least these values:

NFS_SECURITY_GSS="yes"
NFS4_SUPPORT="yes"  
RPCGSSDARGS="-n"
Fedora

Setup for nfs-utils is in the file /etc/nfs.conf. Set up at least these values:

SECURE_NFS="yes"  
RPCGSSDARGS="-n"

Use data saved in ''/etc/krb5.keytab''

Debian/Ubuntu

Settings for nfs-utils is in the file /etc/default/nfs-common. Set the values like this:

NEED_STATD=yes
STATDOPTS=
NEED_IDMAPD=yes
NEED_GSSD=yes      
openSUSE

Setting of nfs-utils is in file /etc/sysconfig/nfs. Set up at least these values:

NFS_SECURITY_GSS="yes"
NFS4_SUPPORT="yes"
Fedora

Setup for nfs-utils is in the file /etc/sysconfig/nfs. Set up at least this value:

SECURE_NFS="yes"

Kerberos ticket for data access

Root doesn't need to special ticket for access to volume, just ticket or keytab ''/etc/krb5.keytab''

Guide for user tickets with the necessity of password input

To access connected drive under different user than root is necessary to generate user ticket under the identity of local user. That can be done by using kinit command.

$localuser@localhost > kinit USER@EINFRA
Password for USER@EINFRA: 

where USER is your login in EINFRA.

To write out the validity of your ticket

$localuser@localhost > klist
Ticket cache: FILE:/tmp/krb5cc_1000
Default principal: USER@EINFRA

Valid starting    Expires           Service principal
27/06/2012 13:52  28/06/2012 13:52  krbtgt/EINFRA@EINFRA
$USER@localhost >
Access under this user to a mounted volume is possible only for the validity time of the ticket. After that you have to get a new ticket.

How to connect using keytab, without the necessity to use the password

In some special cases (typically backuping) could be preferable, that special (service) user could access the data storage without necessity to generate kerberos ticket or the password usage. That could be done by using keytab. Connection the drive. In the following guide we will assume, that the local user name is bacula.

First ew can copy the file /etc/krb5.keytab, which should be accessible only for user root into different location and we will set the premissions for bacula user:

root@localhost > cp /etc/krb5.keytab /etc/krb5.keytab.bacula
root@localhost > chown bacula:bacula /etc/krb5.keytab.bacula

From this file is then possible periodically obtain the ticket by using following command (under bacula bacula user):

bacula@localhost > kinit -k -t /etc/krb5.keytab.bacula nfs/USER@EINFRA 

Like this we will get ticket with one day validity (under the user bacula). The command is no longer required to enter a password, therefore can be automate eg. using cron.

If you have a special service identity, format of principal may vary slightly, but you can get it by the procedure below.

Correct format of principal (string nfs/USER@EINFRA) can be get by:

klist -k /etc/krb5.keytab.bacula
Keytab name: WRFILE:/etc/krb5.keytab.bacula
KVNO Principal
---- --------------------------------------------------------------------------
   1 nfs/USER@EINFRA

/etc/fstab setting

Into file /etc/fstab write line:

nfs.du4.cesnet.cz:/ /storage/cesnet-du4   nfs4   sec=krb5,rsize=1048576,wsize=1048576   0 0

Volume will be then mount automatically.

Options rsize=1048576,wsize=1048576 can improve performence of NFS. Especially over longer distances. They are recommended.

Then you need to start the nfs and ensure its automatic start after reboot:

openSUSE

/etc/init.d/nfs start 
insserv /etc/init.d/nfs

Debian

/etc/init.d/nfs-common start

Ubuntu

service portmap start
service gssd start

CentOS 6

chkconfig rpcbind on
chkconfig nfs on
chkconfig rpcsvcgssd on
chkconfig rpcidmapd on
service nfs

CentOS 7

systemctl enable nfs-idmap
systemctl start nfs-idmap
systemctl enable nfs-secure
systemctl start nfs-secure

/etc/idmapd.conf setting

File /etc/idmapd.conf set up NFSv4 mapping (NFSv4 works with text principal and POSIX interface works with numeric representation of users and groups).

Simple mappping

Simple settings in /etc/idmapd.conf is to set following line in section [General]:

  Domain = EINFRA
If you use ticket obtain from keytab or special service user account, you will need to use static mapping
If you have service user account use instead EINFRA realm EINFRA-SERVICES realm.
Start from libnfsidmap 0.21 (debian 5 and newer) is possible to have in section [General] in /etc/idmapd.conf rows:
Local-Realms = realm1,realm2,realm3
for more realms (only special cases).
e.g.: Domain = META a Local-Realms = EINFRA

For correct functioning is neccessary to install our NFS cleint tools.


After that will be all users from domain @EINFRA mapped by using the files /etc/passwd and /etc/group. It means that for the identity login@EINFRA must exist the record in /etc/passwd with the login name. Common programs (ls -l) shall show names and groups from domain EINFRA correctly, only if there will be for their principal the name in the particular files. For non-existing name will be displayed the mapping as nobody and nogroup.

Probably you will see owner only of your own files (if you have the same login on your computer like in EINFRA. Otherwise you will see the owner “nobody” and group “nogroup” in every file), and in other files the owner will be “nobody” and group “nogroup”. This image can have influence to permissions to the files, depends on optimization of client's kernel, even that permissions are dealed by server and its configuration. So it's possible that with a wrong configuration of mapping, some operation won't be done, even the server sees everything is alright – client doesn't send request to server at all. But it's not possible to client get more permissions like it actually should get.

Actual versions nfs-utils offering the option of the static users mapping (useful when you have a different login on your local computer then login in EINFRA), but an implementation is function-less. Setting of a function static mapping is more difficult and it's described bellow. Standard support is only for mapping method “nsswitch”, so this is necessary in appropriate section:

[Translation]

Method = nsswitch

or you don't have to that at all, method nsswitch is implied.

Static mapping

In some cases is not possible to ensure the same user's login in client's side, as in the storage. Then, some operations can't work properly, for example you can't use ENCFS over NFS volume. For functional static mapping is necessary to have a newer version of program libnfsidmapd, which includes this patch.

If you have installed any client tool from our repositories, you install libnfsidmap with these patch.
If your distribution package contains nfs-utils (sometimes called nfs-common) in version lower than 1.2.6,it is very likely that bug has not been repaired. Bug causes wrong parsing of configuration file idmapd.conf

For the compilation, we first need to download the latest version of libnfsidmap:

cd ~/
git clone git://git.linux-nfs.org/projects/steved/libnfsidmap.git

Then make compilation:

cd libnfsidmap
./autogen.sh 
./configure
make
For compilation process you have to installed packages automake and libtool.

It is also necessary run make install with root privileges. When you start rpc.idmapd daemon again it will use this version of the library.

Then it is necessary to modify the relevant sections in the configuration file /etc/idmapd.conf:

[Translation]
Method = static, nsswitch

[Static]

user@EINFRA = localuser
user2@EINFRA = anotherlocaluser
storage@EINFRA = users

Configuration shows static mapping of user user@EINFRA, user2@EINFRA and group storage@EINFRA. In case of other users and groups will be mapping performed by plugin 'nsswitch which is described above. Users who don't use any of these two methods will be mapped as nobody''.

When you are using tickets get from keytab or with special service account, you have to use the following mapping:
nfs/USER@EINFRA = localuser

prospective

nfs/backup_VO@EINFRA-SERVICES = localuser

How to connect volume

Connection of volume isn't necessary to perform manually (for Ubuntu and openSUSE 11.1 and higher). The nfs service starts automatically because of record in /etc/fstab.

If the connection doesn't established automatically, you can do it manually. Connection of volumes must be performed with root privileges. Make sure you have the correct /etc/krb5.keytab or created user ticket under the root. Also make sure that portmap and nfs-common (nfs-utils) are running. You can verify it by following instructions:

ps ax | grep rpc.gssd
ps ax | grep portmap

To connect perform following command:

mount -t nfs  nfs.du4.cesnet.cz:/ /storage/cesnet-du4 -o vers=4 -o sec=krb5 -o rsize=1048576,wsize=1048576

You can use instead of /storage/xxx je možno zadat any other local directory, where do you want to mount NFS volume. If you use sec=krb5 option instead of sec=krb5i the integrity will be checked. If you use sec=krb5p then data will be transmitted encrypted.

Result

If you followed steps in this guide you have mounted storage and you can use ticket to access data storage.

Difficulties

In case of problems, please contact our support see FAQ. To speed up the solution you can send output of the following commands (report.log) to to the support. All command must be pertorm as root or with sudo.

wget http://www.fi.muni.cz/~xhejtman/check-nfs.sh
mount nfs4 [příkaz pro mount, který následně selže nebo jiný příkaz, který se chová jinak, než by dle návodu měl]
. ./check-nfs.sh &> /tmp/report.log

FAQ

Attaching more volumes with krb5 identities

Problem description Can I attach more volumes with various krb5 identities?

Solution: Start from Kerberos 1.10 it is possible to use the folder to keep your identity tickets instead of the file itself, so you can have more active identities in the same time and you can switch between them. Usually there is created the file in /tmp/ (e.g. /tmp/krb5cc_0_dX1u6IH8jO). It is possible to change the path into the selected folder.

It is necessary for this purpose to run gssd with “-n” option.
All changes mentioned here are not permanent. After reboot it will be neccesary to initialize cache again by the same way.

First of all we will prepare a folder, where we want to save identity tickets. We will create this folder in /run/, because this folder will be cleaned after reboot.

# mkdir -p /run/user/0/krb5cc
# chmod og-rwx /run/user/0/krb5cc

We will set up into the variable KRB5CCNAME path to the given folder.

# export KRB5CCNAME=DIR:/run/user/0/krb5cc

Now our folder for saving identity tickets is prepared. We can start to initialize the identities.

# kinit user@DOMAIN1
# kinit user@EINFRA

To write out all available identity tickets we can use commands klist -A or klist -l.

# klist -A
Ticket cache: DIR::/run/user/0/krb5cc/tkta0Hysv
Default principal: user@DOMAIN1

Valid starting     Expires            Service principal
9.6.2015 13:54:44  9.6.2015 23:54:38  krbtgt/DOMAIN1@DOMAIN1
	renew until 24/06/2015 13:54 

Ticket cache: DIR::/run/user/0/krb5cc/tkt5PDllX
Default principal: user@EINFRA

Valid starting     Expires            Service principal
9.6.2015 13:54:22  9.6.2015 23:54:16  krbtgt/EINFRA@EINFRA
	renew until 16/06/2015 13:54 


# klist -l
Principal name                 Cache name
--------------                 ----------
user@DOMAIN1                   DIR::/run/user/0/krb5cc/tkta0Hysv
user@EINFRA                    DIR::/run/user/0/krb5cc/tkt5PDllX

We can see, which identity is active by using the command klist.

# klist
Ticket cache: DIR::/run/user/0/krb5cc/tkta0Hysv
Default principal: user@DOMAIN1

Valid starting     Expires            Service principal
9.6.2015 13:54:44  9.6.2015 23:54:38  krbtgt/DOMAIN1@DOMAIN1
	renew until 24/06/2015 13:54

To switch the identity we can use command kswitch.

# kswitch -p user@EINFRA

# klist
Ticket cache: DIR::/run/user/0/krb5cc/tkt5PDllX
Default principal: user@EINFRA

Valid starting     Expires            Service principal
9.6.2015 13:54:22  9.6.2015 23:54:16  krbtgt/EINFRA@EINFRA
	renew until 16/06/2015 13:54

In case we want to attach two NFSv4 volumes with different identities, we can run the command mount with active first identity, then we can switch the identity via kswitch and attach second volume with active second identity.

# mount -o sec=krb5i,proto=tcp,port=2049,intr storage-jihlava1-cerit.metacentrum.cz:/ /mnt/storage-cerit
# kswitch -p user@EINFRA
# mount -o rw,nfsvers=4,hard,intr,sec=krb5i nfs.du4.cesnet.cz:~/ /mnt/storage-du4
IPv4 addresses of access servers (hosts) for each data center are:

Ostrava ssh.du4.cesnet.cz
Jihlava ssh.du5.cesnet.cz

IPv6 addresses of access servers (hosts) for each data center are:

Ostrava ssh6.du4.cesnet.cz

Services using IPv6 addresses are currently in testing mode.

Issue with connecting via NFS to the new Data Storage

Problem description I tried to connect to the new Data Storage using NFS protocol, but it was not working.

Solution: In case of Ostrava Data Storage you have to insert the line into krb5.conf for [domain_realm] for Ostrava Data Storage, similarly for other new Data Storages.

.du4.cesnet.cz = EINFRA-SERVICES

Issue with connecting NFS via Kerberos ticket

Problem description: I tried to connect via NFS protocol, but I've got following error report.

mount.nfs: Network is unreachable

Solution: The issue is caused by failed startup rpc-gssd service, see the service status:

systemctl status rpc-gssd
Condition: start condition failed at Thu 2018-05-06 06:12:22 CET; 20s ago
           └─ ConditionPathExists=/etc/krb5.keytab was not met

To resolve this issue, you can create empty file in the /etc folder via following command:

touch /etc/krb5.keytab

Now you can restart rpc-gssd service and remount your storage.