cephfs + Ganesha

cephfs + Ganesha

软件环境

软件名称	版本
CentOS	7.6.1810 (Core)
Ceph	13.2.5 (cbff874f9007f1869bfd3821b7e33b2a6ffd4988) mimic (stable)
NFS-Ganesha	V2.7.6
ntirpc	1.7.4 (50eadf8dce7ff67e4295acfe46b837e11a8f586f)

构建NFS-Ganesha

获取NFS-Ganesha

2.7稳定版：https://github.com/nfs-ganesha/nfs-ganesha/tree/V2.7-stable

进入nfs-ganesha的目录src下查看相对应的ntirpc源码CommitID

对ntrirpc的版本要求是50eadf8(https://github.com/nfs-ganesha/ntirpc/tree/50eadf8dce7ff67e4295acfe46b837e11a8f586f),下载 50eadf8 的ntrirpc源码包，并把源码包的内容拷贝到nfs-ganesha的libntrirpc目录。

安装依赖包

# yum -y install gcc git cmake autoconf libtool bison flex
# yum -y install libgssglue-devel openssl-devel nfs-utils-lib-devel doxygen redhat-lsb gcc-c++
# yum -y install uuid uuidd libuuid libuuid-devel
# yum -y install librgw2 librgw-devel libcephfs2 libcephfs-devel

构建与编译

# mkdir build;cd build
# cmake -DUSE_FSAL_CEPH=ON -DUSE_FSAL_RGW=ON  ~/nfs-ganesha-2.7-stable/src/
# make
# make install

在make install生成的输出中，可以看到:

– Up-to-date: /usr/share/doc/ganesha/config_samples/rgw.conf
…
– Up-to-date: /usr/share/doc/ganesha/config_samples/ceph.conf

这两个文件就是配置将RGW和CephFS配置为ganesha-nfs的配置模板。

编译nfs-ganesha前需安装libcephfs2，否则USE_FSAL_CEPH会一直是OFF状态

配置Ganesha

主要命令

# ganesha.nfsd -v
NFS-Ganesha Release = V2.7.6
nfs-ganesha compiled on Jul 22 2019 at 17:40:12
Release comment = GANESHA file server is 64 bits compliant and supports NFS v3,4.0,4.1 (pNFS) and 9P
Git HEAD = NOT-GIT
Git Describe = /root/nfs-ganesha-2.7-stable/src# ganesha.nfsd -h
Usage: nfs-ganesha [-hd][-L <logfile>][-N <dbg_lvl>][-f <config_file>][-v]                display version information[-L <logfile>]      set the default logfile for the daemon[-N <dbg_lvl>]      set the verbosity level[-f <config_file>]  set the config file to be used[-p <pid_file>]     set the pid file[-F]                the program stays in foreground[-R]                daemon will manage RPCSEC_GSS (default is no RPCSEC_GSS)[-T]                dump the default configuration on stdout[-E] <epoch>]       overrides ServerBootTime for ServerEpoch[-C]                dump trace when segfault[-h]                display this help
----------------- Signals ----------------
SIGUSR1    : Enable/Disable File Content Cache forced flush
SIGTERM    : Cleanly terminate the program
------------- Default Values -------------
LogFile    : SYSLOG
PidFile    : /var/run/ganesha.pid
DebugLevel : NIV_EVENT
ConfigFile : /etc/ganesha/ganesha.conf

编辑配置文件

1. 在ganesha.conf中末尾定义cephfs配置文件。

.....
%include "/etc/ganesha/ceph.conf"

2. 编辑/etc/ganesha/ceph.conf：

NFSv4
{
    Minor_Versions =  1,2;
}MDCACHE {
    Dir_Chunk = 0;NParts = 1;Cache_Size = 1;
}EXPORT
{
    Export_ID=2000;Protocols = 4;Transports = TCP;Path = /;Pseudo = /shares/;Access_Type = RW;Attr_Expiration_Time = 0;Squash = No_root_squash; SecType = sys;FSAL {
    Name = CEPH;}
}

启动Ganesha

执行如下命令：

# ganesha.nfsd -f<location_of_nfs-ganesha.conf_file> -L <location_of_log_file> -N<log_level>

例如：

# ganesha.nfsd -f /etc/ganesha/ganesha.conf -L /var/log/ganesha/ganesha.log -N NIV_DEBUG

其中:

/var/log/ganesha/ganesha.log 是ganesha.nfsd进程的日志文件

NIV_DEBUG是日志级别.

检查ganesha是否启动：

# ps -ef| grep ganesha
root       23092       1 33 16:08 ?        00:17:47 ganesha.nfsd -f /etc/ganesha/ganesha.conf -L /var/log/ganesha/ganesha.log -N NIV_DEBUG

启用/禁用文件内容缓存强制刷新

# killall -s SIGUSR1 ganesha.nfsd

停止Ganesha进程

# killall -s SIGTERM ganesha.nfsd

检查导出

# showmount -e
Export list for caohf08:

理应有输出，不过此处导出列表确实为空；但实际已正常导出

客户端挂载NFS

# mkdir /mnt/nfs
# mount -t nfs -o nfsvers=4.1,proto=tcp 100.10.10.167:/shares /mnt/nfs

参考文档

https://github.com/nfs-ganesha/nfs-ganesha/wiki/Compiling

https://blog.csdn.net/younger_china/article/details/73412191

https://segmentfault.com/a/1190000012348558