初始安装及初始配置
MogileFS是基于Perl语言构建的项目,因此需要从CPAN上面下载MogileFS的模块,因此需要保证被部署的机器能够直接或者通过代理连接至互联网,如果无法联网,则需要梳理模块的依赖关系,通过其他途径下载所有所需要的perl的模块,并且手工编译安装。
首先,在两台机器上面安装cpanm工具,用于安装模块:
wget http://xrl.us/cpanm -O /usr/bin/cpanm; sudo chmod +x /usr/bin/cpanmcpanm安装完毕之后,在两台机器上面安装MogileFS所需要的模块:
cpanm App::cpanminus
cpanm IO::AIO
cpanm IO::WrapTie
cpanm Danga::Socket
cpanm DBD::mysql
cpanm MogileFS::Server
cpanm MogileFS::Client
cpanm MogileFS::Utils模块安装完毕之后,对Sys::Syscall模块进行降级操作。默认情况下通过cpan安装完MogileFS之后,其依赖模块Sys::Syscall的版本为0.25,如果不降级的话,会在运行时的log里面出现crash信息,并且无法完成devcount的冗余。笔者这里下载的旧版本是0.23版本的模块压缩包。由于该模块的特性,可以直接跳过编译步骤,将解压之后的模块文件夹里面的lib/Sys/Syscall.pm直接拷贝到/usr/local/share/perl5/Sys/目录下面,对原有0.25版本的模块进行替换。
tar -zxvf Sys-Syscall-0.23
cp -f Sys-Syscall-0.23/lib/Sys/Syscall.pm /usr/local/share/perl5/Sys/通过如下命令查看模块版本是否已经替换为0.23版本:
grep -E "VERSION\s" /usr/local/share/perl5/Sys/Syscall.pm安装完毕之后,进入配置阶段。首先在两台机器上面创建/etc/mogilefs这个目录,用于存放配置文件。其次在两台机器上面创建mogilefs组和mogilefs用户,用于运行mogileFS相关的进程(笔者将上述用户和组创建为系统用户和系统组)。之后再在两台机器上面创建/mog_data目录,用于模拟storage的设备目录的父目录,并赋予mogilefs用户和mogilefs用户组权限。
mkdir /etc/mogilefs
groupadd -r mogilefs
useradd -r -g mogilefs -d /home/mogilefs -s /bin/bash -m mogilefs
mkdir /mog_data/
chown mogilefs.mogilefs /mog_data由于利用CPAN安装的MogileFS并未涉及到配置文件,因此这里笔者使用了GitHub上面提供的mogilefsd.conf文件和mogstored.conf文件作为MogileFS系统的配置文件。mogilefsd.conf文件作为tracker的配置,而mogstored.conf为storage进行配置。将上述两个文件下载,并拷贝到两台机器的/etc/mogilefs目录下面
cp mogilefsd.conf /etc/mogilefs
cp mogstored.conf /etc/mogilefsmogilefsd.conf配置文件的内容如下所示,其中几个比较重要的参数例如db_dsn用于定义database的IP地址(这里定义为笔者环境里面的192.168.11.130这台机器),db_user定义mogilefs进程连接数据库时所使用的用户名,db_pass定义mogilefs进程连接数据库时所使用的密码,listen定义tracker的监听套接字地址和端口:
# Enable daemon mode to work in background and use syslog
daemonize = 1
# Where to store the pid of the daemon (must be the same in the init script)
pidfile = /home/mogilefs/mogilefsd.pid
# Database connection information
db_dsn = DBI:mysql:mogilefs:host=192.168.11.130
db_user = mogile
db_pass = mogile
# IP:PORT to listen on for mogilefs client requests
listen = 0.0.0.0:7001
# Optional, if you don't define the port above.
conf_port = 7001
# Number of query workers to start by default.
query_jobs = 10
# Number of delete workers to start by default.
delete_jobs = 1
# Number of replicate workers to start by default.
replicate_jobs = 5
# Number of reaper workers to start by default.
# (you don't usually need to increase this)
reaper_jobs = 1
# Number of fsck workers to start by default.
# (these can cause a lot of load when fsck'ing)
#fsck_jobs = 1
# Minimum amount of space to reserve in megabytes
# default: 100
# Consider setting this to be larger than the largest file you
# would normally be uploading.
#min_free_space = 200
# Number of seconds to wait for a storage node to respond.
# default: 2
# Keep this low, so busy storage nodes are quickly ignored.
#node_timeout = 2
# Number of seconds to wait to connect to a storage node.
# default: 2
# Keep this low so overloaded nodes get skipped.
#conn_timeout = 2
# Allow replication to use the secondary node get port,
# if you have apache or similar configured for GET's
#repl_use_get_port = 1mogstored.conf配置文件的内容如下所示,定义最大连接数为10000,定义http接口以及管理接口的套接字端口,定义根目录(所有设备目录的父目录)为/mog_data。
maxconns = 10000
httplisten = 0.0.0.0:7500
mgmtlisten = 0.0.0.0:7501
docroot = /mog_data/之后需要在database里面创建mogilefs的连接用户。登录mysql节点,做如下配置。其中的pwd代表用户的登录密码。
GRANT ALL PRIVILEGES ON *.* TO root@'%' IDENTIFIED BY 'pwd' WITH GRANT OPTION
GRANT ALL PRIVILEGES ON mogilefs.* TO mogile@'%' IDENTIFIED BY 'pwd' WITH GRANT OPTION退出mysql命令行,在linux shell上面输入如下命令,用于初始化mogilefs数据库。
mogdbsetup --dbhost=192.168.11.130 --dbport=3306 --dbrootuser=root \
--dbrootpass=pwd --dbuser=mogile --dbpass=pwd初始化完毕之后,登录mysql,可以查看一下新建的数据库:
MariaDB [(none)]> show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| demo |
| mogilefs |
| mysql |
| performance_schema |
| solo |
| test |
| wp |
+--------------------+
8 rows in set (0.00 sec)
MariaDB [(none)]> use mogilefs
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A
Database changed
MariaDB [mogilefs]> show tables;
+----------------------+
| Tables_in_mogilefs |
+----------------------+
| checksum |
| class |
| device |
| domain |
| file |
| file_on |
| file_on_corrupt |
| file_to_delete |
| file_to_delete2 |
| file_to_delete_later |
| file_to_queue |
| file_to_replicate |
| fsck_log |
| host |
| server_settings |
| tempfile |
| unreachable_fids |
+----------------------+
17 rows in set (0.00 sec)初始化数据库完毕之后,我们需要提供让tracker和storage启动工作的启动脚本。其实,让tracker和storage工作的命令非常简单,如下所示:
启动tracker
su - mogilefs -c "/usr/local/bin/mogilefsd \
-c /etc/mogilefs/mogilefsd.conf --daemon"
启动storage
su - mogilefs -c "/usr/local/bin/mogstored \
-c $configfile --daemon"将上述命令封装为mogilefsd和mogstored启动脚本,内容如下所示:
针对该模块的mogilefs_methods指令,GET方法和DELETE方法经测试没有问题,但是PUT方法一直存在问题,开启nginx的debug模式,通过curl工具进行PUT操作时,在log里面存在upstream timed out (110: Connection timed out) while reading upstream错误,目前没有找到解决方法。
因此,笔者强烈建议,使用了mogilefs的nginx不要安装其他模块,而且该nginx只为了实现mogilefs的反向代理,不做其他功能。
将源码包下载之后,解压到任意路径(笔者这里将源码包解压到nginx1.12的源码目录里面)。在编译之前,先进入mogilefs的源码目录,修改ngx_http_mogilefs_modile.c源代码,找到ngx_http_mogilefs_create_spare_location函数,注释掉定义ngx_http_core_loc_conf_t的结构体指针*pclcf,注释掉pclcf指针的赋值语句。
修改完毕之后,可以进行编译了。在编译的时候,添加--add-module选项,指向mogilefs的源码目录即可。
编译安装Nginx完毕之后,在配置文件里面添加如下内容,笔者使用7010端口作为Nginx反向代理mogilefs的端口,并且定义一个trackers的upstream服务器组,使用nginx默认的wrr轮询调度算法。定义mogilefs的domain为之前创建的imgs,
upstream trackers {
server 192.168.11.130:7001;
server 192.168.11.132:7001;
}
server {
listen 7010;
location / {
mogilefs_tracker trackers;
mogilefs_domain imgs;
mogilefs_methods get delete;
mogilefs_pass {
proxy_pass $mogilefs_path;
proxy_hide_header Content-Type;
proxy_buffering off;
}
}
}编辑完毕之后,启动或者重启nginx进程,通过http://192.168.5.178:7010/1.jpg这个地址来访问之前上传的图片: