|
1、初识分布式文件系统 在生产环境中，后端存储服务器服务器是一个很重要的环节。如果一旦前段服务器需要大量进行IO操作，会对后端存储的文件系统造成很大的压力。一旦后端的存储文件系统down机，那会导致整个生产环境奔溃。因为在生产环境中，后端的存储文件系统会成为一个单点故障。而为了避免因为单点故障导致整个生产环境无法正常使用，所有大家都会采用分布式文件系统，以避免因为单点故障导致生产环境奔溃。 MogileFS由3个部分组成： (1) server：主要包括mogilefsd和mogstored两个应用程序。mogilefsd实现的是tracker，它通过数据库来保存元数据信息，包括站点domain、class、host等；mogstored是存储节点(store node)，它其实是个WebDAV服务，默认监听在7500端口，接受客户端的文件存储请求。在MogileFS安装完后，要运行mogadm工具将所有的store node注册到mogilefsd的数据库里，mogilefsd会对这些节点进行管理和监控。 (2) utils（工具集）：主要是MogileFS的一些管理工具，例如mogadm等。 (3) 客户端API：MogileFS的客户端API很多，例如Perl、PHP、Java、Python等，用这个模块可以编写客户端程序，实现文件的备份管理功能等。 存储主机(节点) 这个是 MogileFS 存储文件存放在这些机器上,也是 mogstored 节点,也叫 Storage Server,一台存储主要都要启动一个 mogstored 服务.扩容就是增加这些机器. 设备(device) 一个存储节点,以就是上面的主机,可以有多个 device, 就是用来存放文件的目录(例如挂载的目录),每个设备都有一个设备id,需要在 mogstored 的配置文件中的 docroot 配置的项目 指定的目录下面创建相应的设备的目录,目录名为 $docroot/dev$id,设备是不能删除的.只能将其设备的状态的值置为dead,当一个设备 dead 之后,就真的 dead了,里面的数据也无法恢复了,且这个dead了的设备的 id 也不能再用. 3、分布式文件系统需要的程序包 MogileFS-Server-2.46-2.el6.noarch.rpm MogileFS-Server-mogilefsd-2.46-2.el6.noarch.rpm MogileFS-Server-mogstored-2.46-2.el6.noarch.rpm MogileFS-Utils-2.19-1.el6.noarch.rpm Perlbal-1.78-1.el6.noarch.rpm Perlbal-doc-1.78-1.el6.noarch.rpm perl-Danga-Socket-1.61-1.el6.rf.noarch.rpm perl-IO-AIO-3.71-2.el6.x86_64 perl-MogileFS-Client-1.14-1.el6.noarch.rpm perl-Net-Netmask-1.9015-8.el6.noarch.rpm perl-Perlbal-1.78-1.el6.noarch.rpm [iyunv@mogilefs1 mogilefs]# yum install perl-IO-AIO perl-Net-Netmask perl-Sys-Syslog [iyunv@mogilefs1 mogilefs]# yum install MogileFS-Server-mogstored-2.46-2.el6.noarch.rpm perl-MogileFS-Client-1.14-1.el6.noarch.rpm MogileFS-Server-2.46-2.el6.noarch.rpm MogileFS-Utils-2.19-1.el6.noarch.rpm perl-Perlbal-1.78-1.el6.noarch.rpm MogileFS-Server-mogilefsd-2.46-2.el6.noarch.rpm perl-Danga-Socket-1.61-1.el6.rf.noarch.rpm 或者在安装包的目录下把所有rpm一次性安装 [iyunv@mogilefs1 mogilefs]# yum install *.rpm [iyunv@mysql1 ~]# mysql mysql> grant all on *.* to 'root'@'172.16.17.%' identified by 'xiaopeng'; Query OK, 0 rows affected (0.00 sec) mysql> create database mogilefs; Query OK, 1 row affected (0.03 sec) mysql> grant all on mogilefs.* to 'moguser'@'172.16.17.%' identified by 'mogpass'; Query OK, 0 rows affected (0.00 sec) mysql> flush privileges; Query OK, 0 rows affected (0.00 sec) [iyunv@mysql1 ~]# 6、数据库授权完成之后，接着去mogilefs文件系统的主控制台上，对数据库进行初始化。（建议此命令慎用，如果mogilefs文件系统在运行的时候，小心使用。） [iyunv@mogilefs1 mogilefs]# mogdbsetup --dbhost=172.16.17.144 --dbrootpass='xiaopeng' --dbuser='moguser' --dbpass='mogpass' This will attempt to setup or upgrade your MogileFS database. It won't destroy existing data. Run with --help for more information. Run with --yes to shut up these prompts. Continue? [N/y]: y 完成之后，可以去数据库确认初始化情况 [iyunv@mogilefs1 mogilefs]# mysql> use mogilefs Database changed mysql> show tables; +----------------------+ | Tables_in_mogilefs | +----------------------+ [iyunv@mogilefs1 ~]# vim /etc/mogilefs/mogilefsd.conf daemonize = 1 pidfile = /var/run/mogilefsd/mogilefsd.pid db_dsn = DBI:mysql:mogilefs:host=172.16.100.68 db_user = moguser db_pass = mogpass listen = 0.0.0.0:7001 conf_port = 7001 query_jobs = 5 delete_jobs = 1 replicate_jobs = 2 reaper_jobs = 1 - 主配置文件修改完成之后，接着所有的存储节点都需要进行一下的配置
[iyunv@mogilefs1 ~]# mkdir -pv /data/mogilefs/dev1 [iyunv@mogilefs1 ~]# chown -R mogilefs.mogilefs /data/mogilefs/ [iyunv@mogilefs1 ~]# vim /etc/mogilefs/mogstored.conf maxconns = 10000 httplisten = 0.0.0.0:7500 mgmtlisten = 0.0.0.0:7501 docroot = /data/mogilefs/ - 对于mogstord的启动文件（所有的节点都的mogstored的启动脚本文件需要加以下的几项）
为脚本添加： pidfile=/var/run/mogilefsd/mogstored.pid echo $(pidof mogstored) > ${pidfile} rm -f ${pidfile} - 所有的配置修改完成之后，接着去启动服务器，进行配置资源
[iyunv@mysql1 ~]# service mogstored start Starting mogstored [ OK ] [iyunv@mysql1 ~]# ss -tanlp [iyunv@mogilefs1 init.d]# service mogilefsd start Starting mogilefsd [ OK ] [iyunv@mogilefs1 ~]# mogadm check Checking trackers... 127.0.0.1:7001 ... OK Checking hosts... No devices found on tracker(s). [iyunv@mogilefs1 ~]# clear 追踪节点所能适应的管理命令 [iyunv@mogilefs1 ~]# mogadm --tracker=172.16.17.141:7001 mogadm check Check the state of the MogileFS world. mogadm stats Show MogileFS system statistics. mogadm host ... host add ... Add a host to MogileFS. host delete ... Delete a host. host list List all hosts. host mark ... Change the status of a host. (equivalent to 'modify --status') host modify ... Modify a host's properties. mogadm device ... device add ... Add a device to a host. device list ... List all devices, for each host. device mark ... Mark a device as {alive,dead,down,drain,readonly} device modify ... Modify a device's properties. device summary ... List the summary of devices, for each host. mogadm domain ... domain add ... Add a domain (namespace) domain delete ... Delete a domain. domain list List all hosts. mogadm class ... class add ... Add a file class to a domain. class delete ... Delete a file class from a domain. class list List all classes, for each domain. class modify ... Modify properties of a file class. mogadm slave ... slave add ... Add a slave node for store usage slave delete ... Delete a slave node for store usage slave list List current store slave nodes. slave modify ... Modify a slave node for store usage mogadm fsck ... fsck clearlog Clear the fsck log fsck printlog Display the fsck log fsck reset ... Reset fsck position back to the beginning fsck start Start (or resume) background fsck fsck status Show fsck status fsck stop Stop (pause) background fsck fsck taillog Tail the fsck log mogadm rebalance ... rebalance policy ... Add or adjust the current policy rebalance reset Reset an existing policy rebalance settings Display rebalance settings rebalance start Start a rebalance job rebalance status Show status of current rebalance job rebalance stop Stop a rebalance job rebalance test Show what devices the current policy would match mogadm settings ... settings list List all server settings settings set ... Set server setting 'key' to 'value'. - 首先需要把节点设备添加至管理设备中。使其能够正常管理
[iyunv@mogilefs1 ~]# mogadm --trackers=172.16.17.140:7001 host add 172.16.17.140 --ip=172.16.17.140 --status=alive [iyunv@mogilefs1 ~]# mogadm --trackers=172.16.17.140:7001 host add 172.16.17.141 --ip=172.16.17.141 --status=alive [iyunv@mogilefs1 ~]# mogadm --trackers=172.16.17.140:7001 host add 172.16.17.144 --ip=172.16.17.144 --status=alive [iyunv@mogilefs1 ~]# 添加完成之后，去确认设备添加情况 [iyunv@mogilefs1 ~]# mogadm host list 172.16.17.140 [1]: alive IP: 172.16.17.140:7500 172.16.17.141 [2]: alive IP: 172.16.17.141:7500 172.16.17.144 [3]: alive IP: 172.16.17.144:7500 12、后端存储的节点添加完成之后，接着去添加设备文件。 [iyunv@mogilefs1 ~]# mogadm device add 172.16.17.140 1 [iyunv@mogilefs1 ~]# mogadm device add 172.16.17.141 2 [iyunv@mogilefs1 ~]# mogadm device add 172.16.17.144 3 [iyunv@mogilefs1 ~]# mogadm device list 172.16.17.140 [1]: alive used(G) free(G) total(G) weight(%) dev1: alive 0.280 18.280 18.561 100 172.16.17.141 [2]: alive used(G) free(G) total(G) weight(%) dev2: alive 0.280 18.280 18.561 100 172.16.17.144 [3]: alive used(G) free(G) total(G) weight(%) dev3: alive 1.671 16.890 18.561 100 再次检查mogadm的状态信息 [iyunv@mogilefs1 ~]# mogadm check Checking trackers... 127.0.0.1:7001 ... OK Checking hosts... [ 1] 172.16.17.140 ... OK [ 2] 172.16.17.141 ... OK [ 3] 172.16.17.144 ... OK Checking devices... host device size(G) used(G) free(G) use% ob state I/O% ---- ------------ ---------- ---------- ---------- ------ ---------- ----- [ 1] dev1 18.561 0.281 18.280 1.51% writeable 0.0 [ 2] dev2 18.561 0.281 18.280 1.51% writeable 0.0 [ 3] dev3 18.561 1.671 16.890 9.00% writeable 1.7 ---- ------------ ---------- ---------- ---------- ------ total: 55.683 2.233 53.451 4.01% [iyunv@mogilefs1 ~]# [iyunv@mogilefs1 ~]# mogadm domain add files [iyunv@mogilefs1 ~]# mogadm domain add images [iyunv@mogilefs1 ~]# mogadm domain list domain class mindevcount replpolicy hashtype -------------------- -------------------- ------------- ------------ ------- files default 2 MultipleHosts() NONE images default 2 MultipleHosts() NONE 在创建domain的时候，根绝自己的需要进行创建。 - 创建好域名空间之后，如果需要有创建class类别。也能创建，这样能够分类别存储文件
基本使用方法 [iyunv@mogilefs1 ~]# mogadm class help mogadm class add <domain> <class> [opts] Add a file class to a domain. <class> Name of class to add. <domain> Domain to add class to. --hashtype=s Hash algorithm string ('MD5', 'NONE'). --mindevcount=i Minimum number of replicas. --replpolicy=s Replication policy string. 添加几个基本的类试试 [iyunv@mogilefs1 ~]# mogadm class add images jpg [iyunv@mogilefs1 ~]# mogadm class add images png [iyunv@mogilefs1 ~]# mogadm class add images gif [iyunv@mogilefs1 ~]# mogadm class add files text --mindevcount=1 [iyunv@mogilefs1 ~]# mogadm class add files html --replpolicy="MultipleHosts(3)" [iyunv@mogilefs1 ~]# mogadm class list domain class mindevcount replpolicy hashtype files default 2 MultipleHosts() NONE files html 2 MultipleHosts(3) NONE files plaintext 1 MultipleHosts() NONE
images default 2 MultipleHosts() NONE images gif 2 MultipleHosts() NONE images jpeg 2 MultipleHosts() NONE images png 2 MultipleHosts() NONE - 如果基本的属性定义完成之后，接着可以尝试去上传资源试试
[iyunv@mogilefs1 ~]# mogupload Usage: /usr/bin/mogupload --trackers=host --domain=foo --key='/hello.jpg' --file='./hello.jpg' [iyunv@mogilefs1 ~]# mogupload --trackers=172.16.17.140 --domain=files --key='/fstab.txt' --file='/etc/fstab' --class=plaintext 上传完成之后，确认上传情况。能够，会生成http连接。但是连接会比较特殊。 [iyunv@mogilefs1 ~]# mogfileinfo --trackers=172.16.17.140 --domain=files --key='/fstab.txt' - file: /fstab.txt class: plaintext devcount: 1 domain: files fid: 2 key: /fstab.txt length: 921 [iyunv@mogilefs1 ~]# [iyunv@mogilefs1 mogilefs]# mogupload --trackers=172.16.17.140 --domain=images --key='/1.jpg' --file='1.jpg' --class=jpeg [iyunv@mogilefs1 mogilefs]# mogfileinfo --trackers=172.16.17.140 --domain=images --key='/1.jpg' - file: /1.jpg class: jpeg devcount: 2 domain: images fid: 3 key: /1.jpg length: 170625
[iyunv@mogilefs1 mogilefs]# mogstats --db_dsn="DBI:mysql:mogilefs:host=172.16.17.144" --db_user="moguser" --db_pass="mogpass" --verbose --stats="all" Statistics for devices... device host files status ---------- ---------------- ------------ ---------- dev1 172.16.17.140 1 alive dev2 172.16.17.141 2 alive ---------- ---------------- ------------ ---------- Statistics for file ids... Max file id: 3 Statistics for files... domain class files size (m) fullsize (m) -------------------- ----------- ---------- ----------- ------------- files plaintext 1 0 0 images jpeg 1 0 0 -------------------- ----------- ---------- ----------- ------------- Statistics for replication... domain class devcount files -------------------- ----------- ---------- ---------- files plaintext 1 1 images jpeg 2 1 -------------------- ----------- ---------- ---------- 如果想要通过更加便捷的方式进行上传数据需要，程序员自己写API接口
前段基于nginx进行反代mogilefs文件系统 [iyunv@backproxy ~]# ll -rw-r--r-- 1 root root 832104 Dec 4 15:32 nginx-1.8.0.tar.gz -rw-r--r-- 1 root root 11208 Dec 4 18:54 nginx_mogilefs_module-1.0.4.tar.gz 需要注意的一点是，对于nginx而言，如果想实现对后端的mogilefs服务器的代理需要nginx_mogilefs_module模块。否则不能实现此功能；同时，还需要注意的一点是模板的版本需要严格要求，模块的版本信息要和nginx版本信息匹配否则会报内核溢出之类的错误 2015/12/04 18:36:42 [alert] 1880#0: worker process 1890 exited on signal 11 (core dumped)
2015/12/04 18:36:42 [alert] 1880#0: worker process 1892 exited on signal 11 (core dumped)
2015/12/04 18:36:42 [alert] 1880#0: worker process 1894 exited on signal 11 (core dumped)
2015/12/04 18:36:42 [alert] 1880#0: worker process 1896 exited on signal 11 (core dumped) [iyunv@backproxy ~]# yum groupinstall "Development Tools" "Server Platform Deveopment" [iyunv@backproxy ~]# yum install openssl-devel pcre-devel 还需要创建一个nginx用户 [iyunv@backproxy ~]# groupadd -r nginx [iyunv@backproxy ~]# useradd -r -g nginx nginx [iyunv@backproxy ~]# ./configure \ --prefix=/usr \ --sbin-path=/usr/sbin/nginx \ --conf-path=/etc/nginx/nginx.conf \ --error-log-path=/var/log/nginx/error.log \ --http-log-path=/var/log/nginx/access.log \ --pid-path=/var/run/nginx/nginx.pid \ --lock-path=/var/lock/nginx.lock \ --user=nginx \ --group=nginx \ --with-http_ssl_module \ --with-http_flv_module \ --with-http_stub_status_module \ --with-http_gzip_static_module \ --http-client-body-temp-path=/var/tmp/nginx/client/ \ --http-proxy-temp-path=/var/tmp/nginx/proxy/ \ --http-fastcgi-temp-path=/var/tmp/nginx/fcgi/ \ --http-uwsgi-temp-path=/var/tmp/nginx/uwsgi \ --http-scgi-temp-path=/var/tmp/nginx/scgi \ --with-pcre \ --with-debug \ --add-module= 这里需要特别注意的一项是，在编译安装nginx的时候，需要为期指定mogilefs的模块相对于的文件--add-module=../nginx_mogilefs_module-1.0.4 编译完成之后进行 Make && make install #!/bin/sh # nginx - this script starts and stops the nginx daemon # chkconfig: - 85 15 # description: Nginx is an HTTP(S) server, HTTP(S) reverse \ # proxy and IMAP/POP3 proxy server # processname: nginx # config: /etc/nginx/nginx.conf # config: /etc/sysconfig/nginx # pidfile: /var/run/nginx.pid # Source function library. . /etc/rc.d/init.d/functions # Source networking configuration. . /etc/sysconfig/network # Check that networking is up. [ "$NETWORKING" = "no" ] && exit 0 nginx="/usr/sbin/nginx" prog=$(basename $nginx)
NGINX_CONF_FILE="/etc/nginx/nginx.conf"
[ -f /etc/sysconfig/nginx ] && . /etc/sysconfig/nginx
lockfile=/var/lock/subsys/nginx
make_dirs() { # make required directories user=`nginx -V 2>&1 | grep "configure arguments:" | sed 's/[^*]*--user=\([^ ]*\).*/\1/g' -` options=`$nginx -V 2>&1 | grep 'configure arguments:'` for opt in $options; do if [ `echo $opt | grep '.*-temp-path'` ]; then value=`echo $opt | cut -d "=" -f 2` if [ ! -d "$value" ]; then # echo "creating" $value mkdir -p $value && chown -R $user $value fi fi done }
start() { [ -x $nginx ] || exit 5 [ -f $NGINX_CONF_FILE ] || exit 6 make_dirs echo -n $"Starting $prog: " daemon $nginx -c $NGINX_CONF_FILE retval=$? echo [ $retval -eq 0 ] && touch $lockfile return $retval }
stop() { echo -n $"Stopping $prog: " killproc $prog -QUIT retval=$? echo [ $retval -eq 0 ] && rm -f $lockfile return $retval }
restart() { configtest || return $? stop sleep 1 start }
reload() { configtest || return $? echo -n $"Reloading $prog: " killproc $nginx -HUP RETVAL=$? echo }
force_reload() { restart }
configtest() { $nginx -t -c $NGINX_CONF_FILE }
rh_status() { status $prog }
rh_status_q() { rh_status >/dev/null 2>&1 }
case "$1" in start) rh_status_q && exit 0 $1 ;; stop) rh_status_q || exit 0 $1 ;; restart|configtest) $1 ;; reload) rh_status_q || exit 7 $1 ;; force-reload) force_reload ;; status) rh_status ;; condrestart|try-restart) rh_status_q || exit 0 ;; *) echo $"Usage: $0 {start|stop|status|restart|condrestart|try-restart|reload|force-reload|configtest}" exit 2 esac 而后为此脚本赋予执行权限： [iyunv@backproxy ~]# chmod +x /etc/rc.d/init.d/nginx 添加至服务管理列表，并让其开机自动启动： [iyunv@backproxy ~]# chkconfig --add nginx [iyunv@backproxy ~]# chkconfig nginx on 而后就可以启动服务并测试了： [iyunv@backproxy ~]# service nginx start upstream trackers { server 172.16.17.140:7001 weight=1; server 172.16.17.141:7001 weight=1; server 172.16.17.144:7001 weight=1; } Server { location /images { mogilefs_tracker trackers; mogilefs_domain images; mogilefs_methods GET; mogilefs_noverify on; mogilefs_pass { proxy_pass $mogilefs_path; proxy_hide_header Content-Type; proxy_buffering off; } } location /files { mogilefs_tracker trackers; mogilefs_domain files; mogilefs_methods GET; mogilefs_noverify on; mogilefs_pass { proxy_pass $mogilefs_path; proxy_hide_header Content-Type; proxy_buffering off; } } } 查看反代日志 [iyunv@backproxy ~]# tail -2 /var/log/nginx/access.log 172.16.1.100 - - [04/Dec/2015:22:18:20 +0800] "GET /images/love.jpg HTTP/1.1" 304 0 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36" 172.16.1.100 - - [04/Dec/2015:22:18:20 +0800] "GET /images/love.jpg HTTP/1.1" 200 190166 "-" "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.36" [iyunv@backproxy ~]# [iyunv@mogilefs2 ~]# service mogstored stop Stop mogstored [ OK ] 对于前段访问后端存储服务器不会出现任何问题
注意点：对于这个版本的mogilefs分布式文件系统，在centos7上，只能存储一份，不能有其他多余的副本
| 
QQ群⑧：
窥视卡
雷达卡
发表于 2016-4-25 09:50:23
提升卡
置顶卡
沉默卡
喧嚣卡
变色卡
千斤顶
显身卡