分布式系统二、MogileFS
一、MogileFS简介MogileFS是一个开源的分布式文件存储系统，由LiveJournal旗下的Danga Interactive公司开发。Danga团队开发了包括 Memcached、MogileFS、Perlbal 等多个知名的开源项目。
目前使用MogileFS 的公司非常多，如日本排名先前的几个互联公司及国内的yupoo(又拍)、digg、豆瓣、1号店、大众点评、搜狗和安居客等，分别为所在的组织或公司管理着海量的图片。
1、Mogilefs特性
工作于应用层：http,nfs
无单点故障
自动完成文件复制
传输无需特殊协议
名称空间（完成复制）
不共享任何数据
2、MogileFS组件
tracker：追踪器，追踪元数据（不存储元数据）， http协议
mogilefsd(守护进程），它的主要职责包括：
replication：节点间文件的复制
deletion：删除文件
queryworker：响应客户请求的文件元数据访问请求
reaper：在存储失败后将文件复制请求重新放置于队列中
monitor：监测主机和设备的健康状态
database：存储元数据
一般使用mysql也可以使用pgsql，建议使用冗余方案以保证其可用性
mogilefs专门提供了数据管理（初始化）工具mogdbsetup;
storage：存储数据， http协议，nfs协议
mogstored(进程名），一个准备好的mogstored节点可通过mogadm命令添加至现在的集群中，存储节点需要定义“设备”用作存储空间，每个设备在当前集群中都需要通过唯一的DevID来标识
client：客户端用于与mogilefs建立通信，完成数据存取 #http协议
注意：
分布式系统应该具有机架感知能力，将冗余数据存放在不同机架上的服务器
MogileFS工作图解：
http://s4.运维网.com/wyfs02/M00/87/C9/wKioL1fh-KbgEF_MAAJQ7WH98PU084.png
3、常用术语
Domain：域name space命名空间 相当于一个目录
一个mogilefs可以有多个Dom，
用来存放不同文件，同一个Domain内key必须唯一，同mogilefs内，key可以相同
class：最小复制单元，文件属性管理
定义文件存储在不同设备上的份数
定位文件：Domain+fid（文件id）
二、安装Mogilefs
1、实验环境
CentOS6.5 x86_64 ,时间同步，epel源
node1：192.168.10.1 tracker，database
node3：192.168.10.3 storage
node4：192.168.10.4 storage
安装包介绍：
# ls mogilefs/
MogileFS-Server-2.46-2.el6.noarch.rpm #基础组件
Perlbal-doc-1.78-1.el6.noarch.rpm
MogileFS-Server-mogilefsd-2.46-2.el6.noarch.rpm #tracker组件
perl-MogileFS-Client-1.14-1.el6.noarch.rpm #客户端组件
MogileFS-Server-mogstored-2.46-2.el6.noarch.rpm #storage组件
perl-Net-Netmask-1.9015-8.el6.noarch.rpm #基础组件
MogileFS-Utils-2.19-1.el6.noarch.rpm #工具集
perl-Perlbal-1.78-1.el6.noarch.rpm #基础组件
Perlbal-1.78-1.el6.noarch.rpm
2、安装步骤
1）node4上安装mysql-server并配置，安装mogilefsd并配置，初始化数据库
# yum install mysql-server
#在/etc/my.cnf配置文件中段添加skip_name_resolve，跳过mysql对外部连接进行DNS解析
；在MySQL的授权表中就不能使用主机名了，只能使用IP
# yum install MogileFS-Server-mogilefsd-2.46-2.el6.noarch.rpmperl-Perlbal-1.78-1.el6.noarch.rpmperl-Net-Netmask-1.9015-8.el6.noarch.rpm
# rpm -ql MogileFS-Server-mogilefsd
/etc/mogilefs/mogilefsd.conf
/etc/rc.d/init.d/mogilefsd
/usr/bin/mogdbsetup
/usr/bin/mogilefsd
/usr/share/man/man1/mogilefsd.1.gz
/usr/share/man/man3/MogileFS::Checksum.3pm.gz
/usr/share/perl5/vendor_perl/MogileFS/Checksum.pm
/var/run/mogilefsd 数据库化数据库：
# mysql
mysql> GRANT ALL ON *.* TO 'root'@'192.168.10.%' IDENTIFIED BY 'magedu' WITH GRANT OPTION;
Query OK, 0 rows affected (0.00 sec)
mysql> FLUSH PRIVILEGES;
Query OK, 0 rows affected (0.00 sec)
# mogdbsetup --help
Usage: mogdbsetup
Options:
Default Description
============ ===========================================
--verbose Be verbose about what's happening.
--dbhost= localhost hostname or IP to database server.
--dbport= dbd defaultport number to database server.
--dbname= mogilefs database name to create/upgrade.
--dbrootuser= root Database administrator username.Only needed
for initial setup, not subsequent upgrades.
--dbrootpass= Database administrator password.Only needed
for initial setup, not subsequent upgrades.
--dbuser= mogile Regular database user to create and/or use
for MogileFS database.This is what the
mogilefsd trackers connect as.
--dbpass= You should change this, especially if your
database servers are accessible to other users
on the network.But they shouldn't be
if you're running MogileFS, because MogileFS
assumes your network is closed.
--type= MySQL Which MogileFS::Store implementation to use.
Available: MySQL, Postgres
--yes Run without questions.
# mogdbsetup --dbhost=192.168.10.1 --dbrootuser=root --dbrootpass=magedu --dbname=mogdb --dbuser=moguser --dbpass=mogpass
This will attempt to setup or upgrade your MogileFS database.
It won't destroy existing data.
Run with --help for more information.Run with --yes to shut up these prompts.
Continue? : y
Create/Upgrade database name 'mogdb'? : y
Grant all privileges to user 'moguser', connecting from anywhere, to the mogilefs database 'mogdb'? : y
mysql> SHOW DATABASES;
+--------------------+
| Database |
+--------------------+
| information_schema |
| mogdb |
| mysql |
| test |
+--------------------+
4 rows in set (0.00 sec)
mysql> 修改/etc/mogilefs/mogilefsd.conf：
# cd /etc/mogilefs/
# ls
mogilefsd.conf
# cp mogilefsd.conf mogilefsd.conf.bak
# ls
mogilefsd.confmogilefsd.conf.bak
# grep -v '^#\|^$' mogilefsd.conf #修改前的配置文件
daemonize = 1
pidfile = /var/run/mogilefsd/mogilefsd.pid
db_dsn = DBI:mysql:mogilefs:host=127.0.0.1
db_user = username
db_pass = password
listen = 127.0.0.1:7001
conf_port = 7001
query_jobs = 10
delete_jobs = 1
replicate_jobs = 5
reaper_jobs = 1
# grep -v '^#\|^$' mogilefsd.conf #修改后的配置文件
daemonize = 1
pidfile = /var/run/mogilefsd/mogilefsd.pid
db_dsn = DBI:mysql:mogdb:host=192.168.10.1
db_user = moguser
db_pass = mogpass
listen = 0.0.0.0:7001
conf_port = 7001
query_jobs = 10
delete_jobs = 3
replicate_jobs = 5
reaper_jobs = 1 2）node3和node4安装mogstored并配置，需要额外安装perl-IO-AIO，
这里node3和node4上的操作都是一样的
# yum install MogileFS-Server-mogstored-2.46-2.el6.noarch.rpmperl-Perlbal-1.78-1.el6.noarch.rpmperl-Net-Netmask-1.9015-8.el6.noarch.rpm
MogileFS-Server-2.46-2.el6.noarch.rpm
MogileFS-Server-mogilefsd-2.46-2.el6.noarch.rpm#被MogileFS-Server所依赖
# yum install perl-IO-AIO
# rpm -ql MogileFS-Server-mogstored
/etc/mogilefs/mogstored.conf
/etc/rc.d/init.d/mogstored
/usr/bin/mogautomount
/usr/bin/mogstored
/usr/share/man/man1/mogautomount.1.gz
/usr/share/man/man1/mogstored.1.gz
/usr/share/perl5/vendor_perl/Mogstored/ChildProcess 创建一个新的分区用来做mogstored：
# df -TH
Filesystem Type SizeUsed Avail Use% Mounted on
/dev/sda2 ext4 20G2.3G 16G13% /
tmpfs tmpfs254M 0254M 0% /dev/shm
/dev/sda1 ext4 508M 34M448M 7% /boot
# fdisk -l
Disk /dev/sda: 42.9 GB, 42949672960 bytes
255 heads, 63 sectors/track, 5221 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00037190
Device Boot Start End Blocks IdSystem
/dev/sda1 * 1 64 512000 83Linux
Partition 1 does not end on cylinder boundary.
/dev/sda2 64 2418 18902016 83Linux
/dev/sda3 2418 2673 2048000 82Linux swap / Solaris
# cat /proc/partitions
major minor#blocksname
8 0 41943040 sda
8 1 512000 sda1
8 2 18902016 sda2
8 3 2048000 sda3
# fdisk /dev/sda
WARNING: DOS-compatible mode is deprecated. It's strongly recommended to
switch off the mode (command 'c') and change display units to
sectors (command 'u').
Command (m for help): n
Command action
e extended
p primary partition (1-4)
e
Selected partition 4
First cylinder (2673-5221, default 2673):
Using default value 2673
Last cylinder, +cylinders or +size{K,M,G} (2673-5221, default 5221): +10G
Command (m for help): w
The partition table has been altered!
Calling ioctl() to re-read partition table.
WARNING: Re-reading the partition table failed with error 16: Device or resource busy.
The kernel still uses the old table. The new table will be used at
the next reboot or after you run partprobe(8) or kpartx(8)
Syncing disks.
# partx -a /dev/sda
BLKPG: Device or resource busy
error adding partition 1
BLKPG: Device or resource busy
error adding partition 2
BLKPG: Device or resource busy
error adding partition 3
# partx -a /dev/sda
BLKPG: Device or resource busy
error adding partition 1
BLKPG: Device or resource busy
error adding partition 2
BLKPG: Device or resource busy
error adding partition 3
BLKPG: Device or resource busy
error adding partition 4
# cat /proc/partitions
major minor#blocksname
8 0 41943040 sda
8 1 512000 sda1
8 2 18902016 sda2
8 3 2048000 sda3
8 4 10490245 sda4
# mke2fs -t ext4 /dev/sda4 #格式化
# mkdir /mogilefs
# chown mogilefs. -R /mogilefs
# vi /etc/fstab #开机自动挂载
# cat /etc/fstab
#
# /etc/fstab
# Created by anaconda on Fri Aug5 17:32:21 2016
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
UUID=e0c0bc25-62e5-4896-8749-fce761bc3af7 / ext4 defaults 1 1
UUID=970e070d-b49f-439b-8bbb-11ef49ee95ce /boot ext4 defaults 1 2
UUID=7e6bc97d-be6d-4ac9-bc26-8be3d80e715d swap swap defaults 0 0
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devptsgid=5,mode=6200 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
/dev/sda4/mogstoreext4defaults0 0
# mount -a
# mount
/dev/sda2 on / type ext4 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw)
/dev/sda1 on /boot type ext4 (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
/dev/sda4 on /mogstore type ext4 (rw)
# ls /mogstore
lost+found
# df -TH
Filesystem Type SizeUsed Avail Use% Mounted on
/dev/sda2 ext4 20G2.6G 16G14% /
tmpfs tmpfs254M 0254M 0% /dev/shm
/dev/sda1 ext4 508M 34M448M 7% /boot
/dev/sda4 ext4 11G158M9.9G 2% /mogstore
# cd /mogstore/
# ls
lost+found
# mkdir dev1 #新建一个设备dev1，
# ls
dev1lost+found
# cd /mogstore/
# ls
lost+found
# mkdir dev2
# ls
dev2lost+found 修改配置文件：/etc/mogilefs/mogstored.conf
# cat /etc/mogilefs/mogstored.conf
maxconns = 10000
httplisten = 0.0.0.0:7500
mgmtlisten = 0.0.0.0:7501
docroot = /mogstore 启动mogstored：
# service mogstored start
Starting mogstored 3）node1上安装客户端工具
# yum install perl-MogileFS-Client-1.14-1.el6.noarch.rpm#被MogileFS-Utils所依赖
MogileFS-Utils-2.19-1.el6.noarch.rpm
# rpm -ql MogileFS-Utils
/usr/bin/mogadm #管理工具
/usr/bin/mogdelete
/usr/bin/mogfetch
/usr/bin/mogfiledebug
/usr/bin/mogfileinfo
/usr/bin/moglistfids
/usr/bin/moglistkeys
/usr/bin/mogrename
/usr/bin/mogstats
/usr/bin/mogtool
/usr/bin/mogupload
/usr/share/man/man1/mogadm.1.gz
# mogadm --trackers=192.168.10.1:7001 check#检查状态，服务器是本机可以省略--trackers
Checking trackers...
192.168.10.1:7001 ... OK
Checking hosts...
No devices found on tracker(s). #没有设备
# mogadm --trackers=192.168.10.1:7001 stats
mogadm stats is deprecated by new 'mogstats' utility
# mogadm stats
mogadm stats is deprecated by new 'mogstats' utility
# mogstats
Fetching statistics... (all)
Can't connect to data source '' because I can't work out what driver to use (it doesn't seem to contain a 'dbi:driver:' prefix and the DBI_DRIVER env var is not set) at /usr/bin/mogstats line 312
# mogstats --help
Usage:
mogstats --db_dsn="DBI:mysql:mfs（库名）:host=mfshost" --db_user="mfs"
--db_pass="mfs" --verbose --stats="devices,files"
mogstats --stats="all"
mogstats
valid stats: all, delete-queue, devices, domains, fids, files, general-queues, replication,
# mogstats --db_dsn="DBI:mysql:mogdb:host=192.168.10.1" --db_user="moguser" --db_pass="mogpass" --stats="all"
Fetching statistics... (all)
Statistics for devices...
device host files status
---------- ---------------- ------------ ----------
---------- ---------------- ------------ ----------
Statistics for file ids...
Max file id: none
Statistics for files...
domain class files size (m)fullsize (m)
-------------------- ----------- ---------- ----------- -------------
-------------------- ----------- ---------- ----------- -------------
Statistics for replication...
domain class devcount files
-------------------- ----------- ---------- ----------
-------------------- ----------- ---------- ----------
Statistics for replication queue...
status count
-------------------- ------------
-------------------- ------------
Statistics for delete queue...
status count
-------------------- ------------
-------------------- ------------
Statistics for general queues...
queue status count
--------------- -------------------- ------------
--------------- -------------------- ------------
done
# mogadm host list
# mogadm host
Help for 'host' command:
(enter any command prefix, leaving off options, for further help)
mogadm host add Add a host to MogileFS.
mogadm host delete Delete a host.
mogadm host list List all hosts.
mogadm host mark Change the status of a host.(equivalent to 'modify --status')
mogadm host modify Modify a host's properties.
# mogadm host add
ERROR: Missing argument 'hostname'
Help for 'host-add' command:
mogadm host add Add a host to MogileFS.
Hostname of machine
--altip=s Alternate IP that is machine is reachable from
--altmask=s Netmask which, when matches client, uses alt IP
--getport=i Alternate HTTP port serving readonly traffic
--ip=s IP address of machine.
--port=i HTTP port of mogstored
--status=s One of {alive,down}.Default 'down'.
# mogadm host add 192.168.10.3 --ip=192.168.10.3#添加主机
# mogadm host list
192.168.10.3 : down
IP: 192.168.10.3:7500
# mogadm host mark 192.168.10.3 alive
# mogadm host list
192.168.10.3 : alive
IP: 192.168.10.3:7500
# mogadm host add 192.168.10.4 --ip=192.168.10.4 --status=alive
# mogadm host list
192.168.10.3 : alive
IP: 192.168.10.3:7500
192.168.10.4 : alive
IP: 192.168.10.4:7500
# mogadm device add 192.168.10.3 1 #添加设备，需要在相应的节点目录下创建目录
# mogadm device add 192.168.10.3 2
# mogadm device list
192.168.10.3 : alive
used(G) free(G) total(G)weight(%)
dev1: alive 0.146 9.200 9.347 100
192.168.10.4 : alive
used(G) free(G) total(G)weight(%)
dev2: alive 0.146 9.200 9.347 100
# mogadm domain add images #添加域
# mogadm domain add confiles
# mogadm domain list
domain class mindevcount replpolicy hashtype
-------------------- -------------------- ------------- ------------ -------
confiles default 2 MultipleHosts() NONE
images default 2 MultipleHosts() NONE
# mogadm class add images nc #添加类
# mogadm class list
domain class mindevcount replpolicy hashtype
-------------------- -------------------- ------------- ------------ -------
confiles default 2 MultipleHosts() NONE
images default 2 MultipleHosts() NONE
images nc 2 MultipleHosts() NONE mindevcount：最小复制文件的份数
replpolicy ：复制类型
hashtype：采用的hash的类型
MogileFS就安装配置完成了，就可以通过调用其API开始使用了，就是开发的事咯；
我们这里就使用mogupload命令上传文件测试一下：
# mogupload
Usage: /usr/bin/mogupload --trackers=host --domain=foo --key='/hello.jpg' --file='./hello.jpg'
# mogupload --trackers=192.168.10.1 --domain=confiles --key='/1.conf' --file="/etc/fstab" #key上传后的访问路径，file是要上传的文件现在的路径
# mogfileinfo --trackers=192.168.10.1 --domain=confiles --key="/1.conf"
- file: /1.conf #查看文件存储的信息
class: default
devcount: 2
domain: confiles
fid: 2
key: /1.conf
length: 805
- http://192.168.10.3:7500/dev1/0/000/000/0000000002.fid
- http://192.168.10.4:7500/dev2/0/000/000/0000000002.fid 验证：http://s1.运维网.com/wyfs02/M00/87/D8/wKiom1fiq6STmpEHAACHvyoO_ps487.png
mogfetch 下载文件：
# mogfetch
Usage: /usr/bin/mogfetch --trackers=host --domain=foo --key='/hello.jpg' --file='./output'
# mogfetch --trackers=192.168.10.1 --domain=confiles --key="/1.conf" --file=/root/1.confs
三、使用Nginx实现代理访问MOgileFS
1、编译安装Nginx
# tar xf nginx-1.8.0.tar.gz -C /usr/src
# tar xf nginx_mogilefs_module-1.0.4.tar.gz-C /usr/src
# cd nginx-1.8.0/
# useraddnginx
#./configure \
--prefix=/usr \
--sbin-path=/usr/sbin/nginx \
--conf-path=/etc/nginx/nginx.conf \
--error-log-path=/var/log/nginx/error.log \
--http-log-path=/var/log/nginx/access.log \
--pid-path=/var/run/nginx/nginx.pid\
--lock-path=/var/lock/nginx.lock \
--user=nginx \
--group=nginx \
--with-http_ssl_module \
--with-http_flv_module \
--with-http_stub_status_module \
--with-http_gzip_static_module \
--http-client-body-temp-path=/var/tmp/nginx/client/ \
--http-proxy-temp-path=/var/tmp/nginx/proxy/ \
--http-fastcgi-temp-path=/var/tmp/nginx/fcgi/ \
--http-uwsgi-temp-path=/var/tmp/nginx/uwsgi \
--http-scgi-temp-path=/var/tmp/nginx/scgi \
--with-pcre \
--with-debug \
--add-module=..//nginx_mogilefs_module-1.0.4/ #添加mogilefs模块
# make;make install
nginx的mogilefs的第三方模块使用地址：http://www.grid.net.ru/nginx/mogilefs.en.html
2、添加服务脚本
http://www.178linux.com/wp-content/plugins/ueditor/ueditor/dialogs/attachment/fileTypeImages/icon_rar.gifnginx.zip
3、编辑nginx配置文件，使其支持mogilefs
# vim /etc/nginx/nginx.conf
location /confiles/ {
mogilefs_tracker 192.168.10.1:7001;
mogilefs_domainconfiles;
mogilefs_pass {
proxy_pass $mogilefs_path;
proxy_hide_header Content-Type;
proxy_buffering off;
}
} 4、测试nginx配置语法，并启动nginx
# nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
# /etc/init.d/nginx start 5、打开浏览器访问测试
http://s2.运维网.com/wyfs02/M02/87/DA/wKioL1fjf5XTkbWeAABSfCcaTl0601.png
页:
[1]