mogilefs安装配置详解
一、MogileFS介绍MogileFS是一个开源的分布式文件存储系统，由LiveJournal旗下的Danga Interactive公司开发。Danga团队开发了包括 Memcached、MogileFS、Perlbal 等多个知名的开源项目。目前使用MogileFS 的公司非常多，如日本排名先前的几个互联公司及国内的yupoo(又拍)、digg、豆瓣、1号店、大众点评、搜狗和安居客等，分别为所在的组织或公司管理着海量的图片。
MogileFS由3个部分组成：
(1) server：主要包括mogilefsd和mogstored两个应用程序。mogilefsd实现的是tracker，它通过数据库来保存元数据信息，包括站点domain、class、host等；mogstored是存储节点(store node)，它其实是个WebDAV服务，默认监听在7500端口，接受客户端的文件存储请求。在MogileFS安装完后，要运行mogadm工具将所有的store node注册到mogilefsd的数据库里，mogilefsd会对这些节点进行管理和监控。
(2) utils（工具集）：主要是MogileFS的一些管理工具，例如mogadm等。
(3) 客户端API：MogileFS的客户端API很多，例如Perl、PHP、Java、Python等，用这个模块可以编写客户端程序，实现文件的备份管理功能等。
存储主机(节点)
这个是 MogileFS 存储文件存放在这些机器上,也是 mogstored 节点,也叫 Storage Server,一台存储主要都要启动一个 mogstored 服务.扩容就是增加这些机器.
设备(device)
一个存储节点,以就是上面的主机,可以有多个 device, 就是用来存放文件的目录(例如挂载的目录),每个设备都有一个设备id,需要在 mogstored 的配置文件中的 docroot 配置的项目 指定的目录下面创建相应的设备的目录,目录名为 $docroot/dev$id,设备是不能删除的.只能将其设备的状态的值置为dead,当一个设备 dead 之后,就真的 dead了,里面的数据也无法恢复了,且这个dead了的设备的 id 也不能再用.
二、扑拓图
10.1.2.180为tracker节点，并且首先安装mysql服务
10.1.2.200-202,总共3台，作为mogstored存储节点
三、180部署步聚
1、安装相关包
# ll
总用量 620
-rw-r--r--. 1 root root 1916 6月 3 17:03 MogileFS-Server-2.46-2.el6.noarch.rpm
-rw-r--r--. 1 root root 176308 6月 3 17:03 MogileFS-Server-mogilefsd-2.46-2.el6.noarch.rpm
-rw-r--r--. 1 root root26872 6月 3 17:03 MogileFS-Server-mogstored-2.46-2.el6.noarch.rpm
-rw-r--r--. 1 root root75916 6月 3 17:49 MogileFS-Utils-2.19-1.el6.noarch.rpm
-rw-r--r--. 1 root root 5880 6月 3 18:04 Perlbal-1.78-1.el6.noarch.rpm
-rw-r--r--. 1 root root 1624 6月 3 18:04 Perlbal-doc-1.78-1.el6.noarch.rpm
-rw-r--r--. 1 root root30312 6月 3 17:49 perl-MogileFS-Client-1.14-1.el6.noarch.rpm
-rw-r--r--. 1 root root25140 6月 3 17:49 perl-Net-Netmask-1.9015-8.el6.noarch.rpm
-rw-r--r--. 1 root root 268620 6月 3 17:50 perl-Perlbal-1.78-1.el6.noarch.rpm# yum install *.rpm
Dependencies Resolved
================================================================================
Package Arch Version Repository Size
================================================================================
Installing:
MogileFS-Server noarch 2.46-2.el6 /MogileFS-Server-2.46-2.el6.noarch 0.0
MogileFS-Server-mogilefsd
noarch 2.46-2.el6 /MogileFS-Server-mogilefsd-2.46-2.el6.noarch
523 k
MogileFS-Server-mogstored
noarch 2.46-2.el6 /MogileFS-Server-mogstored-2.46-2.el6.noarch
53 k
MogileFS-Utils noarch 2.19-1.el6 /MogileFS-Utils-2.19-1.el6.noarch170 k
Perlbal noarch 1.78-1.el6 /Perlbal-1.78-1.el6.noarch 4.4 k
Perlbal-doc noarch 1.78-1.el6 /Perlbal-doc-1.78-1.el6.noarch 0.0
perl-MogileFS-Client
noarch 1.14-1.el6 /perl-MogileFS-Client-1.14-1.el6.noarch
79 k
perl-Net-Netmasknoarch 1.9015-8.el6 /perl-Net-Netmask-1.9015-8.el6.noarch
55 k
perl-Perlbal noarch 1.78-1.el6 /perl-Perlbal-1.78-1.el6.noarch 666 k
Installing for dependencies:
perl-BSD-Resource x86_64 1.29.03-3.el6
base 35 k
perl-DBD-MySQL x86_64 4.013-3.el6base 134 k
perl-DBI x86_64 1.609-4.el6base 705 k
perl-Danga-Socket noarch 1.61-5.el6 epel 28 k
perl-IO-stringy noarch 2.110-10.1.el6
base 68 k
perl-Sys-Syscallnoarch 0.23-1.el6 epel 14 k
perl-Time-HiRes x86_64 4:1.9721-141.el6
base 49 k
Transaction Summary 配置文件
# cd /etc/mogilefs/
# ls
mogilefsd.confmogstored.conf 查看安装哪些包
# rpm -qa | grep -i mogilefs
MogileFS-Server-mogstored-2.46-2.el6.noarch
MogileFS-Server-mogilefsd-2.46-2.el6.noarch
perl-MogileFS-Client-1.14-1.el6.noarch
MogileFS-Server-2.46-2.el6.noarch
MogileFS-Utils-2.19-1.el6.noarch 2、安装MariaDB
# vim /etc/yum.repos.d/MariaDB.repo
name = MariaDB
baseurl = http://yum.mariadb.org/5.5/centos6-amd64
gpkgkey=https://yum.mariadb.org/RPM-GPG-KEY-MariaDB
gpgcheck=1 手动导入MariaDB的签署密钥
# rpm --import https://yum.mariadb.org/RPM-GPG-KEY-MariaDB 开始yum安装并启动
# yum install MariaDB-server MariaDB-client# /etc/init.d/mysql start 3、 启动MogileFS前，设定数据库
3.1在数据前授权
MariaDB > grant all on mogilefs.* to moguser@'10.1.%.%' indentified by 'mog168'; 3.2设定数据库
# mogdbsetup --dbhost=10.1.2.180 --dbport=3306 --dbname=mogilefs --dbrootuser=root --dbrootpass=168 --dbuser=moguser --dbpass=mog168
This will attempt to setup or upgrade your MogileFS database.
It won't destroy existing data.
Run with --help for more information.Run with --yes to shut up these prompts.
Continue? : y
Create/Upgrade database name 'mogilefs'? : y
Grant all privileges to user 'moguser', connecting from anywhere, to the mogilefs database 'mogilefs'? : y
Failed to grant privileges: Access denied for user 'root'@'10.1.%.%' to database 'mogilefs' 注意有错提示Failed to grant privileges: Access denied for user 'root'@'10.1.%.%' to database 'mogilefs'
3.3 去数据库查看，已经创建了，但没有tables
MariaDB > show databases;
+--------------------+
| Database |
+--------------------+
| information_schema |
| mogilefs |
| mysql |
| performance_schema |
| test |
MariaDB > use mogilefs;
Database changed
MariaDB > show tables;
Empty set (0.00 sec)
3.4手动创建用户moguser，并授权mogilefs的所有权限
MariaDB > grant all on mogilefs.* to moguser@'10.1.%.%' identified by 'mog168';
Query OK, 0 rows affected (0.00 sec)
MariaDB > flush privileges;
Query OK, 0 rows affected (0.00 sec)
3.5再返回3.2重新执行
# mogdbsetup --dbhost=10.1.2.180 --dbport=3306 --dbname=mogilefs --dbrootuser=root --dbrootpass=168 --dbuser=moguser --dbpass=mog168
This will attempt to setup or upgrade your MogileFS database.
It won't destroy existing data.
Run with --help for more information.Run with --yes to shut up these prompts.
Continue? : y
Create/Upgrade database name 'mogilefs'? : y
Grant all privileges to user 'moguser', connecting from anywhere, to the mogilefs database 'mogilefs'? : y
Failed to grant privileges: Access denied for user 'root'@'10.1.%.%' to database 'mogilefs'
# mogdbsetup --dbhost=10.1.2.180 --dbport=3306 --dbname=mogilefs --dbrootuser=root --dbrootpass=168 --dbuser=moguser --dbpass=mog168
This will attempt to setup or upgrade your MogileFS database.
It won't destroy existing data.
Run with --help for more information.Run with --yes to shut up these prompts.
Continue? : y
3.6去数库查看
MariaDB > show tables;
+----------------------+
| Tables_in_mogilefs |
+----------------------+
| checksum |
| class |
| device |
| domain |
| file |
| file_on |
| file_on_corrupt |
| file_to_delete |
| file_to_delete2 |
| file_to_delete_later |
| file_to_queue |
| file_to_replicate |
| fsck_log |
| host |
| server_settings |
| tempfile |
| unreachable_fids |
+----------------------+
17 rows in set (0.00 sec)
4、修改主配置文件
# vim /etc/mogilefs/mogilefsd.conf
# Enable daemon mode to work in background and use syslog
daemonize = 1
# Where to store the pid of the daemon (must be the same in the init script)
pidfile = /var/run/mogilefsd/mogilefsd.pid
# Database connection information
db_dsn = DBI:mysql:mogilefs:host=10.1.2.180
db_user = moguser
db_pass = mog168
# IP:PORT to listen on for mogilefs client requests
listen = 10.1.2.180:7001
# Optional, if you don't define the port above.
conf_port = 7001
# Number of query workers to start by default.
query_jobs = 10
# Number of delete workers to start by default.
delete_jobs = 1
# Number of replicate workers to start by default.
replicate_jobs = 5
# Number of reaper workers to start by default.
# (you don't usually need to increase this)
reaper_jobs = 1
# Number of fsck workers to start by default.
# (these can cause a lot of load when fsck'ing)
#fsck_jobs = 1
# Minimum amount of space to reserve in megabytes
# default: 100
# Consider setting this to be larger than the largest file you
# would normally be uploading.
#min_free_space = 200
# Number of seconds to wait for a storage node to respond.
# default: 2
# Keep this low, so busy storage nodes are quickly ignored.
#node_timeout = 2
# Number of seconds to wait to connect to a storage node.
# default: 2
# Keep this low so overloaded nodes get skipped.
#conn_timeout = 2
# Allow replication to use the secondary node get port,
# if you have apache or similar configured for GET's
#repl_use_get_port = 1 5、确保pid文件属主，属组为mogilefs
# ls -ld /var/run/mogilefsd
drwxr-xr-x. 2 mogilefs mogilefs 4096 10月8 2013 /var/run/mogilefsd 6、启动mogilefsd并确定监听端口为7001
# /etc/init.d/mogilefsd start
Starting mogilefsd [确定]# netstat -tlnp | grep 7001 四、mogstored 节点
1、200的主机，同样安装这几个rpn包
MogileFS-Server-mogstored-2.46-2.el6.noarch
MogileFS-Server-mogilefsd-2.46-2.el6.noarch
perl-MogileFS-Client-1.14-1.el6.noarch
MogileFS-Server-2.46-2.el6.noarch
MogileFS-Utils-2.19-1.el6.noarch
2、在/data0下创建数据存放位置，并把属主，属组设好
# mkdir nfs/mogdata -pv# chown -R mogilefs.mogilefs nfs/ 3、编辑配置文件
# vim /etc/mogilefs/mogstored.conf
maxconns = 10000
httplisten = 0.0.0.0:7500
mgmtlisten = 0.0.0.0:7501
docroot = /data0/nfs/mogdata 4、启动mogstored，查看监听端口
# /etc/init.d/mogstored start
Starting mogstored [确定]# netstat -tlnp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:111 0.0.0.0:* LISTEN 1227/rpcbind
tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 1436/sshd
tcp 0 0 127.0.0.1:631 0.0.0.0:* LISTEN 1312/cupsd
tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN 1553/master
tcp 0 0 0.0.0.0:55006 0.0.0.0:* LISTEN 1245/rpc.statd
tcp 0 0 :::111 :::* LISTEN 1227/rpcbind
tcp 0 0 :::22 :::* LISTEN 1436/sshd
tcp 0 0 ::1:631 :::* LISTEN 1312/cupsd
tcp 0 0 ::1:25 :::* LISTEN 1553/master
tcp 0 0 :::38585 :::* LISTEN 1245/rpc.statd 5、找不到监听端口，没起来
# su mogilefs
bash-4.1$ mog
mogadm mogdbsetup mogfetch mogfileinfo moglistfids mogrename
mogautomountmogdelete mogfiledebugmogilefsd moglistkeys mogstats
bash-4.1$ mogstored -c /etc/mogilefs/mogstored.conf
Running.
ERROR: IO::AIO not installed, so async IO not available.Refusing to run
unless you set the environment variable MOGSTORED_RUN_WITHOUT_AIO=1
6、从上面错误显示，依赖关系IO-AIO没解决好
# yum -y install perl-IO-AIO
7、安装，重新启动，查看监听端口
# /etc/init.d/mogstored start
Starting mogstored [确定]
# ss -tln | grep -E '(7500|7501)'
LISTEN 0 128 *:7500 *:*
LISTEN 0 128 *:7501 *:*
五、180主机测试
1、用mogadm
# mogadm -h
Usage:(enter any command prefix, leaving off options, for further help)
mogadm check Check the state of the MogileFS world.
mogadm stats Show MogileFS system statistics.(DEPRECATED: use mogstats instead)
mogadm host ...
host add ... Add a host to MogileFS.
host delete ... Delete a host.
host list List all hosts.
host mark ... Change the status of a host.(equivalent to 'modify --status')
host modify ... Modify a host's properties.
mogadm device ...
device add ... Add a device to a host.
device list ... List all devices, for each host.
device mark ... Mark a device as {alive,dead,down,drain,readonly}
device modify ... Modify a device's properties.
device summary ... List the summary of devices, for each host.
mogadm domain ...
domain add ... Add a domain (namespace)
domain delete ... Delete a domain.
domain list List all hosts.
mogadm class ...
class add ... Add a file class to a domain.
class delete ... Delete a file class from a domain.
class list List all classes, for each domain.
class modify ... Modify properties of a file class.
mogadm slave ...
slave add ... Add a slave node for store usage
slave delete ... Delete a slave node for store usage
slave list List current store slave nodes.
slave modify ... Modify a slave node for store usage
mogadm fsck ...
fsck clearlog Clear the fsck log
fsck printlog Display the fsck log
fsck reset ... Reset fsck position back to the beginning
fsck start Start (or resume) background fsck
fsck status Show fsck status
fsck stop Stop (pause) background fsck
fsck taillog Tail the fsck log
mogadm rebalance ...
rebalance policy ... Add or adjust the current policy
rebalance reset Reset an existing policy
rebalance settings Display rebalance settings
rebalance start Start a rebalance job
rebalance status Show status of current rebalance job
rebalance stop Stop a rebalance job
rebalance test Show what devices the current policy would match
mogadm settings ...
settings list List all server settings
settings set ... Set server setting 'key' to 'value'.
#mogadm --trackers=10.1.2.180:7001 check
Checking trackers...
10.1.2.180:7001 ... OK
Checking hosts...
No devices found on tracker(s).
2、
# mogstats -h
Usage:
mogstats --db_dsn="DBI:mysql:mfs:host=mfshost" --db_user="mfs"
--db_pass="mfs" --verbose --stats="devices,files"
mogstats --stats="all"
mogstats
valid stats: all, delete-queue, devices, domains, fids, files, general-queues, replication, replication-queue
3、没有添加进来，查看主机是为空的
#mogadm --trackers=10.1.2.180:7001 host list
3.1查看帮助添加主机
#mogadm --trackers=10.1.2.180:7001 host add
ERROR: Missing argument 'hostname'
Help for 'host-add' command:
mogadm host add Add a host to MogileFS.
Hostname of machine
--altip=s Alternate IP that is machine is reachable from
--altmask=s Netmask which, when matches client, uses alt IP
--getport=i Alternate HTTP port serving readonly traffic
--ip=s IP address of machine.
--port=i HTTP port of mogstored
--status=s One of {alive,down}.Default 'down'.
3.2添加主机10.1.2.200
#mogadm --trackers=10.1.2.180:7001 host add 10.1.2.200 --ip=10.1.2.200 --status=alive
3.3在查看主机列表
#mogadm --trackers=10.1.2.180:7001 host list
10.1.2.200 : alive
IP: 10.1.2.200:7500
4、查看帮助和添加设备列表
4.1查看
#mogadm --trackers=10.1.2.180:7001 device list
10.1.2.200 : alive
used(G) free(G) total(G)weight(%)
#mogadm --trackers=10.1.2.180:7001 device add
ERROR: Missing argument 'hostname'
Help for 'device-add' command:
mogadm device add Add a device to a host.
Numeric devid.Never reuse these.
Hostname to add a device
--status=s One of 'alive' or 'down'.Defaults to 'alive'.
4.2添加设备，ID号随便取，由于--status默认是alive，可以不指定了
#mogadm --trackers=10.1.2.180:7001 device add 10.1.2.200 001
4.3查看
#mogadm --trackers=10.1.2.180:7001 device list
10.1.2.200 : alive
used(G) free(G) total(G)weight(%)
dev1: alive 0.000 0.000 0.000 100
5、创建domain为files（名称随取）
#mogadm --trackers=10.1.2.180:7001 domain add files
#mogadm --trackers=10.1.2.180:7001 domain list
domain class mindevcount replpolicy hashtype
-------------------- -------------------- ------------- ------------ -------
files default 2 MultipleHosts() NONE
六、在201上同样安装这些rpm包
1、
# yum install *.rpm perl-IO-AIO -y
2、
# vim /etc/mogilefs/mogstored.conf
maxconns = 10000
httplisten = 0.0.0.0:7500
mgmtlisten = 0.0.0.0:7501
docroot = /nfs/mogdata
~
3、
# mkdir /nfs/mogdata -pv# chown -R moigilefs.mogilefs /nfs/mogdata/
4、
# service mogstored start
七、回到180的主机，把10.1.2.201 也添加进去
1、
#mogadm --trackers=10.1.2.180:7001 host add 10.1.2.201 --ip=10.1.2.201 --status=alive 2、
#mogadm --trackers=10.1.2.180:7001 device add 10.1.2.201 002
3、
#mogadm --trackers=10.1.2.180:7001 host list
10.1.2.200 : alive
IP: 10.1.2.200:7500
10.1.2.201 : alive
IP: 10.1.2.201:7500
#mogadm --trackers=10.1.2.180:7001 device list
10.1.2.200 : alive
used(G) free(G) total(G)weight(%)
dev1: alive 0.000 0.000 0.000 100
10.1.2.201 : alive
used(G) free(G) total(G)weight(%)
dev2: alive 0.000 0.000 0.000 100
4、上传文件试试
4.1mogupload 用法
注意这里key，访问mogilefs时，所使用的http所访问的URL
# mogupload -h
Usage: /usr/bin/mogupload --trackers=host --domain=foo --key='/hello.jpg' --file='./hello.jpg' 4.2上传一个文件，有错误提示了
# mogupload --trackers=10.1.2.180:7001 --domain=files --key='fstab.html' --file='/etc/fstab'
Error opening MogileFS file: no_devices No devices found to store file at /usr/bin/mogupload line 83,line 1. 5.回到200主机，创建/data0/nfs/mogdata/dev1
# mkdir dev1# chown -R mogilefs.mogilefs dev1/
重启mogstored服务
6、201主机，创建/data0/nfs/mogdata/dev2
# mkdir dev2# chown -R mogilefs.mogilefs dev2/
7、
#mogadm --trackers=10.1.2.180:7001 device list
10.1.2.200 : alive
used(G) free(G) total(G)weight(%)
dev1: alive 0.064 328.843 328.907 100
10.1.2.201 : alive
used(G) free(G) total(G)weight(%)
dev2: alive 0.718 2.932 3.649 100
8、这时没有错示了
# mogupload --trackers=10.1.2.180:7001 --domain=files --key='fstab.html' --file='/etc/fstab'
9、列出所有key
# moglistkeys --trackers=10.1.2.180:7001 --domain=files
fstab.html
10、
# mogfileinfo --tracker=10.1.2.180:7001 --domain=files --key='fstab.html'
- file: fstab.html
class: default
devcount: 2
domain: files
fid: 4
key: fstab.html
length: 899
- http://10.1.2.201:7500/dev2/0/000/000/0000000004.fid
- http://10.1.2.200:7500/dev1/0/000/000/0000000004.fid
11、把第10步的URL其中一个复制粘贴到浏览器上，即可访问的到
八、常用操作，这里列出两个，可以查看帮助，有详细介绍
1、关闭201存储节点
#mogadm --trackers=10.1.2.180:7001 device -h
Help for 'device' command:
(enter any command prefix, leaving off options, for further help)
mogadm device add Add a device to a host.
mogadm device list List all devices, for each host.
mogadm device mark Mark a device as {alive,dead,down,drain,readonly}
mogadm device modify Modify a device's properties.
mogadm device summary List the summary of devices, for each host.#mogadm --trackers=10.1.2.180:7001 devicemark 10.1.2.201 dev2 down#mogadm --trackers=10.1.2.180:7001 device list
10.1.2.200 : alive
used(G) free(G) total(G)weight(%)
dev1: alive 0.064 328.843 328.907 100
10.1.2.201 : alive
used(G) free(G) total(G)weight(%)
dev2: down 0.718 2.932 3.649 100 2、下载文件到本地
# mogfetch -h
Usage: /usr/bin/mogfetch --trackers=host --domain=foo --key='/hello.jpg' --file='./output'# mogfetch --trackers=10.1.2.180 --domain=files --key='fstab.html' --file='./output.txt'# cat output.txt
#
# /etc/fstab
# Created by anaconda on Thu Dec 25 19:10:03 2014
#
# Accessible filesystems, by reference, are maintained under '/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
UUID=17793804-41a2-4cec-b58d-ffec3766aaf3 / ext4 defaults 1 1
UUID=702f98d6-7f82-4b92-8097-5c1c9498ebda /boot ext4 defaults 1 2
UUID=b47bf314-0653-4f4b-a3ca-5f11a8342377 /data0 ext4 defaults 1 2
UUID=a1172ffe-cc17-4639-9c97-a519fc6f0386 swap swap defaults 0 0
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devptsgid=5,mode=6200 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
页:
[1]