简介
MogileFS是一个开源的分布式文件存储系统，由LiveJournal旗下的DangaInteractive公司开发。Danga团队开发了包括Memcached、MogileFS、Perlbal等多个知名的开源项目。
特性 1、应用层：用户空间文件系统，无须特殊的核心组件 2、无单点：（tracker, mogstore, database(MySQL)） 3、自动文件复制：复制的最小单位不是文件，而class； 4、传输中立，无特殊协议：可以通过NFS或HTTP进行通信； 5、简单的命名空间： 每个文件对应一个key：用于domain定义名称空间
组成部分
MogileFS： tracker：追踪元数据 database：存储元数据 storage：存储数据
tracker： mogilefsd（守护进程）,职责： replication：节点之间的文件复制 deletion：删除文件 queryworker：响应客户请求的文件元数据访问请求 reaper：在存储失败后将文件复制请求重新放置于队列中 monitor：检测主机和设备的健康状态 database： 存储mogilefs的元数据，一般使用mysql，建议使用冗余方案保证可用性（MMM,MHA） mogilefs专门提供了数据结构管理工具mogdbsetup； storage： mogstored（进程名），一个准备停当的mogstored节点可通过mogadm命令添加至现在的集群中， 存储节点需定义“设备dev”用作存储空间；每个设备在当前集群中需通过一个唯一的DevID标识。 client： 客户端用于与mogilefs简历通信，完成数据存取；
安装包： MogileFS-Server-2.46-2.el6.noarch.rpm MogileFS-Server-mogilefsd-2.46-2.el6.noarch.rpm MogileFS-Server-mogstored-2.46-2.el6.noarch.rpm MogileFS-Utils-2.19-1.el6.noarch.rpm perl-MogileFS-Client-1.14-1.el6.noarch.rpm perl-Net-Netmask-1.9015-8.el6.noarch.rpm perl-Perlbal-1.78-1.el6.noarch.rpm
架构：
1.安装mogilefs和数据库 # ls MogileFS-Server-2.46-2.el6.noarch.rpm perl-MogileFS-Client-1.14-1.el6.noarch.rpm MogileFS-Server-mogilefsd-2.46-2.el6.noarch.rpm perl-Net-Netmask-1.9015-8.el6.noarch.rpm MogileFS-Server-mogstored-2.46-2.el6.noarch.rpm perl-Perlbal-1.78-1.el6.noarch.rpm MogileFS-Utils-2.19-1.el6.noarch.rpm # yum localinstall *.rpm -y # yum install mysql-server mysql -y 2.数据库授权远程访问，并继承授权权限 mysql> grant all on *.* to root@'192.168.95.%' identified by 'liaobin' with grant option; mysql> flush privileges; 3.创建mofilefs的数据库用户以及授权 mysql> grant all on *.* to mguser@'192.168.95.%' identified by 'liaobin'; mysql> flush privileges; 4.用mogilefs自带工具mogdbsetup初始化mogilefs数据库 # mogdbsetup --dbhost=192.168.95.30 --dbname=mogilefs --dbrootuser=root --dbrootpass=liaobin --dbuser=mguser --dbpass=liaobin --yes 查看初始化的数据库 mysql> show databases; +--------------------+ | Database | +--------------------+ | information_schema | | mogilefs | | mysql | | test | +--------------------+ 4 rows in set (0.01 sec)
5.修改tracker（mogilefsd）配置文件 # vim /etc/mogilefs/mogilefsd.conf # Enable daemon mode to work in background and use syslog daemonize = 1 # Where to store the pid of the daemon (must be the same in the init script) pidfile = /var/run/mogilefsd/mogilefsd.pid # Database connection information db_dsn = DBI:mysql:mogilefs:host=192.168.95.30 #配置database的连接信息 db_user = mguser #配置mogilefs的数据库用户密码 db_pass = liaobin # IP:PORT to listen on for mogilefs client requests listen = 0.0.0.0:7001 #配置监听的地址和端口 # Optional, if you don't define the port above. conf_port = 7001 # Number of query workers to start by default. query_jobs = 10 #生成多少个用于查询的工作进程 # Number of delete workers to start by default. delete_jobs = 1 #生成多少个用于删除的工作进程 # Number of replicate workers to start by default. replicate_jobs = 5 #生成多少个用于删除的工作进程 # Number of reaper workers to start by default. # (you don't usually need to increase this) reaper_jobs = 1 # Number of fsck workers to start by default. # (these can cause a lot of load when fsck'ing) #fsck_jobs = 1 # Minimum amount of space to reserve in megabytes # default: 100 # Consider setting this to be larger than the largest file you # would normally be uploading. #min_free_space = 200 # Number of seconds to wait for a storage node to respond. # default: 2 # Keep this low, so busy storage nodes are quickly ignored. #node_timeout = 2 # Number of seconds to wait to connect to a storage node. # default: 2 # Keep this low so overloaded nodes get skipped. #conn_timeout = 2 # Allow replication to use the secondary node get port, # if you have apache or similar configured for GET's #repl_use_get_port = 1 6.启动mogilefsd服务 [iyunv@localhost mgfs]# /etc/init.d/mogstored start 7.为每个节点新增加分区，作为存储 # fdisk /dev/sda ......... # partx -a /dev/sda # mkfs.ext4 /dev/sda5 # mkdir /mgfs # vim /etc/fstab #新增如下条目 /dev/sda5 /mgfs ext4 defaults 0 0 # mount -a # mount /dev/sda5 on /mgfs type ext4 (rw) 8.三台服务器分别创建存储目录，并修改权限 95.30： # mkdir /mgfs/dev1 # chown -R /mgfs/dev1 95.34： # mkdir /mgfs/dev2 # chown -R /mgfs/dev2 95.35： # mkdir /mgfs/dev3 # chown -R /mgfs/dev3 9.修改store(mogstored)的配置文件 #vim /etc/mogilefs/mogstored.conf maxconns = 10000 httplisten = 0.0.0.0:7500 mgmtlisten = 0.0.0.0:7501 #注意此处，写到dev(1,2,3)的上级目录就好 docroot = /mgfs 10.启动mogstored服务 # /etc/init.d/mogstored start
11.用mogadm管理 创建存储主机： # mogadm --trackers=192.168.95.30:7001 host add store1 --ip=192.168.95.30 # mogadm --trackers=192.168.95.30:7001 host add store2 --ip=192.168.95.34 # mogadm --trackers=192.168.95.30:7001 host add store3 --ip=192.168.95.35 查看创建的主机： # mogadm --trackers=192.168.95.30:7001 host list
创建存储设备： # mogadm --trackers=192.168.95.30:7001 device add store1 1 --status=alive # mogadm --trackers=192.168.95.30:7001 device add store2 2 --status=alive # mogadm --trackers=192.168.95.30:7001 device add store3 3 --status=alive 查看创建的存储设备： # mogadm --trackers=192.168.95.30:7001 device list
创建域domain： mogadm --trackers=192.168.95.30:7001 domain add img mogadm --trackers=192.168.95.30:7001 domain add conf 查看创建的域： mogadm --trackers=192.168.95.30:7001 domain list
上传文件测试： mogupload --trackers=192.168.95.30:7001 --domain=img --key="1.jpg" --file="/root/1.jpg" 查看上传的文件： # mogfileinfo --trackers=192.168.95.30:7001 --domain=img --key="1.jpg" - file: 1.jpg class: default devcount: 2 domain: img fid: 9 key: 1.jpg length: 34892 用fid的url访问测试：
正常访问
nginx反向代理mogilefs
1.安装nginx和mogilefs模块 安装好开发环境： # yum groupinstall "Development tools" "Server Platform Development" -y # yum install pcre-devel -y 下载解压nginx包和mogilefs包：
开始编译安装： #cd nginx-1.6.2 # ./configure \ --prefix=/usr/local/nginx \ --sbin-path=/usr/local/nginx/sbin/nginx \ --conf-path=/etc/nginx/nginx.conf \ --error-log-path=/var/log/nginx/error.log \ --http-log-path=/var/log/nginx/access.log \ --pid-path=/var/run/nginx/nginx.pid \ --lock-path=/var/lock/nginx.lock \ --user=nginx \ --group=nginx \ --with-http_ssl_module \ --with-http_flv_module \ --with-http_stub_status_module \ --with-http_gzip_static_module \ --http-client-body-temp-path=/var/tmp/nginx/client/ \ --http-proxy-temp-path=/var/tmp/nginx/proxy/ \ --http-fastcgi-temp-path=/var/tmp/nginx/fcgi/ \ --http-uwsgi-temp-path=/var/tmp/nginx/uwsgi \ --http-scgi-temp-path=/var/tmp/nginx/scgi \ --with-pcre --add-module=../nginx_mogilefs_module-1.0.4 # make && make install 给nginx创建服务脚本 # vim /etc/init.d/nginx #!/bin/sh # # nginx - this script starts and stops the nginx daemon # # chkconfig: - 85 15 # description: Nginx is an HTTP(S) server, HTTP(S) reverse \ # proxy and IMAP/POP3 proxy server # processname: nginx # config: /etc/nginx/nginx.conf # config: /etc/sysconfig/nginx # pidfile: /var/run/nginx.pid
# Source function library. . /etc/rc.d/init.d/functions
# Source networking configuration. . /etc/sysconfig/network
# Check that networking is up. [ "$NETWORKING" = "no" ] && exit 0
nginx="/usr/local/nginx/sbin/nginx" prog=$(basename $nginx)
NGINX_CONF_FILE="/etc/nginx/nginx.conf"
[ -f /etc/sysconfig/nginx ] && . /etc/sysconfig/nginx
lockfile=/var/lock/subsys/nginx
make_dirs() { # make required directories user=`nginx -V 2>&1 | grep "configure arguments:" | sed 's/[^*]*--user=\([^ ]*\).*/\1/g' -` options=`$nginx -V 2>&1 | grep 'configure arguments:'` for opt in $options; do if [ `echo $opt | grep '.*-temp-path'` ]; then value=`echo $opt | cut -d "=" -f 2` if [ ! -d "$value" ]; then # echo "creating" $value mkdir -p $value && chown -R $user $value fi fi done }
start() { [ -x $nginx ] || exit 5 [ -f $NGINX_CONF_FILE ] || exit 6 make_dirs echo -n $"Starting $prog: " daemon $nginx -c $NGINX_CONF_FILE retval=$? echo [ $retval -eq 0 ] && touch $lockfile return $retval }
stop() { echo -n $"Stopping $prog: " killproc $prog -QUIT retval=$? echo [ $retval -eq 0 ] && rm -f $lockfile return $retval }
restart() { configtest || return $? stop sleep 1 start }
reload() { configtest || return $? echo -n $"Reloading $prog: " killproc $nginx -HUP RETVAL=$? echo }
force_reload() { restart }
configtest() { $nginx -t -c $NGINX_CONF_FILE }
rh_status() { status $prog }
rh_status_q() { rh_status >/dev/null 2>&1 }
case "$1" in start) rh_status_q && exit 0 $1 ;; stop) rh_status_q || exit 0 $1 ;; restart|configtest) $1 ;; reload) rh_status_q || exit 7 $1 ;; force-reload) force_reload ;; status) rh_status ;; condrestart|try-restart) rh_status_q || exit 0 ;; *) echo $"Usage: $0 {start|stop|status|restart|condrestart|try-restart|reload|force-reload|configtest}" exit 2 esac 给予执行权限： # chmod +x /etc/init.d/nginx 创建nginx用户和必须目录： # useradd -r nginx # mkdir /var/tmp/nginx/client -pv 修改nginx配置文件让其支持mogilefs反向代理（红色字体为添加项）： # vim /etc/nginx/nginx.conf #user nobody; worker_processes 1;
#error_log logs/error.log; #error_log logs/error.log notice; #error_log logs/error.log info;
#pid logs/nginx.pid;
events { worker_connections 1024; }
http { include mime.types; default_type application/octet-stream;
#log_format main '$remote_addr - $remote_user [$time_local] "$request" ' # '$status $body_bytes_sent "$http_referer" ' # '"$http_user_agent" "$http_x_forwarded_for"';
#access_log logs/access.log main;
sendfile on; #tcp_nopush on;
#keepalive_timeout 0; keepalive_timeout 65;
#gzip on; upstream stores { server 192.168.95.30:7001; server 192.168.95.34:7001; server 192.168.95.35:7001; } server { listen 80; server_name localhost;
#charset koi8-r;
#access_log logs/host.access.log main;
location / { root html; index index.html index.htm; } location /img/ { mogilefs_tracker stores; mogilefs_domain img;
mogilefs_pass { proxy_pass $mogilefs_path; proxy_hide_header Content-Type; proxy_buffering off; } } location /upload/ { mogilefs_tracker stores; mogilefs_domain img; mogilefs_methods PUT GET DELETE;
mogilefs_pass { proxy_pass $mogilefs_path; proxy_hide_header Content-Type; proxy_buffering off; } }
#error_page 404 /404.html;
} 用路径访问测试nginx代理：
再上传一个文件测试： # mogupload --trackers=192.168.95.30:7001 --domain=img --key="2.jpg" --file="/root/2.jpg"
测试删除文件：
| 
QQ群⑧：
窥视卡
雷达卡
发表于 2016-6-23 09:42:05
提升卡
置顶卡
沉默卡
喧嚣卡
变色卡
千斤顶
显身卡