MogileFS高级应用(二)最后有疑问
一、准备阶段1、5台机器
10.1.2.180 作为mysql机器
10.1.2.200-202作为集群节点
10.1.2.203作为nginx反向代理
2、拓扑图
http://s3.运维网.com/wyfs02/M00/74/24/wKiom1YV4lmjLv-LAADMyN1laDY075.jpg
三、mogilefs集群部署步聚
1、180机器
1.1由于前面单节，180机器安装了mysql和Mogilefs，而现在180作为单独Mysql机器，所以先停掉Mogilefs服务
# /etc/init.d/mogilefsd stop
Stopping mogilefsd [确定] 1.2创建mysql用户root的密码，方便集群节点连接上去
MariaDB > GRANT ALL ON *.* TO 'root'@'10.1.%.%' IDENTIFIED BY '168';MariaDB > flush privileges;
Query OK, 0 rows affected (0.04 sec) 2、200机器
2.1 重新初始化数据库
# mogdbsetup --dbhost=10.1.2.180 --dbname=mogilefs --dbport=3306 --dbrootuser=root --dbrootpass=168 --dbuser=moguser --dbpass=mog168 2.2修改配置文件
root@xenserver1 mogdata]# vim /etc/mogilefs/mogilefsd.conf
db_dsn = DBI:mysql:mogilefs:host=10.1.2.180
db_user = moguser
db_pass = mog168
listen = 0.0.0.0:7001 2.3重启服务
# service mogilefsd restart# ss -tnl | sed -n '/7001/p'
LISTEN 0 128 127.0.0.1:7001 *:* 2.4启动 mogstored服务
# vim /etc/mogilefs/mogstored.conf
maxconns = 10000
httplisten = 0.0.0.0:7500
mgmtlisten = 0.0.0.0:7501
docroot = /data0/nfs/mogdata/# service mogstored restart# ss -tnl | sed -n '/7500\|7501\|7001/p'
LISTEN 0 128 127.0.0.1:7001 *:*
LISTEN 0 128 *:7500 *:*
LISTEN 0 128 *:7501 *:* 2.3由于原来已经有了，所以先删除再重新添加
# mogadm --trackers=10.1.2.200:7001 host list
10.1.2.200 : alive
IP: 10.1.2.200:7500
10.1.2.201 : alive
IP: 10.1.2.201:7500# mogadm --trackers=10.1.2.200:7001 host delete 10.1.2.200
Failure deleting host: host_not_empty Unable to delete host; it contains devices still 2.4有错误提示了，先去到180数据库下删除dev1,dev2
MariaDB > select * from device
-> ;
+-------+--------+--------+--------+----------+---------+------------+
| devid | hostid | status | weight | mb_total | mb_used | mb_asof |
+-------+--------+--------+--------+----------+---------+------------+
| 1 | 1 | down | 100 | 336801 | 66 | 1443524971 |
| 2 | 2 | alive| 100 | 3737 | 735 | 1443525069 |
+-------+--------+--------+--------+----------+---------+------------+
2 rows in set (0.00 sec)
MariaDB > delete from device where devid=1;
Query OK, 1 row affected (0.02 sec)
MariaDB > delete from device where devid=2;
Query OK, 1 row affected (0.01 sec) 2.5再执行删除主机
# mogadm --trackers=10.1.2.200:7001 host delete 10.1.2.200
# mogadm --trackers=10.1.2.200:7001 host delete 10.1.2.201
# mogadm --trackers=10.1.2.200:7001 host list
2.6把自己添加进来
# mogadm --trackers=10.1.2.200:7001 host add 10.1.2.200 --ip=10.1.2.200 --status=alive
# mogadm --trackers=10.1.2.200:7001 host list
10.1.2.200 : alive
IP: 10.1.2.200:7500
2.7、把mogilefs配置文件复制到201，202机器上，这两台机器安装的mogilefs请参考上章节来安装
# scp /etc/mogilefs/* 10.1.2.201:/etc/mogilefs/
root@10.1.2.201's password:
mogilefsd.conf 100% 1459 1.4KB/s 00:00
mogstored.conf 100% 99 0.1KB/s 00:00# scp /etc/mogilefs/* 10.1.2.202:/etc/mogilefs/
root@10.1.2.201's password:
mogilefsd.conf 100% 1459 1.4KB/s 00:00
mogstored.conf 100% 99 0.1KB/s 00:00 3、201机器上
3.1启动mogilefs,mogstored服务
# service mogilefsd start
Starting mogilefsd [确定]# service mogstored start
Starting mogstored [确定] 3.2通过本机验证一下
# mogadm --tracker=10.1.2.201:7001 host list
10.1.2.200 : alive
IP: 10.1.2.200:7500 3.3把自己也添加进来
# mogadm --tracker=10.1.2.201:7001 host add 10.1.2.201 --ip=10
.1.2.201 --status=alive
You have new mail in /var/spool/mail/root
# mogadm --tracker=10.1.2.201:7001 host list
10.1.2.200 : alive
IP: 10.1.2.200:7500
10.1.2.201 : alive
IP: 10.1.2.201:7500 3.4去到200机子上查看
# mogadm --trackers=10.1.2.200:7001 host list
10.1.2.200 : alive
IP: 10.1.2.200:7500
10.1.2.201 : alive
IP: 10.1.2.201:7500
4、202机器上
4.1创建目录
# mkdir /data0/nfs/mogdata/dev3 -pv
mkdir: 已创建目录 "/data0"
mkdir: 已创建目录 "/data0/nfs"
mkdir: 已创建目录 "/data0/nfs/mogdata"
mkdir: 已创建目录 "/data0/nfs/mogdata/dev3"
# chown -R mogilefs.mogilefs /data0/nfs/mogdata/ 4.2启动服务后，添加主机
# mogadm --trackers=10.1.2.202:7001 host add 10.1.2.202 --ip=10.1.2.202 --status=alive
# mogadm --trackers=10.1.2.202:7001 host list
10.1.2.200 : alive
IP: 10.1.2.200:7500
10.1.2.201 : alive
IP: 10.1.2.201:7500
10.1.2.202 : alive
IP: 10.1.2.202:7500
5、在集群节点任何一台添加设备ID，
5.1现在就选200，此时设备列表为空的
# mogadm device list
10.1.2.200 : alive
used(G) free(G) total(G)weight(%)
10.1.2.201 : alive
used(G) free(G) total(G)weight(%)
10.1.2.202 : alive
used(G) free(G) total(G)weight(%) 5.2添回设备
# mogadm --tracker=10.1.2.200:7001 device add 10.1.2.200 1
# mogadm --tracker=10.1.2.200:7001 device add 10.1.2.201 2
# mogadm --tracker=10.1.2.200:7001 device add 10.1.2.202 3 # mogadm --tracker=10.1.2.200:7001 device list
10.1.2.200 : alive
used(G) free(G) total(G)weight(%)
dev1: alive 0.064 328.843 328.907 100
10.1.2.201 : alive
used(G) free(G) total(G)weight(%)
dev2: alive 0.718 2.932 3.649 100
10.1.2.202 : alive
used(G) free(G) total(G)weight(%)
dev3: alive 0.418 3.169 3.587 100 5.4选201节点验证一下
# mogadm --trackers=10.1.2.201:7001 device list
10.1.2.200 : alive
used(G) free(G) total(G)weight(%)
dev1: alive 0.064 328.843 328.907 100
10.1.2.201 : alive
used(G) free(G) total(G)weight(%)
dev2: alive 0.718 2.932 3.649 100
10.1.2.202 : alive
used(G) free(G) total(G)weight(%)
dev3: alive 0.418 3.169 3.587 100 6、在域内添加class类
6.1
# mogadm --tracker=10.1.2.200:7001 class list
domain class mindevcount replpolicy hashtype
-------------------- -------------------- ------------- ------------ -------
files default 2 MultipleHosts() NONE
images default 2 MultipleHosts() NONE 6.2
# mogadm --tracker=10.1.2.200:7001 class add
ERROR: Missing argument 'domain'
Help for 'class-add' command:
mogadm class add Add a file class to a domain.
Name of class to add.
Domain to add class to.
--hashtype=s Hash algorithm string ('MD5', 'NONE').
--mindevcount=i Minimum number of replicas.
--replpolicy=s Replication policy string. 6.3
# mogadm --trackers=10.1.2.200:7001 class add images class0 --mindevcount=2
# mogadm --trackers=10.1.2.200:7001 class add images class1 --mindevcount=2
# mogadm --trackers=10.1.2.200:7001 class add images class2 --mindevcount=2root@xenserver1 ~]# mogadm --tracker=10.1.2.200:7001 class list
domain class mindevcount replpolicy hashtype
-------------------- -------------------- ------------- ------------ -------
files default 2 MultipleHosts() NONE
images class0 2 MultipleHosts() NONE
images class1 2 MultipleHosts() NONE
images class2 2 MultipleHosts() NONE
images default 2 MultipleHosts() NONE 四、203机器上nginx部署
1、下载地址
http://nginx.org/en/download.html
http://www.grid.net.ru/nginx/mogilefs.en.html
2、
# yum -y groupinstall "Development Tools" "Server Platform Develoment" 3、
# yum install pcre-devel -y 4、
# ls
nginx-1.8.0 nginx_mogilefs_module-1.0.4
nginx-1.8.0.tar.gznginx_mogilefs_module-1.0.4.tar.gz 5、进到相应nginx-1.8.0目录下编译安装
# ./configure --prefix=/usr --sbin-path=/usr/sbin/nginx --conf-path=/etc/nginx/nginx.conf --error-log-path=/var/log/nginx/error.log --http-log-path=/var/log/nginx/access.log --pid-path=/var/run/nginx/nginx.pid --lock-path=/var/lock/nginx.lock --user=nginx --group=nginx --with-http_ssl_module --with-http_flv_module --with-http_stub_status_module --with-http_gzip_static_module --http-client-body-temp-path=/var/tmp/nginx/client/ --http-proxy-temp-path=/var/tmp/nginx/proxy/ --http-fastcgi-temp-path=/var/tmp/nginx/fcgi/ --http-uwsgi-temp-path=/var/tmp/nginx/uwsgi --http-scgi-temp-path=/var/tmp/nginx/scgi --with-pcre --with-debug --add-module=../nginx_mogilefs_module-1.0.4 6、为nginx提供init脚本
# vim /etc/init.d/nginx
#!/bin/sh
#
# nginx - this script starts and stops the nginx daemon
#
# chkconfig: - 85 15
# description:Nginx is an HTTP(S) server, HTTP(S) reverse \
# proxy and IMAP/POP3 proxy server
# processname: nginx
# config: /etc/nginx/nginx.conf
# config: /etc/sysconfig/nginx
# pidfile: /var/run/nginx.pid
# Source function library.
. /etc/rc.d/init.d/functions
# Source networking configuration.
. /etc/sysconfig/network
# Check that networking is up.
[ "$NETWORKING" = "no" ] && exit 0
nginx="/usr/sbin/nginx"
prog=$(basename $nginx)
NGINX_CONF_FILE="/etc/nginx/nginx.conf"
[ -f /etc/sysconfig/nginx ] && . /etc/sysconfig/nginx
lockfile=/var/lock/subsys/nginx
make_dirs() {
# make required directories
user=`nginx -V 2>&1 | grep "configure arguments:" | sed 's/[^*]*--user=\([^ ]*\).*/\1/g' -`
options=`$nginx -V 2>&1 | grep 'configure arguments:'`
for opt in $options; do
if [ `echo $opt | grep '.*-temp-path'` ]; then
value=`echo $opt | cut -d "=" -f 2`
if [ ! -d "$value" ]; then
# echo "creating" $value
mkdir -p $value && chown -R $user $value
fi
fi
done
}
start() {
[ -x $nginx ] || exit 5
[ -f $NGINX_CONF_FILE ] || exit 6
make_dirs
echo -n $"Starting $prog: "
daemon $nginx -c $NGINX_CONF_FILE
retval=$?
echo
[ $retval -eq 0 ] && touch $lockfile
return $retval
}
stop() {
echo -n $"Stopping $prog: "
killproc $prog -QUIT
retval=$?
echo
[ $retval -eq 0 ] && rm -f $lockfile
return $retval
}
restart() {
configtest || return $?
stop
sleep 1
start
}
reload() {
configtest || return $?
echo -n $"Reloading $prog: "
killproc $nginx -HUP
RETVAL=$?
echo
}
force_reload() {
restart
}
configtest() {
$nginx -t -c $NGINX_CONF_FILE
}
rh_status() {
status $prog
}
rh_status_q() {
rh_status >/dev/null 2>&1
}
case "$1" in
start)
rh_status_q && exit 0
$1
;;
stop)
rh_status_q || exit 0
$1
;;
restart|configtest)
$1
;;
reload)
rh_status_q || exit 7
$1
;;
force-reload)
force_reload
;;
status)
rh_status
;;
condrestart|try-restart)
rh_status_q || exit 0
;;
*)
echo $"Usage: $0 {start|stop|status|restart|condrestart|try-restart|reload|force-reload|configtest}"
exit 2
esac 7、
# groupadd -r nginx
# useradd -r -g nginx nginx# chmod +x /etc/init.d/nginx
# chkconfig --add nginx# service nginx configtest
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful 五、上传图片
# mogupload --trackers=10.1.2.200:7001 --domain=images --key='/images/1.png' --file='/root/picture/1.png'
# mogupload --trackers=10.1.2.200:7001 --domain=images --key='/images/2.png' --file='/root/picture/2.png'
# moglistkeys --trackers=10.1.2.200 --domain=images
/images/1.png
/images/2.png# mogfileinfo --tracker=10.1.2.201:7001 --domain=images --key='/images/1.png'
- file: /images/1.png
class: default
devcount: 2
domain: images
fid: 10
key: /images/1.png
length: 178950
- http://10.1.2.201:7500/dev2/0/000/000/0000000010.fid
- http://10.1.2.202:7500/dev3/0/000/000/0000000010.fid
# mogfileinfo --tracker=10.1.2.201:7001 --domain=images --key='/images/2.png'
- file: /images/2.png
class: default
devcount: 2
domain: images
fid: 11
key: /images/2.png
length: 149341
- http://10.1.2.202:7500/dev3/0/000/000/0000000011.fid
- http://10.1.2.201:7500/dev2/0/000/000/0000000011.fid 六、配置nginx使用mogilefs模块
1、
# cp /etc/nginx/nginx.conf /etc/nginx/nginx.conf.bak 2、编辑配置文件
# vim /etc/nginx/nginx.conf
# }
server {
listen 80;
server_namelocalhost;
#charset koi8-r;
#access_loglogs/host.access.logmain;
location / {
root html;
indexindex.html index.htm;
}
location /files/ {
mogilefs_tracker mogcluster;
mogilefs_domain files;
# mogilefs_methods GET;
# mogilefs_noverify on;
mogilefs_pass {
proxy_pass $mogilefs_path;
proxy_hide_header Content-Type;
proxy_buffering off;
}
}
location /images/ {
# mogilefs_tracker 10.1.2.200:7001;
mogilefs_tracker mogcluster;
# }
server {
listen 80;
server_namelocalhost;
#charset koi8-r;
#access_loglogs/host.access.logmain;
location / {
root html;
indexindex.html index.htm;
}
location /files/ {
mogilefs_tracker mogcluster;
mogilefs_domain files;
# mogilefs_methods GET;
# mogilefs_noverify on;
mogilefs_pass {
proxy_pass $mogilefs_path;
proxy_hide_header Content-Type;
proxy_buffering off;
}
}
location /images/ {
# mogilefs_tracker 10.1.2.200:7001;
mogilefs_tracker mogcluster;
#mogilefs_domain files;
mogilefs_domain images;
mogilefs_methods GET;
mogilefs_noverify on;
mogilefs_pass {
proxy_pass $mogilefs_path;
proxy_hide_header Content-Type;
proxy_buffering off;
}
}
#error_page404 /404.html;
# redirect server error pages to the static page /50x.html
#
error_page 500 502 503 504/50x.html;
location = /50x.html {
root html;
}
# proxy the PHP scripts to Apache listening on 127.0.0.1:80
#
#location ~ \.php$ {
# proxy_pass http://127.0.0.1;
#}
# pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000
#
#location ~ \.php$ {
# root html;
# fastcgi_pass 127.0.0.1:9000;
# fastcgi_indexindex.php;
# fastcgi_paramSCRIPT_FILENAME/scripts$fastcgi_script_name;
# include fastcgi_params;
#}
# deny access to .htaccess files, if Apache's document root
# concurs with nginx's one
#
#location ~ /\.ht {
# denyall;
#}
}
# another virtual host using mix of IP-, name-, and port-based configuration
#
#server {
# listen 8000;
# listen somename:8080;
# server_namesomenamealiasanother.alias;
# location / {
# root html;
# indexindex.html index.htm;
# }
#}
# HTTPS server
#
#server {
# listen 443 ssl;
# server_namelocalhost;
# ssl_certificate cert.pem;
# ssl_certificate_keycert.key;
# ssl_session_cache shared:SSL:1m;
# ssl_session_timeout5m;
# ssl_ciphersHIGH:!aNULL:!MD5;
# ssl_prefer_server_cipherson;
# location / {
# root html;
# indexindex.html index.htm;
# }
#}
} 七、查看当配置状态信息
# mogstats -h
Usage:
mogstats --db_dsn="DBI:mysql:mfs:host=mfshost" --db_user="mfs"
--db_pass="mfs" --verbose --stats="devices,files"
mogstats --stats="all"
mogstats
valid stats: all, delete-queue, devices, domains, fids, files, general-queues, replication, replication-queue
You have new mail in /var/spool/mail/root
# mogstats --config=/etc/mogilefs/mogilefsd.conf
Fetching statistics... (all)
Statistics for devices...
device host files status
---------- ---------------- ------------ ----------
dev1 10.1.2.200 3 alive
dev2 10.1.2.201 4 alive
dev3 10.1.2.202 3 alive
---------- ---------------- ------------ ----------
Statistics for file ids...
Max file id: 14
Statistics for files...
domain class files size (m)fullsize (m)
-------------------- ----------- ---------- ----------- -------------
files default 1 0 0
images default 4 0 0
-------------------- ----------- ---------- ----------- -------------
Statistics for replication...
domain class devcount files
-------------------- ----------- ---------- ----------
files default 2 1
images default 2 4
-------------------- ----------- ---------- ----------
Statistics for replication queue...
status count
-------------------- ------------
-------------------- ------------
Statistics for delete queue...
status count
-------------------- ------------
-------------------- ------------
Statistics for general queues...
queue status count
--------------- -------------------- ------------
--------------- -------------------- ------------
done
# mogstats -h
Usage:
mogstats --db_dsn="DBI:mysql:mfs:host=mfshost" --db_user="mfs"
--db_pass="mfs" --verbose --stats="devices,files"
mogstats --stats="all"
mogstats
valid stats: all, delete-queue, devices, domains, fids, files, general-queues, replication, replication-queue
You have new mail in /var/spool/mail/root
# mogstats --config=/etc/mogilefs/mogilefsd.conf --stats="domains"
Fetching statistics... (domains)
Statistics for domains...
domain class files
-------------------- ----------- ----------
files default 1
images default 4
-------------------- ----------- ----------
done
八、结果
1，访问图片，提示查找不到
http://s3.运维网.com/wyfs02/M01/74/21/wKioL1YV4U-jXqEcAADxCOoQ45k988.jpg
查看错误日志
# cat /var/log/nginx/error.log
2015/10/08 11:17:03 2158#0: *1 connect() failed (113: No route to host) while connecting to upstream, client: 10.1.2.62, server: localhost, request: "GET /files/fstab.html HTTP/1.1", upstream: "10.1.2.200:7001", host: "10.1.2.203"
" while reading response header from upstream, client: 10.1.2.62, server: localhost, request: "GET /images/1.png HTTP/1.1", upstream: "10.1.2.202:7001", host: "10.1.2.203"
" while reading response header from upstream, client: 10.1.2.62, server: localhost, request: "GET /images/2.png HTTP/1.1", upstream: "10.1.2.201:7001", host: "10.1.2.203"
2、访问文件却可以
http://s3.运维网.com/wyfs02/M01/74/21/wKioL1YV4gXihRTfAAKk_xGd3nk857.jpg
3、百思不得其解，莫非我模块不兼容么
页:
[1]