|
|
一. 准备环境:
| 服务器角色 | IP | | image fastdfs_tracker_server | 10.1.11.35 | | 10.1.11.36 | | image fastdfs_storage_server | 10.1.11.35 | | 10.1.11.36 | | image nginx_with_fdfs_mod_server | 10.1.11.35 | | 10.1.11.36 | | news fastdfs_tracker_server | 10.1.12.51 | | 10.1.12.52 | | news fastdfs_storage_server | 10.1.12.51 | | 10.1.12.52 | | news nginx_with_fdfs_mod_server | 10.1.12.51 | | 10.1.12.52 | | tenginx | 10.1.11.22 |
客户端通过调用java client api的上传接口完成文件上传.
内部流程是:
1) client api请求tracker server,
2) tracker server根据storage server的连接数,磁盘空间,负载等计算选择一台storage server将其ip返回client api
3) client api向storage server发送上传请求, 完成文件上传
4) storage server向client api返回file id(storage server本地的文件逻辑路径,如:
M00/00/FB/CgELI1XqgSuAWgRfAAD7wSFTiOA15.jpeg,其中:
M00: fastdfs中的pathid
00/FB: 在M00/下创建的两级目录, 这里为每个path创建了256*256个目录
CgELI1XqgSuAWgRfAAD7wSFTiOA15.jpeg: fastdfs根据文件创建时间,大小,storageserver ip,crc32,随机数组合生成文件名)
1) client发送http请求tengine
2) tengine按照设定负载均衡策略将请求代理到后端带fastdfs模块的nginx服务器
3) nginx检查本地是否存在http请求的资源,有则直接返回client, 没有则调用fastdfs的下载api向指定的storage server获取资源后再返回client
下载fastdfs安装包和依赖:
FastDFS_v5.05.tar.gz ------>fastdfs服务,包括tracker,storage
fastdfs-nginx-module_v1.16.tar.gz ------>fastdfs的nginx模块
libfastcommon-master.zip ------>fastdfs的依赖库
nginx-1.8.0.tar.gz ------>nginx
二. 安装部署:
这里使用shell脚本完成一键安装:
首先将fastdfs所需的软件包打包成如下目录:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
| [iyunv@test fastdfs_install_pkg]# tree -L 1 .
.
├── client.conf
├── fastdfs-nginx-module_v1.16.tar.gz
├── FastDFS_v5.05.tar.gz
├── http.conf
├── install_fastdfs.sh
├── libfastcommon-master.zip
├── mime.types
├── mod_fastdfs.conf
├── nginx-1.8.0.tar.gz
├── nginx.conf
├── README.txt
├── storage.conf
└── tracker.conf
|
其中, 里面几个.conf文件用户需要根据自身的部署环境进行相应的修改.
在需要部署fastdfs的所有节点下, cd到fastdfs_install_pkg目录下,执行./install_fastdfs.sh即可完成安装.以下是./install_fastdfs.sh的内容:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
| [iyunv@test fastdfs_install_pkg]# cat install_fastdfs.sh
#!/bin/bash
#
#set -e
#fastdfs安装包,依赖包所在目录
base_src_path=/usr/local/src/fastdfs_install_pkg
required1=libfastcommon-master.zip
required2=FastDFS_v5.05.tar.gz
required3=nginx-1.8.0.tar.gz
required4=fastdfs-nginx-module_v1.16.tar.gz
#指定nginx的安装路径,启动用户和组
nginx_base_dir=/usr/local/nginx
nginx_user=www
nginx_group=www
#指定fastdfs启动的用户和组
fdfs_user=fdfs
fdfs_group=fdfs
#指定fastdfs的数据目录路径
fdfs_data_path=/data/fdfs
#指定storage组名
storage_groupname=group1
#指定storage路径名
storage_pathname=("html1" "html2" "html3" "html4" "html5")
index=0
errors=()
#check required
cd $base_src_path
[ -f $required1 ] || (echo "$require1 is require, but not found" && exit 1)
[ -f $required2 ] || (echo "$require2 is require, but not found" && exit 1)
[ -f $required3 ] || (echo "$require3 is require, but not found" && exit 1)
#install libfastcommon
function install_libfastcommon() {
cd $base_src_path
unzip -qo $required1
cd ${required1%%.*}
./make.sh
./make.sh install
ln -s /usr/lib64/libfastcommon.so /usr/local/lib/libfastcommon.so
ln -s /usr/lib64/libfastcommon.so /usr/lib/libfastcommon.so
ln -s /usr/lib64/libfdfsclient.so /usr/local/lib/libfdfsclient.so
ln -s /usr/lib64/libfdfsclient.so /usr/lib/libfdfsclient.so
}
#install fastdfs
function install_fastdfs() {
cd $base_src_path
id -g $fdfs_group &>/dev/null || groupadd $fdfs_group
id -u $fdfs_user &>/dev/null || useradd -M -r -g $fdfs_group -s /sbin/nologin $fdfs_user
mkdir -p $fdfs_data_path/{tracker,storage,client}
for pathname in ${storage_pathname[@]}; do
mkdir -p $fdfs_data_path/storage/${storage_groupname}/${pathname}
done
chown -R ${fdfs_user}:${fdfs_group} $fdfs_data_path
tar xf $required2
cd ${required2%%_*}
./make.sh
./make.sh install
mkdir -p /etc/fdfs/sample && /bin/mv /etc/fdfs/*.sample /etc/fdfs/sample
/bin/cp ${base_src_path}/*.conf /etc/fdfs
}
#install nginx with fdfs_mod
function install_nginx_with_fdfsmod() {
cd $base_src_path
rpm -q pcre-devel &>/dev/null || yum -y install pcre-devel
id -g $nginx_group || groupadd $nginx_group
id -u $nginx_user || useradd -M -r -g $nginx_group -s /sbin/nologin $nginx_user
tar xf $required3
tar xf $required4
sed -i '/CORE_INCS/ s#=.*#=\"\$CORE_INCS /usr/include/fastdfs /usr/include/fastcommon\"#' ${required4%%_*}/src/config
cd ${required3%%.tar.*}
./configure --prefix=/usr/local/nginx --with-pcre --with-http_ssl_module --add-module=../${required4%%_*}/src/ --user=${nginx_user} --group=${nginx_group}
make -j `cat /proc/cpuinfo | grep processor| wc -l` && make install
/bin/cp ${base_src_path}/nginx.conf /usr/local/nginx/conf
}
function usage() {
echo "Usage: ./$(basename $0) [ fastdfs ] [ nginx ] [ libfastcommon ]"
exit 1
}
if [ $# -eq 0 ]; then
install_libfastcommon
install_fastdfs
install_nginx_with_fdfsmod
[ $? -eq 0 ] && echo -e "...\nfinish" && exit 0
exit 1
fi
for arg in $@; do
case $arg in
"libfastcommon")
install_libfastcommon;;
"fastdfs")
install_fastdfs;;
"nginx")
install_nginx_with_fdfsmod;;
"-h"|"--help")
usage;;
*)
errors[${index}]=$arg
index=$[ ${index} + 1 ]
esac
done
if [ ${#errors[@]} -ne 0 ]; then
for err in ${errors[@]}; do
echo -e "Error: unknown argument $err"
done
usage
fi
|
三. 配置文件说明
tracker.conf:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
| [iyunv@test fastdfs_install_pkg]# cat tracker.conf
disabled=false
bind_addr=
#服务端口
port=22122
connect_timeout=5
network_timeout=5
#tracker的数据目录路径
base_path=/data/fdfs/tracker
max_connections=1536
accept_threads=1
work_threads=24
store_lookup=2
store_group=group1
store_server=0
store_path=0
download_server=0
reserved_storage_space = 4G
log_level=debug
#运行用户和组
run_by_group=fdfs
run_by_user=fdfs
allow_hosts=*
sync_log_buff_interval = 10
check_active_interval = 120
thread_stack_size = 64KB
storage_ip_changed_auto_adjust = true
storage_sync_file_max_delay = 86400
storage_sync_file_max_time = 300
use_trunk_file = false
slot_min_size = 256
slot_max_size = 16MB
trunk_file_size = 64MB
trunk_create_file_advance = false
trunk_create_file_time_base = 02:00
trunk_create_file_interval = 86400
trunk_create_file_space_threshold = 20G
trunk_init_check_occupying = false
trunk_init_reload_from_binlog = false
trunk_compress_binlog_min_interval = 0
use_storage_id = false
storage_ids_filename = storage_ids.conf
id_type_in_filename = ip
store_slave_file_use_link = false
rotate_error_log = false
error_log_rotate_time=00:00
rotate_error_log_size = 0
log_file_keep_days = 30
use_connection_pool = true
connection_pool_max_idle_time = 3600
http.server_port=8080
http.check_alive_interval=30
http.check_alive_type=tcp
http.check_alive_uri=/status.html
|
storage.conf:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
| [iyunv@test fastdfs_install_pkg]# cat storage.conf
disabled=false
#指定存储组的名称
group_name=group1
bind_addr=
client_bind=true
#服务端口
port=23000
connect_timeout=30
network_timeout=30
heart_beat_interval=30
stat_report_interval=60
#storage节点的数据目录
base_path=/data/fdfs/storage
max_connections=1536
buff_size = 256KB
accept_threads=1
work_threads=24
disk_rw_separated = true
disk_reader_threads = 10
disk_writer_threads = 1
sync_wait_msec=50
sync_interval=0
sync_start_time=00:00
sync_end_time=23:59
write_mark_file_freq=500
#指定存储path的个数和路径
store_path_count=5
store_path0=/data/fdfs/storage/group1/image0
store_path1=/data/fdfs/storage/group1/image1
store_path2=/data/fdfs/storage/group1/image2
store_path3=/data/fdfs/storage/group1/image3
store_path4=/data/fdfs/storage/group1/image4
#指定每个存储path中每个目录层级下创建的目录数
subdir_count_per_path=256
#指定所有的tracker server ip和端口
tracker_server=10.1.12.51:22122
tracker_server=10.1.12.52:22122
log_level=debug
#指定运行用户和组
run_by_group=fdfs
run_by_user=fdfs
allow_hosts=*
file_distribute_path_mode=0
file_distribute_rotate_count=100
fsync_after_written_bytes=0
sync_log_buff_interval=10
sync_binlog_buff_interval=10
sync_stat_file_interval=300
thread_stack_size=512KB
upload_priority=10
if_alias_prefix=
check_file_duplicate=0
file_signature_method=hash
key_namespace=FastDFS
keep_alive=0
use_access_log = false
rotate_access_log = false
access_log_rotate_time=00:00
rotate_error_log = false
error_log_rotate_time=00:00
rotate_access_log_size = 0
rotate_error_log_size = 0
log_file_keep_days = 0
file_sync_skip_invalid_record=false
use_connection_pool = true
connection_pool_max_idle_time = 3600
http.domain_name=
http.server_port=8888
|
client.conf :
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
| [iyunv@test fastdfs_install_pkg]# cat client.conf
connect_timeout=5
network_timeout=5
#指定client的数据目录
base_path=/data/fdfs/client
#指定tracker server的ip和端口
tracker_server=10.1.11.35:22122
tracker_server=10.1.11.36:22122
log_level=info
use_connection_pool = true
connection_pool_max_idle_time = 3600
load_fdfs_parameters_from_tracker=false
use_storage_id = false
storage_ids_filename = storage_ids.conf
http.tracker_server_port=80
|
mod_fastdfs.conf :
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
| [iyunv@test fastdfs_install_pkg]# egrep -v '^$|^#' mod_fastdfs.conf
connect_timeout=2
network_timeout=30
base_path=/tmp
load_fdfs_parameters_from_tracker=false
storage_sync_file_max_delay = 86400
use_storage_id = false
storage_ids_filename = storage_ids.conf
#指定tracker server的ip
tracker_server=192.168.124.43:22122
tracker_server=192.168.124.44:22122
#指定storage server的端口,storage group,path
storage_server_port=23000
group_name=vol1
url_have_group_name = false
store_path_count=5
store_path0=/data/fdfs/storage/group1/image0
store_path1=/data/fdfs/storage/group1/image1
store_path2=/data/fdfs/storage/group1/image2
store_path3=/data/fdfs/storage/group1/image3
store_path4=/data/fdfs/storage/group1/image4
log_level=debug
log_filename=
response_mode=proxy
if_alias_prefix=
flv_support = true
flv_extension = flv
|
nginx.conf:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
| [iyunv@test fastdfs_install_pkg]# egrep -v '^[[:space:]]*#|^$' nginx.conf
user www www;
worker_processes 24;
pid logs/nginx.pid;
events {
worker_connections 51200;
}
http {
include mime.types;
default_type application/octet-stream;
sendfile on;
keepalive_timeout 65;
server {
listen 8090;
server_name localhost;
expires 5m;
add_header Cache-Control 300;
location /group1/M00 {
root /data/fdfs/storage/group1/image0/data;
ngx_fastdfs_module;
}
location /group1/M01 {
root /data/fdfs/storage/group1/image1/data;
ngx_fastdfs_module;
}
location /group1/M02 {
root /data/fdfs/storage/group1/image2/data;
ngx_fastdfs_module;
}
location /group1/M03 {
root /data/fdfs/storage/group1/image3/data;
ngx_fastdfs_module;
}
location /group1/M04 {
root /data/fdfs/storage/group1/image4/data;
ngx_fastdfs_module;
}
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root html;
}
}
}
|
tengine中的fastdfs代理配置:
upstream部分:
1
2
3
4
5
| upstream imagefdfs {
check interval=500 rise=2 fall=5 timeout=3000 type=tcp;
server 10.1.11.35:8090 max_fails=10 fail_timeout=10s;
server 10.1.11.36:8090 max_fails=10 fail_timeout=10s;
}
|
vhost和url location配置部分:
1
2
3
4
5
6
7
8
9
10
11
12
13
| server
{
listen 80;
server_name image.test.com;
index index.html index.htm index.php;
charset utf-8;
expires 5m;
add_header Cache-Control 300;
location ~ ^/group[0-9]+/M[0-9]+/.*\.(jpeg|jpg|gif|png)$ {
proxy_pass http://imagefdfs;
}
access_log /data/nginx/logs/image.access.log;
}
|
四. 服务启动运行:
通过前面的安装, fastdfs的配置文件都位于/etc/fdfs目录下:
[iyunv@test fdfs]# tree .
/etc/fdfs
├── client.conf
├── http.conf
├── mime.types
├── mod_fastdfs.conf
├── sample
│ ├── client.conf.sample
│ ├── storage.conf.sample
│ └── tracker.conf.sample
├── storage.conf
└── tracker.conf
启动trackerd server:
$(which fdfs_trackerd) /etc/tracker.conf start
启动storage的server:
$(which fdfs_storaged) /etc/storage.conf start
将其做成sysv启动脚本如下:
/etc/init.d/trackerd
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
| [iyunv@test fdfs]# cat /etc/init.d/fdfs_trackerd
#!/bin/bash
#
# fdfs_trackerd Starts fdfs_trackerd
#
#
# chkconfig: 2345 99 01
# description: FastDFS tracker server
### BEGIN INIT INFO
# Provides: $fdfs_trackerd
### END INIT INFO
# Source function library.
. /etc/init.d/functions
PRG=/usr/local/bin/fdfs_trackerd
CONF=/etc/fdfs/tracker.conf
if [ ! -f $PRG ]; then
echo "file $PRG does not exist!"
exit 2
fi
if [ ! -f /usr/bin/stop.sh ]; then
echo "file /usr/bin/stop.sh does not exist!"
exit 2
fi
if [ ! -f /usr/bin/restart.sh ]; then
echo "file /usr/bin/restart.sh does not exist!"
exit 2
fi
if [ ! -f $CONF ]; then
echo "file $CONF does not exist!"
exit 2
fi
CMD="$PRG $CONF"
RETVAL=0
start() {
echo -n $"Starting FastDFS tracker server: "
$CMD &
RETVAL=$?
echo
return $RETVAL
}
stop() {
/usr/bin/stop.sh $CMD
RETVAL=$?
return $RETVAL
}
rhstatus() {
status fdfs_trackerd
}
restart() {
/usr/bin/restart.sh $CMD &
}
case "$1" in
start)
start
;;
stop)
stop
;;
status)
rhstatus
;;
restart|reload)
restart
;;
condrestart)
restart
;;
*)
echo $"Usage: $0 {start|stop|status|restart|condrestart}"
exit 1
esac
exit $?
|
/etc/init.d/fdfs_storaged
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
| [iyunv@test fdfs]# cat /etc/init.d/fdfs_storaged
#!/bin/bash
#
# fdfs_storaged Starts fdfs_storaged
#
#
# chkconfig: 2345 99 01
# description: FastDFS storage server
### BEGIN INIT INFO
# Provides: $fdfs_storaged
### END INIT INFO
# Source function library.
. /etc/init.d/functions
PRG=/usr/local/bin/fdfs_storaged
CONF=/etc/fdfs/storage.conf
if [ ! -f $PRG ]; then
echo "file $PRG does not exist!"
exit 2
fi
if [ ! -f /usr/bin/stop.sh ]; then
echo "file /usr/bin/stop.sh does not exist!"
exit 2
fi
if [ ! -f /usr/bin/restart.sh ]; then
echo "file /usr/bin/restart.sh does not exist!"
exit 2
fi
if [ ! -f $CONF ]; then
echo "file $CONF does not exist!"
exit 2
fi
CMD="$PRG $CONF"
RETVAL=0
start() {
echo -n $"Starting FastDFS storage server: "
$CMD &
RETVAL=$?
echo
return $RETVAL
}
stop() {
/usr/bin/stop.sh $CMD
RETVAL=$?
return $RETVAL
}
rhstatus() {
status fdfs_storaged
}
restart() {
/usr/bin/restart.sh $CMD &
}
case "$1" in
start)
start
;;
stop)
stop
;;
status)
rhstatus
;;
restart|reload)
restart
;;
condrestart)
restart
;;
*)
echo $"Usage: $0 {start|stop|status|restart|condrestart}"
exit 1
esac
exit $?
|
chmod +x /etc/init.d/{fdfs_storaged,fdfs_trackerd}
添加为开机自动启动:
chkconfig --add fdfs_storaged
chkconfig --add fdfs_trackerd
chkconfig fdfs_storaged on
chkconfig fdfs_trackerd on
五. 测试fastdfs
上传文件:
1
2
3
| Usage: fdfs_upload_file <config_file> <local_filename> [storage_ip:port] [store_path_index]
[iyunv@test fdfs]# fdfs_upload_file /etc/fdfs/client.conf /etc/inittab
group1/M00/55/1E/CgELJFZSoqCAAj2KAAADdDn5EZ87778834
|
监控fastdfs状态:
1
| [iyunv@test fdfs]# fdfs_monitor /etc/fdfs/client.conf list
|
|
|
QQ群⑧：
窥视卡
雷达卡
发表于 2015-11-24 09:16:41
提升卡
置顶卡
沉默卡
喧嚣卡
变色卡
千斤顶
显身卡