tomcat(2)
mode_jk做tomcat负载均衡http://s5.运维网.com/wyfs02/M02/77/EE/wKiom1ZxNPOyQ4mmAABwFMZ2TBw454.jpg
# yum install httpd-devel -y
# ifconfig
eth0 Link encap:EthernetHWaddr 00:0C:29:9F:1F:E5
inet addr:192.168.112.200Bcast:192.168.112.255Mask:255.255.255.0
inet6 addr: fe80::20c:29ff:fe9f:1fe5/64 Scope:Link
UP BROADCAST RUNNING MULTICASTMTU:1500Metric:1
RX packets:238528 errors:0 dropped:0 overruns:0 frame:0
TX packets:88185 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:246189087 (234.7 MiB)TX bytes:17095170 (16.3 MiB)
eth1 Link encap:EthernetHWaddr 00:0C:29:9F:1F:EF
inet addr:172.168.254.200Bcast:172.168.254.255Mask:255.255.255.0
inet6 addr: fe80::20c:29ff:fe9f:1fef/64 Scope:Link
UP BROADCAST RUNNING MULTICASTMTU:1500Metric:1
RX packets:176 errors:0 dropped:0 overruns:0 frame:0
TX packets:20 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:17742 (17.3 KiB)TX bytes:2760 (2.6 KiB)
lo Link encap:Local Loopback
inet addr:127.0.0.1Mask:255.0.0.0
inet6 addr: ::1/128 Scope:Host
UP LOOPBACK RUNNINGMTU:65536Metric:1
RX packets:146763 errors:0 dropped:0 overruns:0 frame:0
TX packets:146763 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:65389907 (62.3 MiB)TX bytes:65389907 (62.3 MiB)
部署192.168.112.130的tomcat
# ls
anaconda-ks.cfginstall.log.syslog公共的视频文档音乐
install.log ntpupdate.sh 模板 图片下载桌面
You have new mail in /var/spool/mail/root
# ls
anaconda-ks.cfg install.log jdk-7u67-linux-x64.rpm公共的视频文档音乐
apache-tomcat-8.0.30.tar.gzinstall.log.syslogntpupdate.sh 模板 图片下载桌面
You have new mail in /var/spool/mail/root
# rpm -ivh jdk-7u67-linux-x64.rpm
Preparing... ###########################################
1:jdk ###########################################
Unpacking JAR files...
rt.jar...
jsse.jar...
charsets.jar...
tools.jar...
localedata.jar...
jfxrt.jar...
# vim /etc/profile.d/java.sh
1 export JAVA_HOME=/usr/java/latest
2 export PATH=$JAVA_HOME/bin:$PATH
# java -version
java version "1.7.0_67"
Java(TM) SE Runtime Environment (build 1.7.0_67-b01)
Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode)
# ls
anaconda-ks.cfg install.log jdk-7u67-linux-x64.rpm公共的视频文档音乐
apache-tomcat-8.0.30.tar.gzinstall.log.syslogntpupdate.sh 模板 图片下载桌面
# tar xf apache-tomcat-8.0.30.tar.gz -C /usr/local/
You have new mail in /var/spool/mail/root
# cd /usr/local/
# ls
apache-tomcat-8.0.30binetcgamesincludeliblib64libexecsbinsharesrc
# ln -sv apache-tomcat-8.0.30/ tomcat-----------------------注明：在生产环境中都要需要改变文件属组或者属主，这样就不要需要使用root用户
"tomcat" -> "apache-tomcat-8.0.30/"
# ll
总用量 44
drwxr-xr-x. 9 root root 4096 12月 16 19:18 apache-tomcat-8.0.30
drwxr-xr-x. 2 root root 4096 9月23 2011 bin
drwxr-xr-x. 2 root root 4096 9月23 2011 etc
drwxr-xr-x. 2 root root 4096 9月23 2011 games
drwxr-xr-x. 2 root root 4096 9月23 2011 include
drwxr-xr-x. 2 root root 4096 9月23 2011 lib
drwxr-xr-x. 2 root root 4096 9月23 2011 lib64
drwxr-xr-x. 2 root root 4096 9月23 2011 libexec
drwxr-xr-x. 2 root root 4096 9月23 2011 sbin
drwxr-xr-x. 5 root root 4096 8月25 22:04 share
drwxr-xr-x. 2 root root 4096 9月23 2011 src
lrwxrwxrwx. 1 root root 21 12月 16 19:18 tomcat -> apache-tomcat-8.0.30/
You have new mail in /var/spool/mail/root
# cd tomcat/webapps/
# mkdir -pv testapp/{WEB-INF,classes,lib,META-INF}
mkdir: 已创建目录 "testapp"
mkdir: 已创建目录 "testapp/WEB-INF"
mkdir: 已创建目录 "testapp/classes"
mkdir: 已创建目录 "testapp/lib"
mkdir: 已创建目录 "testapp/META-INF"
You have new mail in /var/spool/mail/root
# vim testapp/index.jsp
1
2
3 TomcatA
4
5 TomcatA.zye.com
6
7
8 Session ID
9
10
11
12
13 Created on
14
15
16
17
18
# vim /etc/profile.d/tomcat.sh
1 export CATALINA_HOME=/usr/local/tomcat
2 export PATH=$CATALINA_HOME/bin:$PATH
# catalina.sh version
Using CATALINA_BASE: /usr/local/tomcat
Using CATALINA_HOME: /usr/local/tomcat
Using CATALINA_TMPDIR: /usr/local/tomcat/temp
Using JRE_HOME: /usr/java/latest
Using CLASSPATH: /usr/local/tomcat/bin/bootstrap.jar:/usr/local/tomcat/bin/tomcat-juli.jar
Server version: Apache Tomcat/8.0.30
Server built: Dec 1 2015 22:30:46 UTC
Server number:8.0.30.0
OS Name: Linux
OS Version: 2.6.32-573.el6.x86_64
Architecture: amd64
JVM Version: 1.7.0_67-b01
JVM Vendor: Oracle Corporation
# catalina.sh start
Using CATALINA_BASE: /usr/local/tomcat
Using CATALINA_HOME: /usr/local/tomcat
Using CATALINA_TMPDIR: /usr/local/tomcat/temp
Using JRE_HOME: /usr/java/latest
Using CLASSPATH: /usr/local/tomcat/bin/bootstrap.jar:/usr/local/tomcat/bin/tomcat-juli.jar
Tomcat started.
You have new mail in /var/spool/mail/root
# ss -tnl
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 1 ::ffff:127.0.0.1:8005 :::*
LISTEN 0 100 :::8009 :::*
LISTEN 0 100 :::8080 :::*
LISTEN 0 128 :::80 :::*
LISTEN 0 128 :::22 :::*
LISTEN 0 128 *:22 *:*
LISTEN 0 128 127.0.0.1:631 *:*
LISTEN 0 128 ::1:631 :::*
LISTEN 0 100 ::1:25 :::*
LISTEN 0 100 127.0.0.1:25 *:*
LISTEN 0 128 127.0.0.1:6010 *:*
LISTEN 0 128 ::1:6010 :::*
http://s3.运维网.com/wyfs02/M00/77/EE/wKioL1ZxT6PyTIfWAAECgXk9LGo107.jpg
http://s3.运维网.com/wyfs02/M02/77/EF/wKiom1ZxT52SULlUAAQBT-8bZHY002.jpg
# vim /etc/httpd/conf/httpd.conf
292 #DocumentRoot "/var/www/html"
990 NameVirtualHost *:80
# vim /etc/httpd/conf.d/ajp-tomcat.conf
1
2 ServerName www.zye.com
3 ProxyVia On
4 ProxyRequests Off
5 ProxyPreserveHost On
6
7 Order deny,allow
8 Allow from all
9
10 ProxyPass /status !
11 ProxyPass / ajp://192.168.112.130:8009/
12 ProxyPa***everse / ajp://192.168.112.130:8009/
13
14 Order deny,allow
15 Allow from all
16
17
# httpd -t
Syntax OK
# service httpd restart
停止 httpd： [确定]
正在启动 httpd： [确定]
http://s5.运维网.com/wyfs02/M01/77/F7/wKioL1ZyZvmxFXzQAAMSTKLk44g799.jpg
http://s5.运维网.com/wyfs02/M01/77/F9/wKiom1ZyZ9ixvGtcAADGkwLnmJw412.jpg
mode_jk安装（第三方）
# ls
anaconda-ks.cfg install.log.syslog 公共的
apache-tomcat-8.0.30.tar.gz JavaCenter_Home_2.0_GBK 模板
apr-1.4.6 JavaCenter_Home_2.0_GBK.tar.bz2 视频
apr-1.4.6.tar.bz2 jdk-7u67-linux-x64.rpm 说明.htm
apr-util-1.4.1.tar.bz2 mariadb-10.0.10-linux-x86_64.tar.gz图片
connect-web.war nginx-1.8.0-1.el6.ngx.x86_64.rpm 文档
epel-release-latest-6.noarch.rpmshopxx-a5-Beta.zip 下载
httpd-2.4.6.tar.bz2 shopxx-v3.0-Beta 音乐
install.log tomcat-connectors-1.2.40-src.tar.gz桌面
# tar xf tomcat-connectors-1.2.40-src.tar.gz
# cd tomcat-connectors-1.2.40-src
# ls
confdocsHOWTO-RELEASE.txtjkstatusLICENSEnativeNOTICEREADME.txtsupporttoolsxdocs
# cd native/
# ls
aclocal.m4buildconf.shconfigure iis netscape STATUS.txt
apache-1.3BUILDING.txtconfigure.acMakefile.amREADME.txtTODO.txt
apache-2.0common docs Makefile.inscripts
# less README.txt
# less BUILDING.txt
# whereis apxs
apxs: /usr/sbin/apxs /usr/share/man/man8/apxs.8.gz
# ./configure --with-apxs=/usr/sbin/apxs
# make && make install
# cd /usr/lib64/httpd/modules/
# ls
mod_actions.so mod_authz_user.somod_filter.so mod_proxy_http.so
mod_alias.so mod_autoindex.so mod_headers.so mod_proxy_scgi.so
mod_asis.so mod_cache.so mod_ident.so mod_proxy.so
mod_auth_basic.so mod_cern_meta.so mod_include.so mod_reqtimeout.so
mod_auth_digest.so mod_cgid.so mod_info.so mod_rewrite.so
mod_authn_alias.so mod_cgi.so mod_jk.so mod_setenvif.so
mod_authn_anon.so mod_dav_fs.so mod_ldap.so mod_speling.so
mod_authn_dbd.so mod_dav.so mod_log_config.so mod_status.so
mod_authn_dbm.so mod_dbd.so mod_log_forensic.so mod_substitute.so
mod_authn_default.so mod_deflate.so mod_logio.so mod_suexec.so
mod_authn_file.so mod_dir.so mod_mime_magic.so mod_unique_id.so
mod_authnz_ldap.so mod_disk_cache.somod_mime.so mod_userdir.so
mod_authz_dbm.so mod_dnssd.so mod_negotiation.so mod_usertrack.so
mod_authz_default.so mod_dumpio.so mod_proxy_ajp.so mod_version.so
mod_authz_groupfile.somod_env.so mod_proxy_balancer.somod_vhost_alias.so
mod_authz_host.so mod_expires.so mod_proxy_connect.so
mod_authz_owner.so mod_ext_filter.somod_proxy_ftp.so
# mv /etc/httpd/conf.d/
ajp-tomcat.conf mod_dnssd.conf welcome.conf
http-tomcat.conf.bakREADME
# mv /etc/httpd/conf.d/ajp-tomcat.conf{,.bak}
# cd /etc/httpd/conf.d/
# ls
ajp-tomcat.conf.bakhttp-tomcat.conf.bakmod_dnssd.confREADMEwelcome.conf
# vim mod_jk.conf
1 LoadModule jk_module modules/mod_jk.so
# httpd -M | grep jk
NameVirtualHost *:80 has no VirtualHosts
Syntax OK
jk_module (shared)
# vim /etc/httpd/conf.d/mod_jk.conf
1 LoadModule jk_module modules/mod_jk.so
2 JkWorkersFile /etc/httpd/conf.d/workers.properties
3 JkLogFile logs/mod_jk.log
4 JkLogLevel info
5 JkMount /status !
6 JkMount /* TomcatA
7 JkMount /status StatA
# vim /etc/httpd/conf.d/workers.properties
1 worker.list=TomcatA,StatA
2 worker.TomcatA.host=192.168.112.130
3 worker.TomcatA.port=8009
4 worker.TomcatA.type=ajp13
5 worker.StatA.type=status
# httpd -t
NameVirtualHost *:80 has no VirtualHosts
Syntax OK
# service httpd restart
停止 httpd： [确定]
正在启动 httpd： NameVirtualHost *:80 has no VirtualHosts
[确定]
http://s5.运维网.com/wyfs02/M00/77/FB/wKioL1ZyplWyKkweAAUtMAHYI1A446.jpg
http://s1.运维网.com/wyfs02/M00/77/FC/wKiom1ZypofDm2ojAADjPpbdZx0048.jpg
# tail /var/log/httpd/mod_jk.log
init_jk::mod_jk.c (3383): mod_jk/1.2.40 initialized
init_jk::mod_jk.c (3383): mod_jk/1.2.40 initialized
#
Tomcat负载均衡
主机112.130；112.140；0.20（112.200）
112.130
# scp /etc/profile.d/{java.sh,tomcat.sh} 192.168.112.140:/etc/profile.d/
The authenticity of host '192.168.112.140 (192.168.112.140)' can't be established.
RSA key fingerprint is 0e:95:8d:de:b9:2f:c4:75:8d:70:af:e2:84:65:7f:86.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.112.140' (RSA) to the list of known hosts.
root@192.168.112.140's password:
java.sh 100% 67 0.1KB/s 00:00
tomcat.sh 100% 76 0.1KB/s 00:00
You have new mail in /var/spool/mail/root
# scp apache-tomcat-8.0.30.tar.gz jdk-7u67-linux-x64.rpm 192.168.112.140:/root
root@192.168.112.140's password:
apache-tomcat-8.0.30.tar.gz 100% 8936KB 8.7MB/s 00:00
jdk-7u67-linux-x64.rpm 100%121MB20.2MB/s 00:06
You have new mail in /var/spool/mail/root
112.140
# scp /etc/profile.d/{java.sh,tomcat.sh} 192.168.112.140:/etc/profile.d/
The authenticity of host '192.168.112.140 (192.168.112.140)' can't be established.
RSA key fingerprint is 0e:95:8d:de:b9:2f:c4:75:8d:70:af:e2:84:65:7f:86.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.112.140' (RSA) to the list of known hosts.
root@192.168.112.140's password:
java.sh 100% 67 0.1KB/s 00:00
tomcat.sh 100% 76 0.1KB/s 00:00
You have new mail in /var/spool/mail/root
# scp apache-tomcat-8.0.30.tar.gz jdk-7u67-linux-x64.rpm 192.168.112.140:/root
root@192.168.112.140's password:
apache-tomcat-8.0.30.tar.gz 100% 8936KB 8.7MB/s 00:00
jdk-7u67-linux-x64.rpm 100%121MB20.2MB/s 00:06
You have new mail in /var/spool/mail/root
# cd /usr/local/
# ls
apache-tomcat-8.0.30binetcgamesincludeliblib64libexecsbinsharesrc
# ln -sv apache-tomcat-8.0.30/ tomcat
"tomcat" -> "apache-tomcat-8.0.30/"
# ls
apache-tomcat-8.0.30binetcgamesincludeliblib64libexecsbinsharesrctomcat
# cd tomcat/
# ls
binconflibLICENSElogsNOTICERELEASE-NOTESRUNNING.txttempwebappswork
# ls
docsexampleshost-managermanagerROOT
# mkdir -pv testapp/{classes,lib,WEB-INF,META-INF}
mkdir: 已创建目录 "testapp"
mkdir: 已创建目录 "testapp/classes"
mkdir: 已创建目录 "testapp/lib"
mkdir: 已创建目录 "testapp/WEB-INF"
mkdir: 已创建目录 "testapp/META-INF"
# ls
docsexampleshost-managermanagerROOTtestapp
You have new mail in /var/spool/mail/root
# ll
总用量 24
drwxr-xr-x. 14 root root 4096 12月 17 21:16 docs
drwxr-xr-x.6 root root 4096 12月 17 21:16 examples
drwxr-xr-x.5 root root 4096 12月 17 21:16 host-manager
drwxr-xr-x.5 root root 4096 12月 17 21:16 manager
drwxr-xr-x.3 root root 4096 12月 17 21:16 ROOT
drwxr-xr-x.6 root root 4096 12月 17 21:19 testapp
# ll testapp/
总用量 16
drwxr-xr-x. 2 root root 4096 12月 17 21:19 classes
drwxr-xr-x. 2 root root 4096 12月 17 21:19 lib
drwxr-xr-x. 2 root root 4096 12月 17 21:19 META-INF
drwxr-xr-x. 2 root root 4096 12月 17 21:19 WEB-INF
# vim testapp/index.jsp
TomcatB
TomcatB.zye.com
Session ID
Created on
# . /etc/profile.d/tomcat.sh
# vim /usr/local/tomcat/conf/server.xml
103
# catalina.sh version
Using CATALINA_BASE: /usr/local/tomcat
Using CATALINA_HOME: /usr/local/tomcat
Using CATALINA_TMPDIR: /usr/local/tomcat/temp
Using JRE_HOME: /usr/java/latest
Using CLASSPATH: /usr/local/tomcat/bin/bootstrap.jar:/usr/local/tomcat/bin/tomcat-juli.jar
Server version: Apache Tomcat/8.0.30
Server built: Dec 1 2015 22:30:46 UTC
Server number:8.0.30.0
OS Name: Linux
OS Version: 2.6.32-573.el6.x86_64
Architecture: amd64
JVM Version: 1.7.0_67-b01
JVM Vendor: Oracle Corporation
# catalina.sh start
Using CATALINA_BASE: /usr/local/tomcat
Using CATALINA_HOME: /usr/local/tomcat
Using CATALINA_TMPDIR: /usr/local/tomcat/temp
Using JRE_HOME: /usr/java/latest
Using CLASSPATH: /usr/local/tomcat/bin/bootstrap.jar:/usr/local/tomcat/bin/tomcat-juli.jar
Tomcat started.
# ss -tnl
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 1 ::ffff:127.0.0.1:8005 :::*
LISTEN 0 100 :::8009 :::*
LISTEN 0 128 :::36751 :::*
LISTEN 0 128 :::111 :::*
LISTEN 0 128 *:111 *:*
LISTEN 0 100 :::8080 :::*
LISTEN 0 128 *:44177 *:*
LISTEN 0 128 :::22 :::*
LISTEN 0 128 *:22 *:*
LISTEN 0 128 127.0.0.1:631 *:*
LISTEN 0 128 ::1:631 :::*
LISTEN 0 100 ::1:25 :::*
LISTEN 0 100 127.0.0.1:25 *:*
LISTEN 0 128 127.0.0.1:6010 *:*
LISTEN 0 128 ::1:6010 :::*
http://s4.运维网.com/wyfs02/M02/77/FB/wKioL1ZyufaxUJz3AAEBB8sko-s995.jpg
http://s4.运维网.com/wyfs02/M00/77/FB/wKioL1Zyufix3NktAALI0m4WtS8766.jpg
112.130
# vim /usr/local/tomcat/conf/server.xml
103
# catalina.sh stop
Using CATALINA_BASE: /usr/local/tomcat
Using CATALINA_HOME: /usr/local/tomcat
Using CATALINA_TMPDIR: /usr/local/tomcat/temp
Using JRE_HOME: /usr/java/latest
Using CLASSPATH: /usr/local/tomcat/bin/bootstrap.jar:/usr/local/tomcat/bin/tomcat-juli.jar
# catalina.sh start
Using CATALINA_BASE: /usr/local/tomcat
Using CATALINA_HOME: /usr/local/tomcat
Using CATALINA_TMPDIR: /usr/local/tomcat/temp
Using JRE_HOME: /usr/java/latest
Using CLASSPATH: /usr/local/tomcat/bin/bootstrap.jar:/usr/local/tomcat/bin/tomcat-juli.jar
Tomcat started.
# ss -tnl
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 100 :::8009 :::*
LISTEN 0 100 :::8080 :::*
LISTEN 0 128 :::22 :::*
LISTEN 0 128 *:22 *:*
LISTEN 0 128 127.0.0.1:631 *:*
LISTEN 0 128 ::1:631 :::*
LISTEN 0 100 ::1:25 :::*
LISTEN 0 100 127.0.0.1:25 *:*
LISTEN 0 128 127.0.0.1:6010 *:*
LISTEN 0 128 ::1:6010 :::*
You have new mail in /var/spool/mail/root
0.20(代理服务器)
# cd /etc/httpd/conf.d/
# vim mod_jk.conf
1 LoadModule jk_module modules/mod_jk.so
2 JkWorkersFile /etc/httpd/conf.d/workers.properties
3 JkLogFile logs/mod_jk.log
4 JkLogLevel info
5 JkMount /status !
6 JkMount /* TomcatB
7 JkMount /status StatA
~
# vim workers.properties
1 worker.list=TomcatB,StatA
2 worker.TomcatB.host=192.168.112.140
3 worker.TomcatB.port=8009
4 worker.TomcatB.type=ajp13
5 worker.StatA.type=status
~
# service httpd reload
重新载入 httpd：
http://s4.运维网.com/wyfs02/M00/77/FD/wKiom1ZyvO7xGul_AAD_qmBcX-Y189.jpg
http://s1.运维网.com/wyfs02/M01/77/FB/wKioL1ZyvPrzPstOAAJOp83Mb8M098.jpg
0.20（负载）
# vim mod_jk.conf
1 LoadModule jk_module modules/mod_jk.so
2 JkWorkersFile /etc/httpd/conf.d/workers.properties
3 JkLogFile logs/mod_jk.log
4 JkLogLevel debug
5 JkMount /status !
6 JkMount /* tcsrvs
7 JkMount /status StatA
# vim workers.properties
1 worker.list=tcsrvs,StatA
2 worker.TomcatB.host=192.168.112.140
3 worker.TomcatB.port=8009
4 worker.TomcatB.type=ajp13
5 worker.TomcatB.lbfactor=1
6 worker.TomcatA.host=192.168.112.130
7 worker.TomcatA.port=8009
8 worker.TomcatA.type=ajp13
9 worker.TomcatA.lbfactor=1
10 worker.StatA.type=status
11 worker.tcsrvs.type=lb
12 worker.tcsrvs.balance_workers=TomcatA,TomcatB---------跟之前调用要匹配
# !ser
service httpd reload
重新载入 httpd：
以下基于session绑定
http://s4.运维网.com/wyfs02/M01/77/FD/wKiom1ZywQSRw2tXAADx6uDUagY756.jpg
# vim workers.properties
1 worker.list=tcsrvs,StatA
2 worker.TomcatB.host=192.168.112.140
3 worker.TomcatB.port=8009
4 worker.TomcatB.type=ajp13
5 worker.TomcatB.lbfactor=1
6 worker.TomcatA.host=192.168.112.130
7 worker.TomcatA.port=8009
8 worker.TomcatA.type=ajp13
9 worker.TomcatA.lbfactor=1
10 worker.StatA.type=status
11 worker.tcsrvs.type=lb
12 worker.tcsrvs.balance_workers=TomcatA,TomcatB
13 worker.tcsrvs.sticky_session=0--------基于浏览器做粘性，一般为1
# !ser
service httpd reload
重新载入 httpd：
http://s1.运维网.com/wyfs02/M02/77/FD/wKiom1ZywfXyTh0vAAD-APM8GS8624.jpg
http://s2.运维网.com/wyfs02/M00/77/FB/wKioL1ZywiHxnegNAAD_MM_6DGE840.jpg
# !tail
tail /var/log/httpd/mod_jk.log ------工作环境中不会开启debug级别
ajp_connection_tcp_get_message::jk_ajp_common.c (1403): 03d0 FF FF FF FF FF FF FF FF FF FF FF FF FF FF 00 00- ................
ajp_connection_tcp_get_message::jk_ajp_common.c (1403): 03e0 00 00 00 00 FF FF FF FF FF FF FF FF FF FF FF FF- ................
ajp_connection_tcp_get_message::jk_ajp_common.c (1403): 03f0 FF FF FF FC FC FC 5F 5F 5F C5 C5 C5 FF FF FF 9F- ......___.......
ws_write::mod_jk.c (552): written 5262 out of 5262
ajp_connection_tcp_get_message::jk_ajp_common.c (1403): received from ajp13 pos=0 len=2 max=8192
ajp_connection_tcp_get_message::jk_ajp_common.c (1403): 0000 05 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00- ................
ajp_process_callback::jk_ajp_common.c (2054): AJP13 protocol: Reuse is OK
ajp_reset_endpoint::jk_ajp_common.c (810): (TomcatA) resetting endpoint with socket 14
ajp_done::jk_ajp_common.c (3144): recycling connection pool for worker TomcatA and socket 14
jk_handler::mod_jk.c (2790): Service finished with status=200 for worker=tcsrvs
http://s3.运维网.com/wyfs02/M01/77/FB/wKioL1Zyx76CvfpmAAao8fhsal4606.jpg
基于mod_porxy做负载均衡----接上
# mv mod_jk.conf mod_jk.conf.bak
# ls
ajp-tomcat.conf.bakhttp-tomcat.conf.bakmod_dnssd.confmod_jk.conf.bakREADMEwelcome.confworkers.properties
# mv ajp-tomcat.conf.bak mod_proxy.conf
# vim mod_proxy.conf
1
2 BalancerMember http://192.168.112.130:8080 loadfactor=1
3 BalancerMember http://192.168.112.140:8080 loadfactor=1
4
5
6
7 ServerName www.zye.com
8 ProxyVia On
9 ProxyRequests Off
10 ProxyPreserveHost On
11
12 Order deny,allow
13 Allow from all
14
15 ProxyPass /status !
16 ProxyPass / balancer://tcsrvs/
17 ProxyPa***everse / balancer://tcsrvs/
18
19 Order deny,allow
20 Allow from all
21
22
# httpd -t
Syntax OK
# httpd -t
Syntax OK
# service httpd start
正在启动 httpd： [确定]
http://s3.运维网.com/wyfs02/M02/77/FD/wKiom1ZyzVSScWrXAAD4oC9JgZU697.jpg
根据请求来调度
# vim mod_proxy.conf
1
2 BalancerMember http://192.168.112.130:8080 loadfactor=1
3 BalancerMember http://192.168.112.140:8080 loadfactor=1
4 ProxySet lbmethod=byrequests
5
6
7 ServerName www.zye.com
8 ProxyVia On
9 ProxyRequests Off
10 ProxyPreserveHost On
11
12 Order deny,allow
13 Allow from all
14
15 ProxyPass /status !
16 ProxyPass / balancer://tcsrvs/
17 ProxyPa***everse / balancer://tcsrvs/
18
19 Order deny,allow
20 Allow from all
21
22
# httpd -t
Syntax OK
# service httpd reload
重新载入 httpd：
http://s2.运维网.com/wyfs02/M01/77/FD/wKiom1ZyzpmARh2KAAD-lOTvZGE456.jpg
http://s2.运维网.com/wyfs02/M00/77/FC/wKioL1ZyzqSy3QMoAAECPpPTnsU297.jpg
根据流量做调度
# vim mod_proxy.conf
1
2 BalancerMember http://192.168.112.130:8080 loadfactor=1
3 BalancerMember http://192.168.112.140:8080 loadfactor=1
4 ProxySet lbmethod=bytraffic
5
6
7 ServerName www.zye.com
8 ProxyVia On
9 ProxyRequests Off
10 ProxyPreserveHost On
11
12 Order deny,allow
13 Allow from all
14
15 ProxyPass /status !
16 ProxyPass / balancer://tcsrvs/
17 ProxyPa***everse / balancer://tcsrvs/
18
19 Order deny,allow
20 Allow from all
21
22
# httpd -t
Syntax OK
# service httpd reload
重新载入 httpd：
http://s2.运维网.com/wyfs02/M02/77/FD/wKiom1Zyzz-DuDCCAADkTiAI6QY546.jpg
http://s4.运维网.com/wyfs02/M02/77/FC/wKioL1Zyz0nwV9h1AADmXJ8UdKk754.jpg
根据主机的繁忙程度
# vim mod_proxy.conf
1
2 BalancerMember http://192.168.112.130:8080 loadfactor=1
3 BalancerMember http://192.168.112.140:8080 loadfactor=1
4 ProxySet lbmethod=bybusyness
5
6
7 ServerName www.zye.com
8 ProxyVia On
9 ProxyRequests Off
10 ProxyPreserveHost On
11
12 Order deny,allow
13 Allow from all
14
15 ProxyPass /status !
16 ProxyPass / balancer://tcsrvs/
17 ProxyPa***everse / balancer://tcsrvs/
18
19 Order deny,allow
20 Allow from all
21
22
# httpd -t
Syntax OK
# service httpd reload
重新载入 httpd：
http://s1.运维网.com/wyfs02/M01/77/FD/wKiom1Zy0PHCQMuRAAD6H91_F6Y485.jpg
http://s1.运维网.com/wyfs02/M01/77/FC/wKioL1Zy0Pvw38E8AADie9iyDFA482.jpg
http修改ajp
# vim mod_proxy.conf
1
2 BalancerMember ajp://192.168.112.130:8009 loadfactor=1
3 BalancerMember ajp://192.168.112.140:8009 loadfactor=1
4 ProxySet lbmethod=bybusyness
5
6
7 ServerName www.zye.com
8 ProxyVia On
9 ProxyRequests Off
10 ProxyPreserveHost On
11
12 Order deny,allow
13 Allow from all
14
15 ProxyPass /status !
16 ProxyPass / balancer://tcsrvs/
17 ProxyPa***everse / balancer://tcsrvs/
18
19 Order deny,allow
20 Allow from all
21
22
# httpd -t
Syntax OK
# service httpd reload
重新载入 httpd：
http://s1.运维网.com/wyfs02/M02/77/FC/wKioL1Zy0fqS3J5LAAD515-CUUs491.jpg
http://s1.运维网.com/wyfs02/M02/77/FD/wKiom1Zy0e-TKv0MAADpVmODG8w855.jpg
会话粘性
# vim mod_proxy.conf
1
2 BalancerMember ajp://192.168.112.130:8009 loadfactor=1
3 BalancerMember ajp://192.168.112.140:8009 loadfactor=1
4 ProxySet lbmethod=bybusyness
5
6
7 ServerName www.zye.com
8 ProxyVia On
9 ProxyRequests Off
10 ProxyPreserveHost On
11
12 Order deny,allow
13 Allow from all
14
15 ProxyPass /status !
16 ProxyPass / balancer://tcsrvs/ stickysession=JSESSIONID
17 ProxyPa***everse / balancer://tcsrvs/ stickysession=JSESSIONID
18
19 Order deny,allow
20 Allow from all
21
22
# httpd -t
Syntax OK
# service httpd reload
重新载入 httpd：
session没有实现
实现状态查看
# vim mod_proxy.conf
1
2 BalancerMember ajp://192.168.112.130:8009 loadfactor=1
3 BalancerMember ajp://192.168.112.140:8009 loadfactor=1
4 ProxySet lbmethod=bybusyness
5
6
7 ServerName www.zye.com
8 ProxyVia On
9 ProxyRequests Off
10 ProxyPreserveHost On
11
12 Order deny,allow
13 Allow from all
14
15 ProxyPass /status !
16 ProxyPass / balancer://tcsrvs/ stickysession=JSESSIONID
17 ProxyPa***everse / balancer://tcsrvs/ stickysession=JSESSIONID
18
19 Order deny,allow
20 Allow from all
21
22
23 SetHandler balancer-manager
24 Proxypass !
25 Order Deny,Allow
26 Allow from all
27
28
# httpd -t
Syntax OK
# service httpd reload
重新载入 httpd：
http://s4.运维网.com/wyfs02/M02/77/FC/wKioL1Zy2lfiZBYaAAOjQrFanYo540.jpg
多播的会话集群
tomcat支持标准会话管理器和持久会话管理器
tomcat默认的标准会话管理器
delta会话管理器，如果是对所有主机，放在引擎内部；否则放入host中，对于单个主机
将两个后端主机（0.130；0.140）tomcat停掉
#
# catalina.sh stop
Using CATALINA_BASE: /usr/local/tomcat
Using CATALINA_HOME: /usr/local/tomcat
Using CATALINA_TMPDIR: /usr/local/tomcat/temp
Using JRE_HOME: /usr/java/latest
Using CLASSPATH: /usr/local/tomcat/bin/bootstrap.jar:/usr/local/tomcat/bin/tomcat-juli.jar
You have new mail in /var/spool/mail/root
# catalina.sh stop
Using CATALINA_BASE: /usr/local/tomcat
Using CATALINA_HOME: /usr/local/tomcat
Using CATALINA_TMPDIR: /usr/local/tomcat/temp
Using JRE_HOME: /usr/java/latest
Using CLASSPATH: /usr/local/tomcat/bin/bootstrap.jar:/usr/local/tomcat/bin/tomcat-juli.jar
You have new mail in /var/spool/mail/root
0.130
# pwd
/usr/local/tomcat/conf
# cp server.xml{,.bak}
# ls
Catalina catalina.propertieslogging.propertiesserver.xml.bak tomcat-users.xsd
catalina.policycontext.xml server.xml tomcat-users.xmlweb.xml
# vim server.xml
123
125
127
128
131
132
133
138
144
145
146
147
148
149
150
151
152
154
155
156
161
162
163
# catalina.sh configtest
Using CATALINA_BASE: /usr/local/tomcat
Using CATALINA_HOME: /usr/local/tomcat
Using CATALINA_TMPDIR: /usr/local/tomcat/temp
Using JRE_HOME: /usr/java/latest
Using CLASSPATH: /usr/local/tomcat/bin/bootstrap.jar:/usr/local/tomcat/bin/tomcat-juli.jar
十二月 19, 2015 1:03:15 上午 org.apache.catalina.startup.VersionLoggerListener log
信息: Server version: Apache Tomcat/8.0.30
十二月 19, 2015 1:03:15 上午 org.apache.catalina.startup.VersionLoggerListener log
信息: Server built: Dec 1 2015 22:30:46 UTC
十二月 19, 2015 1:03:15 上午 org.apache.catalina.startup.VersionLoggerListener log
信息: Server number: 8.0.30.0
十二月 19, 2015 1:03:15 上午 org.apache.catalina.startup.VersionLoggerListener log
信息: OS Name: Linux
十二月 19, 2015 1:03:15 上午 org.apache.catalina.startup.VersionLoggerListener log
信息: OS Version: 2.6.32-573.el6.x86_64
十二月 19, 2015 1:03:15 上午 org.apache.catalina.startup.VersionLoggerListener log
信息: Architecture: amd64
十二月 19, 2015 1:03:15 上午 org.apache.catalina.startup.VersionLoggerListener log
信息: Java Home: /usr/java/jdk1.7.0_67/jre
十二月 19, 2015 1:03:15 上午 org.apache.catalina.startup.VersionLoggerListener log
信息: JVM Version: 1.7.0_67-b01
十二月 19, 2015 1:03:15 上午 org.apache.catalina.startup.VersionLoggerListener log
信息: JVM Vendor: Oracle Corporation
十二月 19, 2015 1:03:15 上午 org.apache.catalina.startup.VersionLoggerListener log
信息: CATALINA_BASE: /usr/local/apache-tomcat-8.0.30
十二月 19, 2015 1:03:15 上午 org.apache.catalina.startup.VersionLoggerListener log
信息: CATALINA_HOME: /usr/local/apache-tomcat-8.0.30
十二月 19, 2015 1:03:16 上午 org.apache.catalina.startup.VersionLoggerListener log
信息: Command line argument: -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
十二月 19, 2015 1:03:16 上午 org.apache.catalina.startup.VersionLoggerListener log
信息: Command line argument: -Djava.endorsed.dirs=/usr/local/tomcat/endorsed
十二月 19, 2015 1:03:16 上午 org.apache.catalina.startup.VersionLoggerListener log
信息: Command line argument: -Dcatalina.base=/usr/local/tomcat
十二月 19, 2015 1:03:16 上午 org.apache.catalina.startup.VersionLoggerListener log
信息: Command line argument: -Dcatalina.home=/usr/local/tomcat
十二月 19, 2015 1:03:16 上午 org.apache.catalina.startup.VersionLoggerListener log
信息: Command line argument: -Djava.io.tmpdir=/usr/local/tomcat/temp
十二月 19, 2015 1:03:16 上午 org.apache.catalina.core.AprLifecycleListener lifecycleEvent
信息: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: /usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
十二月 19, 2015 1:03:17 上午 org.apache.coyote.AbstractProtocol init
信息: Initializing ProtocolHandler ["http-nio-8080"]
十二月 19, 2015 1:03:18 上午 org.apache.tomcat.util.net.NioSelectorPool getSharedSelector
信息: Using a shared selector for servlet write/read
十二月 19, 2015 1:03:18 上午 org.apache.coyote.AbstractProtocol init
信息: Initializing ProtocolHandler ["ajp-nio-8009"]
十二月 19, 2015 1:03:18 上午 org.apache.tomcat.util.net.NioSelectorPool getSharedSelector
信息: Using a shared selector for servlet write/read
十二月 19, 2015 1:03:18 上午 org.apache.catalina.startup.Catalina load
信息: Initialization processed in 6942 ms
You have new mail in /var/spool/mail/root
创建支持集群会话的集群化管理器
一般在webapp文件里的WEB-INF的xml文件
# cd /usr/local/tomcat/webapps/testapp/WEB-INF/
# ls
# cp /usr/local/tomcat/conf/web.xml ./
You have new mail in /var/spool/mail/root
# ls
web.xml
# vim web.xml
You have new mail in /var/spool/mail/root
101
102
103
104
105
106 default
107 org.apache.catalina.servlets.DefaultServlet
108
# scp web.xml 192.168.112.140:/usr/local/tomcat/webapps/testapp/WEB-INF/
root@192.168.112.140's password:
web.xml 100%165KB 164.9KB/s 00:00
You have new mail in /var/spool/mail/root
#
# catalina.sh start
Using CATALINA_BASE: /usr/local/tomcat
Using CATALINA_HOME: /usr/local/tomcat
Using CATALINA_TMPDIR: /usr/local/tomcat/temp
Using JRE_HOME: /usr/java/latest
Using CLASSPATH: /usr/local/tomcat/bin/bootstrap.jar:/usr/local/tomcat/bin/tomcat-juli.jar
Tomcat started.
You have new mail in /var/spool/mail/root
# tail -f logs/catalina.out
# tail -200 logs/catalina.out
19-Dec-2015 01:32:53.595 INFO org.apache.catalina.tribes.membership.McastServiceImpl.setupSocket Setting cluster mcast soTimeout to 500
19-Dec-2015 01:32:53.618 INFO org.apache.catalina.tribes.membership.McastServiceImpl.waitForMembers Sleeping for 1000 milliseconds to establish cluster membership, start level:4
19-Dec-2015 01:32:54.620 INFO org.apache.catalina.tribes.membership.McastServiceImpl.waitForMembers Done sleeping, membership established, start level:4
19-Dec-2015 01:32:54.636 INFO org.apache.catalina.tribes.membership.McastServiceImpl.waitForMembers Sleeping for 1000 milliseconds to establish cluster membership, start level:8
19-Dec-2015 01:32:55.638 INFO org.apache.catalina.tribes.membership.McastServiceImpl.waitForMembers Done sleeping, membership established, start level:8
19-Dec-2015 01:32:55.646 INFO org.apache.catalina.ha.deploy.FarmWarDeployer.start Cluster FarmWarDeployer started.
19-Dec-2015 01:32:55.823 INFO org.apache.catalina.ha.session.JvmRouteBinderValve.startInternal JvmRouteBinderValve started
19-Dec-2015 01:32:59.292 INFO org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-8080"]
19-Dec-2015 01:32:59.331 INFO org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["ajp-nio-8009"]
19-Dec-2015 01:32:59.336 INFO org.apache.catalina.startup.Catalina.start Server startup in 5861 ms
19-Dec-2015 01:35:59.648 INFO org.apache.catalina.tribes.io.BufferPool.getBufferPool Created a buffer pool with max size:104857600 bytes of type: org.apache.catalina.tribes.io.BufferPool15Impl
19-Dec-2015 01:36:00.537 INFO org.apache.catalina.ha.tcp.SimpleTcpCluster.memberAdded Replication member added:org.apache.catalina.tribes.membership.MemberImpl
0.140
# cd /usr/local/tomcat/conf/
You have new mail in /var/spool/mail/root
# ls
Catalina catalina.propertieslogging.propertiestomcat-users.xmlweb.xml
catalina.policycontext.xml server.xml tomcat-users.xsd
# cp server.xml{,.bak}
# ls
Catalina catalina.propertieslogging.propertiesserver.xml.bak tomcat-users.xsd
catalina.policycontext.xml server.xml tomcat-users.xmlweb.xml
# vim server.xml
123
125
127
128
132
133
138
144
145
146
147
148
149
150
151
152
154
155
156
161
162
163
# catalina.sh configtest
Using CATALINA_BASE: /usr/local/tomcat
Using CATALINA_HOME: /usr/local/tomcat
Using CATALINA_TMPDIR: /usr/local/tomcat/temp
Using JRE_HOME: /usr/java/latest
Using CLASSPATH: /usr/local/tomcat/bin/bootstrap.jar:/usr/local/tomcat/bin/tomcat-juli.jar
十二月 19, 2015 1:01:39 上午 org.apache.catalina.startup.VersionLoggerListener log
信息: Server version: Apache Tomcat/8.0.30
十二月 19, 2015 1:01:39 上午 org.apache.catalina.startup.VersionLoggerListener log
信息: Server built: Dec 1 2015 22:30:46 UTC
十二月 19, 2015 1:01:39 上午 org.apache.catalina.startup.VersionLoggerListener log
信息: Server number: 8.0.30.0
十二月 19, 2015 1:01:39 上午 org.apache.catalina.startup.VersionLoggerListener log
信息: OS Name: Linux
十二月 19, 2015 1:01:39 上午 org.apache.catalina.startup.VersionLoggerListener log
信息: OS Version: 2.6.32-573.el6.x86_64
十二月 19, 2015 1:01:39 上午 org.apache.catalina.startup.VersionLoggerListener log
信息: Architecture: amd64
十二月 19, 2015 1:01:39 上午 org.apache.catalina.startup.VersionLoggerListener log
信息: Java Home: /usr/java/jdk1.7.0_67/jre
十二月 19, 2015 1:01:39 上午 org.apache.catalina.startup.VersionLoggerListener log
信息: JVM Version: 1.7.0_67-b01
十二月 19, 2015 1:01:39 上午 org.apache.catalina.startup.VersionLoggerListener log
信息: JVM Vendor: Oracle Corporation
十二月 19, 2015 1:01:39 上午 org.apache.catalina.startup.VersionLoggerListener log
信息: CATALINA_BASE: /usr/local/apache-tomcat-8.0.30
十二月 19, 2015 1:01:39 上午 org.apache.catalina.startup.VersionLoggerListener log
信息: CATALINA_HOME: /usr/local/apache-tomcat-8.0.30
十二月 19, 2015 1:01:39 上午 org.apache.catalina.startup.VersionLoggerListener log
信息: Command line argument: -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
十二月 19, 2015 1:01:39 上午 org.apache.catalina.startup.VersionLoggerListener log
信息: Command line argument: -Djava.endorsed.dirs=/usr/local/tomcat/endorsed
十二月 19, 2015 1:01:39 上午 org.apache.catalina.startup.VersionLoggerListener log
信息: Command line argument: -Dcatalina.base=/usr/local/tomcat
十二月 19, 2015 1:01:39 上午 org.apache.catalina.startup.VersionLoggerListener log
信息: Command line argument: -Dcatalina.home=/usr/local/tomcat
十二月 19, 2015 1:01:39 上午 org.apache.catalina.startup.VersionLoggerListener log
信息: Command line argument: -Djava.io.tmpdir=/usr/local/tomcat/temp
十二月 19, 2015 1:01:39 上午 org.apache.catalina.core.AprLifecycleListener lifecycleEvent
信息: The APR based Apache Tomcat Native library which allows optimal performance in production environments was not found on the java.library.path: /usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
十二月 19, 2015 1:01:42 上午 org.apache.coyote.AbstractProtocol init
信息: Initializing ProtocolHandler ["http-nio-8080"]
十二月 19, 2015 1:01:42 上午 org.apache.tomcat.util.net.NioSelectorPool getSharedSelector
信息: Using a shared selector for servlet write/read
十二月 19, 2015 1:01:42 上午 org.apache.coyote.AbstractProtocol init
信息: Initializing ProtocolHandler ["ajp-nio-8009"]
十二月 19, 2015 1:01:42 上午 org.apache.tomcat.util.net.NioSelectorPool getSharedSelector
信息: Using a shared selector for servlet write/read
十二月 19, 2015 1:01:42 上午 org.apache.catalina.startup.Catalina load
信息: Initialization processed in 8859 ms
# catalina.sh start
Using CATALINA_BASE: /usr/local/tomcat
Using CATALINA_HOME: /usr/local/tomcat
Using CATALINA_TMPDIR: /usr/local/tomcat/temp
Using JRE_HOME: /usr/java/latest
Using CLASSPATH: /usr/local/tomcat/bin/bootstrap.jar:/usr/local/tomcat/bin/tomcat-juli.jar
Tomcat started.
You have new mail in /var/spool/mail/root
# tail -f ../logs/catalina.out
19-Dec-2015 01:35:43.136 INFO org.apache.catalina.ha.session.DeltaManager.startInternal Starting clustering manager at /testapp
19-Dec-2015 01:35:43.187 INFO org.apache.catalina.ha.session.DeltaManager.getAllClusterSessions Manager , requesting session state from org.apache.catalina.tribes.membership.MemberImpl. This operation will timeout if no session state has been received within 60 seconds.
19-Dec-2015 01:35:43.491 INFO org.apache.catalina.ha.session.DeltaManager.waitForSendAllSessions Manager ; session state send at 12/19/15 1:35 AM received in 354 ms.
19-Dec-2015 01:35:43.550 INFO org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory /usr/local/apache-tomcat-8.0.30/webapps/testapp has finished in 702 ms
19-Dec-2015 01:35:43.560 INFO org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-8080"]
19-Dec-2015 01:35:43.586 INFO org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["ajp-nio-8009"]
19-Dec-2015 01:35:43.587 INFO org.apache.catalina.startup.Catalina.start Server startup in 9636 ms
19-Dec-2015 01:35:43.187 INFO org.apache.catalina.ha.session.DeltaManager.getAllClusterSessions Manager , requesting session state from org.apache.catalina.tribes.membership.MemberImpl. This operation will timeout if no session state has been received within 60 seconds.
19-Dec-2015 01:35:43.491 INFO org.apache.catalina.ha.session.DeltaManager.waitForSendAllSessions Manager ; session state send at 12/19/15 1:35 AM received in 354 ms.
19-Dec-2015 01:35:43.550 INFO org.apache.catalina.startup.HostConfig.deployDirectory Deployment of web application directory /usr/local/apache-tomcat-8.0.30/webapps/testapp has finished in 702 ms
19-Dec-2015 01:35:43.560 INFO org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-8080"]
19-Dec-2015 01:35:43.586 INFO org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["ajp-nio-8009"]
19-Dec-2015 01:35:43.587 INFO org.apache.catalina.startup.Catalina.start Server startup in 9636 ms
http://s2.运维网.com/wyfs02/M00/78/26/wKioL1Z2rabzSH0bAADWy6qp4Bo516.jpg
http://s5.运维网.com/wyfs02/M01/78/26/wKioL1Z2rabxxVvDAADcR-5Zxq0155.jpg
页:
[1]