Varnish---原理及应用

0987 · 发表于 2014-9-30 08:57:53

原理如下图:

用户请求到达Varnish服务器,经由网卡将请求接入进来到达tcp/ip协议栈解封装后由varnish将报文中请求资源的uri进行hash计算,而后根据计算的得到的键,到进程维持的hash表对比,若键相同,则根据相应的键去值指针所标识的,内存地址空间或是硬盘地址空间将结果取出,而后构建响应报文,将结果响应给客户端,若在hash表中没有响应的键,则进入过程2将请求的报文重新封装,发送给后端的backend server去处理,得到结果保存在缓存中,而后重新构建响应报文,响应给客户端.
Varnish的软件架构

Varnish的安装
官方网站下载相应发行版的软件包共享库和帮助文档repo.varnish-cache.org
然后执行yum安装
[iyunv@www ~]# yum install
varnish-3.0.5-1.el6.x86_64.rpm

varnish-docs-3.0.0-1.el6.x86_64.
varnish-libs-3.0.5-1.el6.x86_64.rpm
需要配置epel源

配置文件分为两部分
1,master的配置文件:/etc/sysyconfig/varnish
2,child 的配置文件:/etc/varnish/default.vcl------>>>由master调用vcl编译器转换为C语言形式而后c编译器编译后交由child使用这样避免语法错误child可直接使用

简单应用篇:
实验拓扑

配置varnish

[iyunv@www ~]# vim /etc/sysconfig/varnish
NFILES=131072
MEMLOCK=82000
NPROCS="unlimited"
RELOAD_VCL=1
VARNISH_VCL_CONF=/etc/varnish/default.vcl
VARNISH_LISTEN_PORT=80
VARNISH_ADMIN_LISTEN_ADDRESS=127.0.0.1
VARNISH_ADMIN_LISTEN_PORT=6082
VARNISH_SECRET_FILE=/etc/varnish/secret
VARNISH_MIN_THREADS=50
VARNISH_MAX_THREADS=1000
VARNISH_THREAD_TIMEOUT=120
VARNISH_STORAGE_SIZE=64M
VARNISH_STORAGE="malloc,${VARNISH_STORAGE_SIZE}"
VARNISH_TTL=120
DAEMON_OPTS="-a ${VARNISH_LISTEN_ADDRESS}:${VARNISH_LISTEN_PORT}
         -f ${VARNISH_VCL_CONF}
         -T ${VARNISH_ADMIN_LISTEN_ADDRESS}:${VARNISH_ADMIN_LISTEN_PORT}
         -t ${VARNISH_TTL}
         -p thread_pool_min=${VARNISH_MIN_THREADS}
         -p thread_pool_max=${VARNISH_MAX_THREADS}
         -p thread_pool_timeout=${VARNISH_THREAD_TIMEOUT}
         -u varnish -g varnish
         -S ${VARNISH_SECRET_FILE}
         -s ${VARNISH_STORAGE}"
[iyunv@www ~]# vim /etc/varnish/default.vcl
vcl 4.0;

backend default {
.host = "192.168.1.2";
.port = "80";
}
配置http编辑网页文件
Vim /var/www/html/index.html

启动各节点服务
Service vanish start
Service httpd start
客户端测试

进阶应用
需要使用VCL[varnish configure language]
请求数据在varnish中的流向

每个vcl_*函数都要有终止语句return(X)来说明下一个步要交给哪个函数处理

VCL基本语法[详情及变量man vcl]
1,//,#,/*comment*/ 表示注释
2,sub name,定义一个函数
3,不支持循环,有众多内置变量
4,使用终止语句,没有返回值
5,域专用
6,操作符  =赋值 ==等值比较 ~模式匹配 !取反 &&逻辑与 || 逻辑或

内置函数:
regsub (str,regex,sub) 搜索str中能被指定模式匹配的第一个字符串替换为sub
resuball (str,regex,sub) 搜索str中能被指定模式匹配的全部字符串替换为sub
Purge:从缓从中挑选出某对象及相关变种一并删除
Reyurn():当vcl域运行结束时控制权交由指定的下一个域处理
Return():重新运行整个vcl每次重启都会增加req.restarts变量中的值 max_restarts限定最大次数

Vcl_recv示例:

sub vcl_recv {
if (req.http.User-Agent ~ "iPad" ||
req.http.User-Agent ~ "iPhone" ||
req.http.User-Agent ~ "Android") {
set req.http.X-Device = "mobile";
} else {
set req.http.X-Device = "desktop";
}
}

此示例的含义是
如果请求的个护短代理类型是ipad,iphone,或是android那么就在求情的首部中添加一个新的首部X-device为mobile否则就设置为desktop
下面实验说明;在后端service的httpd访问日志中记录下X-Device的值
以下实验紧接上一个实验
首先配置varnish
sub vcl_recv {
# Happens before we check if we have this in cache already.
#
# Typically you clean up the request here, removing cookies you don't need,
# rewriting the request, etc.
if (req.http.User-Agent ~ "ipad"||
      req.http.User-Agent ~  "iphone"||
      req.http.User-Agent ~ "iphone"){
      set req.http.X-Device = "mobile";
      }else {
      set req.http.X-Device = "desktop";
      }
}

配置http日志格式加入X-Device首部的显示

重启httpd varnish使用客户端访问站点
验证结果
查看访问日志

上图所示的客户端地址为前端的varnish地址若向显示真实的地址方法很简单只需要在上述配置中增加一条vcl语句
sub vcl_recv {

if (req.http.User-Agent ~ "ipad"||
      req.http.User-Agent ~  "iphone"||
      req.http.User-Agent ~ "iphone"){
      set req.http.X-Device = "mobile";
      }else {
      set req.http.X-Device = "desktop";
      }
      set req.http.X-Forwarded-For = client.ip;
}

编辑http访问日志格式

查看日志,客户端的真实地址就显示出来了

判断缓存是否命中
sub vcl_deliver {
accounting or modifying the final object here.
if (obj.hits > 0 ) {
      set resp.http.X-Cache = "hit cache" + " " + server.hostname;
      }else {
      set resp.http.X-Cache = "miss cache" + " " + server.hostname;
      }
}

测试结果

如何绕过缓存
sub vcl_recv {
if (req.http.User-Agent ~ "ipad"||
      req.http.User-Agent ~  "iphone"||
      req.http.User-Agent ~ "iphone"){
      set req.http.X-Device = "mobile";
      }else {
      set req.http.X-Device = "desktop";
      }
      set req.http.X-Forwarded-For = client.ip;
if (req.url ~ "^/") {
   return(pass);
   }
}

测试结果,无论刷新几次都不会命中缓存

如何删除指定的缓存
acl purgers {
      "127.0.0.1";
      "172.16.0.0"/16;
}

sub vcl_recv {
      if (req.request == "PURGE") {
            if (!client.ip ~ purgers) {
                     error 405 "Method not allowed";
            }
            return (lookup);
      }
}
sub vcl_hit {
      if (req.request == "PURGE") {
            purge;
            error 200 "Purged";
      }
}
sub vcl_miss {
      if (req.request == "PURGE") {
            purge;
            error 404 "Not in cache";
      }
}
sub vcl_pass {
      if (req.request == "PURGE") {
            error 502 "PURGE on a passed object";
      }
}

客户端测试

这里为了方便测试使用linux工具curl
[iyunv@localhost html]# curl http://192.168.1.2

查看头部信息
[iyunv@localhost html]# curl -I http://192.168.1.2
HTTP/1.1 200 OK
Server: nginx/1.0.15
Content-Type: text/html
Last-Modified: Mon, 29 Sep 2014 06:15:54 GMT
Content-Length: 37
Accept-Ranges: bytes
Date: Mon, 29 Sep 2014 06:23:24 GMT
X-Varnish: 1450346254 1450346253
Age: 48
Via: 1.1 varnish
Connection: keep-alive
X-Cache: HIT via localhost.localdomain
此时缓存时命中的下面指定请求的方法看是否能够将缓存清除
[iyunv@localhost html]# curl -X PURGE http://172.16.34.1

"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">


200 Purged


Error 200 Purged
Purged

Guru Meditation:
XID: 1450346261

Varnish cache server



看到返回的信息缓存已被清除
下面再次请求主页面验证是否清除如果缓存没有hit 证明清除成功
[iyunv@localhost html]# curl -I http://172.16.34.1
HTTP/1.1 200 OK
Server: nginx/1.0.15
Content-Type: text/html
Last-Modified: Mon, 29 Sep 2014 06:15:54 GMT
Content-Length: 37
Accept-Ranges: bytes
Date: Mon, 29 Sep 2014 06:27:00 GMT
X-Varnish: 1450346262
Age: 0
Via: 1.1 varnish
Connection: keep-alive
X-Cache: MISS via localhost.localdomain

由此可见缓存MISS证明清除成功

Varnish实现负载均衡
编辑配置文件
backend web1 {
  .host = "192.168.1.2";
  .port = "8080";
  .probe = {
      .url="/.health.html";
      .interval = 2s;
      .window  = 8;
      .threshold = 3;
      .initial = 2 ;
      }

}
backend web2 {
  .host = "192.168.1.3";
  .port = "8080";
  .probe = {
      .url="/.health.html"; ------------->检测的页面
      .interval = 2s;    ------------->健康监测的时间间隔
      .window  = 8;    ------------->检测的次数
      .threshold = 3;    -------------->3次失败才认为后端主机不可用
      .initial = 2 ;       ------------->2次成功就说明后端主机正常
      }
}

director http_server round-robin { ----->算法为rr
   {.backend=web1;}
   {.backend=web2;}
}
acl purgers {
"127.0.0.1";
"172.16.0.0"/16;
}

sub vcl_recv {
      if (req.url ~ "^/"){
         set req.backend = http_server;
         return (pass);
      }

if (req.request == "PURGE") {
if (!client.ip ~ purgers) {
error 405 "Method not allowed";
}
return (lookup);
}
}
sub vcl_hit {
if (req.request == "PURGE") {
purge;
error 200 "Purged";
}
}
sub vcl_miss {
if (req.request == "PURGE") {
purge;
error 404 "Not in cache";
}
}
sub vcl_pass {
if (req.request == "PURGE") {
error 502 "PURGE on a passed object";
}
}

sub vcl_deliver {
      if (obj.hits > 0) {
            set resp.http.X-Cache = "HIT via" + " " + server.hostname;
      } else {
            set resp.http.X-Cache = "MISS via" + " " + server.hostname;
      }
}

配置后端服务器页面建立健康监测页面两个节点分别建立
Server1  :Vim /var/www/html/{index.html,.health.html}
Index.html内容 -----> Bacekend-server1 192.16.1.2
health.html内容 -----> OK
Server2  :Vim /var/www/html/{index.html,.health.html}
Index.html内容 -----> Bacekend-server2 192.16.1.3
health.html内容 -----> OK

客户端测试
[iyunv@localhost ~]# curl http://172.16.101.200
Backend-server1 192.168.1.2
[iyunv@localhost ~]# curl http://172.16.101.200
Back-server2 192.168.1.3
[iyunv@localhost ~]# curl http://172.16.101.200
Backend-server1 192.168.1.2
[iyunv@localhost ~]# curl http://172.16.101.200
Back-server2 192.168.1.3
查看后端状态
[iyunv@localhost ~]# varnishadm
varnish> backend.list
200
Backend name                Refs Admin    Probe
web1(192.168.1.2,,8080)       1    probe    Healthy 8/8
web2(192.168.1.3,,8080)       1    probe    Healthy 8/8
停止其中一个backend-server
varnish> backend.list
200
Backend name                Refs Admin    Probe
web1(192.168.1.2,,8080)       1    probe    Sick 0/8
web2(192.168.1.3,,8080)       1    probe    Healthy 8/8
这是只能访问web2

账号		自动登录	找回密码
密码			立即注册

wirelessnetview好用的无线分析工具

Red Hat RHCE 8 (EX294) Cert Guide

Shell从入门到精通（阿良）

亿图图示专家(EDraw Max) V7.9 中文破解版

zabbix3.4.1安装部署+微信推送信息+大屏显

Red Hat OpenShift I: Containers & Kubern

2025 年，C++ 还能“硬核”多久？

[经验分享] Varnish---原理及应用

浏览过的版块

扫码加入运维网微信交流群