从零搭建基于Istio的ServiceMesh-03流量管理

发表于 | 更新于: | 分类于 ， | 阅读次数:
字数统计: 5k

说明

  • istio版本号1.4.2
  • k8s集群版本v1.14.8
  • istio在1.4提供了基于istioctl命令直接部署的功能，这里使用istioctl部署istio。
    • 自带配置校验、和丰富的自定义配置选项
    • API版本号还在Alpha阶段，install.istio.io/v1alpha2，请自行判断是否适用
    • 部署要求
      • 至少得有Kubernetes集群
      • Istio-1.4版本在1.13、1.14、1.15的k8s集群上是做过测试通过的
      • 最新的1.16没在官方文档里注明，应该也是可以用的。官方说明在此
  • 这里通过官方示例熟悉一下istio的ServiceMesh特性

流量管理 Traffic Management

说明

  • 流量管理包含了Ingress和Egress，这里单独拎出来介绍，避免章节篇幅太长
  • 下面继续

请求路由 Request Routing

部署VirtualService

1
kubectl apply -f samples/bookinfo/networking/virtual-service-all-v1.yaml

文件内容如下

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: productpage
spec:
  hosts:
  - productpage
  http:
  - route:
    - destination:
        host: productpage
        subset: v1
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: reviews
spec:
  hosts:
  - reviews
  http:
  - route:
    - destination:
        host: reviews
        subset: v1
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: ratings
spec:
  hosts:
  - ratings
  http:
  - route:
    - destination:
        host: ratings
        subset: v1
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: details
spec:
  hosts:
  - details
  http:
  - route:
    - destination:
        host: details
        subset: v1
---

检查VirtualService

1
kubectl -n default get virtualservice.networking.istio.io

输出示例

1
2
3
4
5
6
NAME          GATEWAYS             HOSTS           AGE
bookinfo      [bookinfo-gateway]   [*]             28m
details                            [details]       43s
productpage                        [productpage]   43s
ratings                            [ratings]       43s
reviews                            [reviews]       43s

检查DestinationRule

1
kubectl -n default get destinationrules.networking.istio.io

输出示例

1
2
3
4
5
NAME          HOST          AGE
details       details       11m
productpage   productpage   11m
ratings       ratings       11m
reviews       reviews       11m

测试路由配置

  • 通过浏览器打开http://${GATEWAY_URL}/productpage

  • 会发现无论刷新多少次页面都只能访问到reviewers-v1的页面。

  • 即所有流量都被路由到了reviewers:v1

配置新的VirtualService

1
kubectl apply -f samples/bookinfo/networking/virtual-service-reviews-test-v2.yaml

文件内容如下

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: reviews
spec:
  hosts:
    - reviews
  http:
  - match:
    - headers:
        end-user:
          exact: jason
    route:
    - destination:
        host: reviews
        subset: v2
  - route:
    - destination:
        host: reviews
        subset: v1

测试新的配置

  • 不登陆，返回的页面依旧是reviews-v1
  • 点击右上角的sign in，登录之后，返回的页面是reviews-v2
    • UserName填入jason
    • Password不填或者随便填
  • 登录其他用户，返回的页面是reviews-v1

技术总结

通过配置VirtualService将请求的数据流修改如下

  • productpage → reviews:v2 → ratings (only for user jason)
  • productpage → reviews:v1 (for everyone else)

清理现场

1
kubectl delete -f samples/bookinfo/networking/virtual-service-all-v1.yaml

HTTP故障注入 HTTP Fault Injection

环境准备

  • 部署VirtualService
1
2
kubectl apply -f samples/bookinfo/networking/virtual-service-all-v1.yaml
kubectl apply -f samples/bookinfo/networking/virtual-service-reviews-test-v2.yaml
  • 上面两个YAML文件会将请求的数据流变成这样(这一步跟流量管理一样)
    • productpage → reviews:v2 → ratings (only for user jason)
    • productpage → reviews:v1 (for everyone else)

注入HTTP延迟错误

实验设计

  • 为jason用户访问reviews-v2和rateings注入7s延迟
  • reviews-v2服务，代码里设置了10s的连接超时时间
  • 理论上，注入7s延迟不会导致reviews-v2超时出错

创建故障注入规则

1
kubectl apply -f samples/bookinfo/networking/virtual-service-ratings-test-delay.yaml

文件内容如下

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: ratings
spec:
  hosts:
  - ratings
  http:
  - match:
    - headers:
        end-user:
          exact: jason
    fault:
      delay:
        percentage:
          value: 100.0
        fixedDelay: 7s
    route:
    - destination:
        host: ratings
        subset: v1
  - route:
    - destination:
        host: ratings
        subset: v1

测试故障注入

  • 浏览器打开http://${GATEWAY_URL}/productpage
  • 按F12打开开发者工具，切换到network标签
  • 登录jason用户，等待7s之后可以看到页面返回以下内容
1
2
Error fetching product reviews!
Sorry, product reviews are currently unavailable for this book.
  • 在开发者工具里面能看到请求productpage的时间是6s
  • 登录其他用户或者不登陆，刷新页面并不会引入延迟

技术总结

  1. reviews-v2服务，代码里面设置10s超时，延时7s并不会引发报错
  2. productpage和ratings服务之间的超时时间是3s，重试次数为1，因此只能容忍3s+3s总共6s的延迟，这里设置7s，导致超时出错
  3. 这样就可以通过延迟注入，来检查服务间调用的超时时间是否合理

清理现场

1
kubectl apply -f samples/bookinfo/networking/virtual-service-all-v1.yaml

注入HTTP状态码错误

实验设计

  • 为jason用户引入HTTP状态码错误

创建故障注入规则

1
kubectl apply -f samples/bookinfo/networking/virtual-service-ratings-test-abort.yaml

文件内容如下

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: ratings
spec:
  hosts:
  - ratings
  http:
  - match:
    - headers:
        end-user:
          exact: jason
    fault:
      abort:
        percentage:
          value: 100.0
        httpStatus: 500
    route:
    - destination:
        host: ratings
        subset: v1
  - route:
    - destination:
        host: ratings
        subset: v1

测试故障注入

  • 浏览器打开http://${GATEWAY_URL}/productpage
  • 按F12打开开发者工具，切换到network标签
  • 登录jason用户，可以看到
    • 页面很快就加载完毕
    • ratings服务不可用，提示Ratings service is currently unavailable
  • 登录其他用户或者不登陆，刷新页面并不会看到此报错信息

技术总结

  1. HTTP状态码注入只影响jason用户，由于没有引入延迟错误，所以是立刻返回HTTP 500
  2. 在浏览器开发者工具的network标签里，看不到请求有HTTP500状态码
  3. 查看reviews-v2服务的日志，可以看到日志，证明是在reviews-v2和ratings之间注入故障的
1
Error: unable to contact http://ratings:9080/ratings got status of 500

清理现场

1
kubectl delete -f samples/bookinfo/networking/virtual-service-all-v1.yaml

HTTP流量切换 HTTP Traffic Shifting

环境准备

  • 部署bookinfo应用
  • 配置默认的DestinationRule

配置所有流量走v1服务

1
kubectl apply -f samples/bookinfo/networking/virtual-service-all-v1.yaml

配置权重路由

1
kubectl apply -f samples/bookinfo/networking/virtual-service-reviews-50-v3.yaml

文件内容如下

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: reviews
spec:
  hosts:
    - reviews
  http:
  - route:
    - destination:
        host: reviews
        subset: v1
      weight: 50
    - destination:
        host: reviews
        subset: v3
      weight: 50

验证流量切换

  • 浏览器打开http://${GATEWAY_URL}/productpage
  • 不断按ctrl+F5刷新页面，可以看到页面不断在reviews-v1和reviews-v3之间切换

技术总结

  • 在所有流量都走v1的情况下，通过权重路由的方式，将部分流量切换到了新版本
  • 通过流量比例的调整，达到灰度更新的目的

清理现场

1
kubectl delete -f samples/bookinfo/networking/virtual-service-all-v1.yaml

TCP流量切换 TCP Traffic Shifting

环境准备

  • 部署多版本TCP-echo服务

    1
    kubectl apply -f samples/tcp-echo/tcp-echo-services.yaml

  • 检查部署情况

    1
    kubectl -n default get pod -l app=tcp-echo

    输出示例

    1
    2
    3
    4
    NAME                           READY   STATUS    RESTARTS   AGE
    tcp-echo-5ff7fd7996-ds72s      2/2     Running   0          40m
    tcp-echo-v1-b559f69bd-5mbj8    2/2     Running   0          3m48s
    tcp-echo-v2-547bb5f4d5-trlh9   2/2     Running   0          3m48s

配置默认TCP流量路由到v1版本

这里把TCP流量路由到v1版本

1
kubectl apply -f samples/tcp-echo/tcp-echo-all-v1.yaml

文件内容如下

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: tcp-echo-gateway
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 31400
      name: tcp
      protocol: TCP
    hosts:
    - "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: tcp-echo-destination
spec:
  host: tcp-echo
  subsets:
  - name: v1
    labels:
      version: v1
  - name: v2
    labels:
      version: v2
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: tcp-echo
spec:
  hosts:
  - "*"
  gateways:
  - tcp-echo-gateway
  tcp:
  - match:
    - port: 31400
    route:
    - destination:
        host: tcp-echo
        port:
          number: 9000
        subset: v1

测试TCP路由

  1. 获取IngressGateway访问方式，这里的istio-ingressGateway使用NodePort
1
2
export INGRESS_HOST=$(kubectl get pod -l istio=ingressgateway -n istio-system -o jsonpath='{.items[0].status.hostIP}')
export INGRESS_PORT=$(kubectl -n istio-system get service istio-ingressgateway -o jsonpath='{.spec.ports[?(@.name=="tcp")].NodePort}')
  1. 访问TCP-echo
1
for i in {1..10};do /bin/bash -c "(date ; sleep 1) | nc $INGRESS_HOST $INGRESS_PORT";done

输出示例

1
2
3
4
5
6
7
8
9
10
one Mon Dec 23 10:13:54 CST 2019
one Mon Dec 23 10:13:55 CST 2019
one Mon Dec 23 10:13:56 CST 2019
one Mon Dec 23 10:13:57 CST 2019
one Mon Dec 23 10:13:58 CST 2019
one Mon Dec 23 10:13:59 CST 2019
one Mon Dec 23 10:14:00 CST 2019
one Mon Dec 23 10:14:01 CST 2019
one Mon Dec 23 10:14:02 CST 2019
one Mon Dec 23 10:14:03 CST 2019
可以看到输出的信息是以one开头的，证明访问的是v1的服务

配置新的TCP流量路由

把20%的流量路由到tcp-echo:v2

1
kubectl apply -f samples/tcp-echo/tcp-echo-20-v2.yaml

文件内容如下

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: tcp-echo
spec:
  hosts:
  - "*"
  gateways:
  - tcp-echo-gateway
  tcp:
  - match:
    - port: 31400
    route:
    - destination:
        host: tcp-echo
        port:
          number: 9000
        subset: v1
      weight: 80
    - destination:
        host: tcp-echo
        port:
          number: 9000
        subset: v2
      weight: 20

测试新的TCP路由

1
for i in {1..10};do /bin/bash -c "(date ; sleep 1) | nc $INGRESS_HOST $INGRESS_PORT";done

输出示例

1
2
3
4
5
6
7
8
9
10
one Mon Dec 23 10:19:14 CST 2019
one Mon Dec 23 10:19:15 CST 2019
two Mon Dec 23 10:19:16 CST 2019
one Mon Dec 23 10:19:17 CST 2019
one Mon Dec 23 10:19:18 CST 2019
one Mon Dec 23 10:19:19 CST 2019
one Mon Dec 23 10:19:20 CST 2019
two Mon Dec 23 10:19:21 CST 2019
one Mon Dec 23 10:19:22 CST 2019
one Mon Dec 23 10:19:23 CST 2019

技术总结

  • istio-ingressGateway通过VirtualService定义的流量转发权重分配流量
  • Kubernetes的Service是根据Pod数量，轮询分配TCP流量，无法灵活调配流量比例
  • VirtualService流量分配比例独立于Kubernetes的Pod副本数量，因此可以独立控制两个版本的数量。

清理现场

1
2
kubectl delete -f samples/tcp-echo/tcp-echo-all-v1.yaml
kubectl delete -f samples/tcp-echo/tcp-echo-services.yaml

请求超时 Request Timeout

环境准备

  • 部署带有默认DestionationRules的bookinfo
  • 配置所有流量走v1服务
1
kubectl apply -f samples/bookinfo/networking/virtual-service-all-v1.yaml

配置请求延迟

通过配置HTTP路由里面的delay字段，可以人为地把请求延迟。

  1. 把请求路由到reviews-v2
1
2
3
4
5
6
7
8
9
10
11
12
13
14
kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: reviews
spec:
  hosts:
    - reviews
  http:
  - route:
    - destination:
        host: reviews
        subset: v2
EOF
  1. 给ratings添加2s延迟，让istio-proxy接到请求时延迟2s再处理
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: ratings
spec:
  hosts:
  - ratings
  http:
  - fault:
      delay:
        percent: 100
        fixedDelay: 2s
    route:
    - destination:
        host: ratings
        subset: v1
EOF

测试请求延迟

  1. 用浏览器打开http://${GATEWAY_URL}/productpage

  2. 等待2s之后页面才加载完毕，reviews显示正常

配置请求超时

  • 现在给reviews服务添加0.5秒的超时
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: reviews
spec:
  hosts:
  - reviews
  http:
  - route:
    - destination:
        host: reviews
        subset: v2
    timeout: 0.5s
EOF

测试请求超时

  1. 用浏览器打开http://${GATEWAY_URL}/productpage
  2. 等待0.5s之后页面加载完毕，reviews服务提示unavailable

技术总结

  1. 请求延迟是在ratings服务引入的，也就是reviews服务的upstream

  2. reviews服务配置了超时时间后，在访问ratings，超过0.5s之后，ratings还没返回数据，直接就报unavailable

  3. 请求流程如下

    productpage → reviews:v2(timeout=0.5s) → ratings(delay=2s)

  4. 这里的延迟和超时，是由istio-proxy控制，程序无法感知

  5. 除了在istio路由规则中配置注入超时时间，程序可以在HTTP Header里添加x-envoy-upstream-rq-timeout-ms来覆盖掉istio路由规则里配置的超时时间

清理现场

1
kubectl delete -f samples/bookinfo/networking/virtual-service-all-v1.yaml

熔断 Circuit Breaking

环境准备

  1. 部署httpbin
1
kubectl apply -f samples/httpbin/httpbin.yaml
  1. 检查httpbin
1
kubectl get pod -l app=httpbin

配置熔断

  1. 为httpbin创建带有熔断配置的DestinationRule，这里超过1个以上的并发请求会触发熔断
    • maxConnections限制对后端服务发起的 HTTP/1.1 连接数，如果超过了这个限制，就会开启熔断。
    • http1MaxPendingRequests: 1最大HTTP请求pending数
    • maxRequestsPerConnection: 1在任何给定时间内限制对后端服务发起的 HTTP/2 请求数，如果超过了这个限制，就会开启熔断。
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: httpbin
spec:
  host: httpbin
  trafficPolicy:
    connectionPool:
      tcp:
        maxConnections: 1
      http:
        http1MaxPendingRequests: 1
        maxRequestsPerConnection: 1
    outlierDetection:
      consecutiveErrors: 1
      interval: 1s
      baseEjectionTime: 3m
      maxEjectionPercent: 100
EOF

创建客户端程序

  1. 部署客户端
1
kubectl apply -f samples/httpbin/sample-client/fortio-deploy.yaml
  1. 查看客户端日志

这里登录到客户端Pod终端使用fortio工具调用httpbin

1
2
FORTIO_POD=$(kubectl get pod | grep fortio | awk '{ print $1 }')
kubectl exec -it $FORTIO_POD  -c fortio /usr/bin/fortio -- load -curl  http://httpbin:8000/get

输出示例

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
HTTP/1.1 200 OK
server: envoy
date: Sat, 28 Dec 2019 02:45:09 GMT
content-type: application/json
content-length: 371
access-control-allow-origin: *
access-control-allow-credentials: true
x-envoy-upstream-service-time: 3

{
  "args": {}, 
  "headers": {
    "Content-Length": "0", 
    "Host": "httpbin:8000", 
    "User-Agent": "fortio.org/fortio-1.3.1", 
    "X-B3-Parentspanid": "fcd70a478623c81e", 
    "X-B3-Sampled": "1", 
    "X-B3-Spanid": "6f6213dc72981add", 
    "X-B3-Traceid": "8aabe5c0b403be4ffcd70a478623c81e"
  }, 
  "origin": "127.0.0.1", 
  "url": "http://httpbin:8000/get"
}

触发熔断

上面配置的熔断触发条件，只要超过1个并发请求就会触发熔断，这里演示一下

  1. 在客户端并发2个连接并发送20个请求

    1
    kubectl exec -it $FORTIO_POD  -c fortio /usr/bin/fortio -- load -c 2 -qps 0 -n 20 -loglevel Warning http://httpbin:8000/get

    输出示例

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    08:09:41 I logger.go:97> Log level is now 3 Warning (was 2 Info)
    Fortio 1.3.1 running at 0 queries per second, 12->12 procs, for 20 calls: http://httpbin:8000/get
    Starting at max qps with 2 thread(s) [gomax 12] for exactly 20 calls (10 per thread + 0)
    08:09:41 W http_client.go:679> Parsed non ok code 503 (HTTP/1.1 503)
    Ended after 16.699105ms : 20 calls. qps=1197.7
    Aggregated Function Time : count 20 avg 0.0016200426 +/- 0.0003949 min 0.00039727 max 0.002612624 sum 0.032400851
    # range, mid point, percentile, count
    >= 0.00039727 <= 0.001 , 0.000698635 , 5.00, 1
    > 0.001 <= 0.002 , 0.0015 , 90.00, 17
    > 0.002 <= 0.00261262 , 0.00230631 , 100.00, 2
    # target 50% 0.00152941
    # target 75% 0.00182353
    # target 90% 0.002
    # target 99% 0.00255136
    # target 99.9% 0.0026065
    Sockets used: 3 (for perfect keepalive, would be 2)
    Code 200 : 19 (95.0 %)
    Code 503 : 1 (5.0 %)
    Response Header Sizes : count 20 avg 218.5 +/- 50.13 min 0 max 230 sum 4370
    Response Body/Total Sizes : count 20 avg 583 +/- 78.46 min 241 max 601 sum 11660
    All done 20 calls (plus 0 warmup) 1.620 ms avg, 1197.7 qps

  2. 在客户端并发3个连接并发送30个请求

    1
    kubectl exec -it $FORTIO_POD  -c fortio /usr/bin/fortio -- load -c 3 -qps 0 -n 30 -loglevel Warning http://httpbin:8000/get

    输出示例

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    10:25:52 I logger.go:97> Log level is now 3 Warning (was 2 Info)
    Fortio 1.3.1 running at 0 queries per second, 12->12 procs, for 30 calls: http://httpbin:8000/get
    Starting at max qps with 3 thread(s) [gomax 12] for exactly 30 calls (10 per thread + 0)
    10:25:52 W http_client.go:679> Parsed non ok code 503 (HTTP/1.1 503)
    10:25:52 W http_client.go:679> Parsed non ok code 503 (HTTP/1.1 503)
    10:25:52 W http_client.go:679> Parsed non ok code 503 (HTTP/1.1 503)
    10:25:52 W http_client.go:679> Parsed non ok code 503 (HTTP/1.1 503)
    10:25:52 W http_client.go:679> Parsed non ok code 503 (HTTP/1.1 503)
    10:25:52 W http_client.go:679> Parsed non ok code 503 (HTTP/1.1 503)
    10:25:52 W http_client.go:679> Parsed non ok code 503 (HTTP/1.1 503)
    10:25:52 W http_client.go:679> Parsed non ok code 503 (HTTP/1.1 503)
    10:25:52 W http_client.go:679> Parsed non ok code 503 (HTTP/1.1 503)
    Ended after 29.802582ms : 30 calls. qps=1006.6
    Aggregated Function Time : count 30 avg 0.0022023969 +/- 0.001246 min 0.000148903 max 0.00409224 sum 0.066071906
    # range, mid point, percentile, count
    >= 0.000148903 <= 0.001 , 0.000574451 , 26.67, 8
    > 0.001 <= 0.002 , 0.0015 , 30.00, 1
    > 0.002 <= 0.003 , 0.0025 , 73.33, 13
    > 0.003 <= 0.004 , 0.0035 , 96.67, 7
    > 0.004 <= 0.00409224 , 0.00404612 , 100.00, 1
    # target 50% 0.00246154
    # target 75% 0.00307143
    # target 90% 0.00371429
    # target 99% 0.00406457
    # target 99.9% 0.00408947
    Sockets used: 11 (for perfect keepalive, would be 3)
    Code 200 : 21 (70.0 %)
    Code 503 : 9 (30.0 %)
    Response Header Sizes : count 30 avg 161 +/- 105.4 min 0 max 230 sum 4830
    Response Body/Total Sizes : count 30 avg 493 +/- 165 min 241 max 601 sum 14790
    All done 30 calls (plus 0 warmup) 2.202 ms avg, 1006.6 qps

查看istio-proxy状态

1
kubectl exec $FORTIO_POD -c istio-proxy -- pilot-agent request GET stats | grep httpbin | grep upstream_rq_pending

输出示例

1
2
3
4
cluster.outbound|8000||httpbin.default.svc.cluster.local.upstream_rq_pending_active: 0
cluster.outbound|8000||httpbin.default.svc.cluster.local.upstream_rq_pending_failure_eject: 0
cluster.outbound|8000||httpbin.default.svc.cluster.local.upstream_rq_pending_overflow: 21
cluster.outbound|8000||httpbin.default.svc.cluster.local.upstream_rq_pending_total: 140

其中upstream_rq_pending_overflow为溢出连接池熔断和失败的总请求，它的值是21，说明有 21 次调用触发了熔断

技术总结

流量镜像 Traffic Mirror

环境准备

  1. 部署httpbin-v1

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    cat <<EOF | kubectl create -f -
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: httpbin-v1
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: httpbin
          version: v1
      template:
        metadata:
          labels:
            app: httpbin
            version: v1
        spec:
          containers:
          - image: docker.io/kennethreitz/httpbin
            imagePullPolicy: IfNotPresent
            name: httpbin
            command: ["gunicorn", "--access-logfile", "-", "-b", "0.0.0.0:80", "httpbin:app"]
            ports:
            - containerPort: 80
    EOF

  2. 部署httpbin-v2

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    cat <<EOF | kubectl create -f -
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: httpbin-v2
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: httpbin
          version: v2
      template:
        metadata:
          labels:
            app: httpbin
            version: v2
        spec:
          containers:
          - image: docker.io/kennethreitz/httpbin
            imagePullPolicy: IfNotPresent
            name: httpbin
            command: ["gunicorn", "--access-logfile", "-", "-b", "0.0.0.0:80", "httpbin:app"]
            ports:
            - containerPort: 80
    EOF

  3. 配置Service

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    kubectl create -f - <<EOF
    apiVersion: v1
    kind: Service
    metadata:
      name: httpbin
      labels:
        app: httpbin
    spec:
      ports:
      - name: http
        port: 8000
        targetPort: 80
      selector:
        app: httpbin
    EOF

  4. 创建curl服务

    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    cat <<EOF | kubectl create -f -
    apiVersion: apps/v1
    kind: Deployment
    metadata:
      name: sleep
    spec:
      replicas: 1
      selector:
        matchLabels:
          app: sleep
      template:
        metadata:
          labels:
            app: sleep
        spec:
          containers:
          - name: sleep
            image: tutum/curl
            command: ["/bin/sleep","infinity"]
            imagePullPolicy: IfNotPresent
    EOF

配置默认路由

将所有流量路由到httpbin-v1服务

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: httpbin
spec:
  hosts:
    - httpbin
  http:
  - route:
    - destination:
        host: httpbin
        subset: v1
      weight: 100
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: httpbin
spec:
  host: httpbin
  subsets:
  - name: v1
    labels:
      version: v1
  - name: v2
    labels:
      version: v2
EOF

测试默认路由

1
2
export SLEEP_POD=$(kubectl get pod -l app=sleep -o jsonpath={.items..metadata.name})
kubectl exec -it $SLEEP_POD -c sleep -- sh -c 'curl  http://httpbin:8000/headers' | python -m json.tool

输出示例

1
2
3
4
5
6
7
8
9
10
11
12
{
    "headers": {
        "Accept": "*/*",
        "Content-Length": "0",
        "Host": "httpbin:8000",
        "User-Agent": "curl/7.35.0",
        "X-B3-Parentspanid": "601d07480eb04924",
        "X-B3-Sampled": "1",
        "X-B3-Spanid": "bffbefc545619374",
        "X-B3-Traceid": "be4d5069467d569f601d07480eb04924"
    }
}

检查日志

检查v1和v2两个版本的Pod日志，只会在v1上看到访问日志

  • v1的日志
1
2
export V1_POD=$(kubectl get pod -l app=httpbin,version=v1 -o jsonpath={.items..metadata.name})
kubectl logs -f $V1_POD -c httpbin

输出示例

1
127.0.0.1 - - [30/Dec/2019:05:27:31 +0000] "GET /headers HTTP/1.1" 200 303 "-" "curl/7.35.0"
  • v2的日志
1
2
export V2_POD=$(kubectl get pod -l app=httpbin,version=v2 -o jsonpath={.items..metadata.name})
kubectl logs -f $V2_POD -c httpbin

输出示例

1
<none>

配置镜像流量到v2

这里把所有流量路由到v1，并且把流量镜像到v2，镜像流量的比例是100%。

当流量被镜像的时候，被镜像请求流量会在HTTP Header里面添加-shadow

mirror_persent可以控制镜像流量的比例，如果不定义则所有流量会被镜像

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
kubectl apply -f - <<EOF
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: httpbin
spec:
  hosts:
    - httpbin
  http:
  - route:
    - destination:
        host: httpbin
        subset: v1
      weight: 100
    mirror:
      host: httpbin
      subset: v2
    mirror_percent: 100
EOF

测试镜像流量

1
kubectl exec -it $SLEEP_POD -c sleep -- sh -c 'curl  http://httpbin:8000/headers' | python -m json.tool

验证镜像流量

查看v1和v2两个Pod的日志，可以发现会同时收到请求

  • v1的日志
1
127.0.0.1 - - [30/Dec/2019:05:33:18 +0000] "GET /headers HTTP/1.1" 200 303 "-" "curl/7.35.0"
  • v2的日志
1
127.0.0.1 - - [30/Dec/2019:05:33:18 +0000] "GET /headers HTTP/1.1" 200 303 "-" "curl/7.35.0"

网络抓包

为了验证流量是真的被镜像了，通过tcpdump命令抓包

  • 另起一个终端，运行tcpdump命令
1
2
3
4
export SLEEP_POD=$(kubectl get pod -l app=sleep -o jsonpath={.items..metadata.name})
export V1_POD_IP=$(kubectl get pod -l app=httpbin,version=v1 -o jsonpath={.items..status.podIP})
export V2_POD_IP=$(kubectl get pod -l app=httpbin,version=v2 -o jsonpath={.items..status.podIP})
kubectl exec -it $SLEEP_POD -c istio-proxy -- sudo tcpdump -A -s 0 "((tcp) and (host $V1_POD_IP or host $V2_POD_IP))"
  • 继续发起访问
1
kubectl exec -it $SLEEP_POD -c sleep -- sh -c 'curl  http://httpbin:8000/headers' | python -m json.tool
  • 在tcpdump的输出中可以看到，确实会往v2 POD发送了HTTP Header为Host: httpbin-shadow的请求
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 262144 bytes
05:50:37.899317 IP sleep-b48bfdb67-79sss.39106 > 10-244-0-97.httpbin.default.svc.cluster.local.80: Flags [P.], seq 1377715903:1377716635, ack 3041513480, win 288, options [nop,nop,TS val 616957010 ecr 616814401], length 732: HTTP: GET /headers HTTP/1.1
E....!@.@...
..c
..a...PR.F..I..... .......
$..R$..AGET /headers HTTP/1.1
host: httpbin:8000
user-agent: curl/7.35.0
accept: */*
x-forwarded-proto: http
x-request-id: 6a86ff11-0a97-9d96-9ac4-55e83dca7af3
x-envoy-decorator-operation: httpbin.default.svc.cluster.local:8000/*
x-istio-attributes: CiUKGGRlc3RpbmF0aW9uLnNlcnZpY2UubmFtZRIJEgdodHRwYmluCioKHWRlc3RpbmF0aW9uLnNlcnZpY2UubmFtZXNwYWNlEgkSB2RlZmF1bHQKOgoKc291cmNlLnVpZBIsEiprdWJlcm5ldGVzOi8vc2xlZXAtYjQ4YmZkYjY3LTc5c3NzLmRlZmF1bHQKPwoYZGVzdGluYXRpb24uc2VydmljZS5ob3N0EiMSIWh0dHBiaW4uZGVmYXVsdC5zdmMuY2x1c3Rlci5sb2NhbAo9ChdkZXN0aW5hdGlvbi5zZXJ2aWNlLnVpZBIiEiBpc3RpbzovL2RlZmF1bHQvc2VydmljZXMvaHR0cGJpbg==
x-b3-traceid: 8bd08e38b48384de25be6a75798bf068
x-b3-spanid: 25be6a75798bf068
x-b3-sampled: 1
content-length: 0


05:50:37.899386 IP sleep-b48bfdb67-79sss.48708 > 10-244-0-98.httpbin.default.svc.cluster.local.80: Flags [P.], seq 818093502:818094295, ack 1670553383, win 293, options [nop,nop,TS val 616957010 ecr 616814401], length 793: HTTP: GET /headers HTTP/1.1
E..M,.@.@..N
..c
..b.D.P0...c..'...%.......
$..R$..AGET /headers HTTP/1.1
host: httpbin-shadow:8000
user-agent: curl/7.35.0
accept: */*
x-forwarded-proto: http
x-request-id: 6a86ff11-0a97-9d96-9ac4-55e83dca7af3
x-envoy-decorator-operation: httpbin.default.svc.cluster.local:8000/*
x-istio-attributes: CiUKGGRlc3RpbmF0aW9uLnNlcnZpY2UubmFtZRIJEgdodHRwYmluCioKHWRlc3RpbmF0aW9uLnNlcnZpY2UubmFtZXNwYWNlEgkSB2RlZmF1bHQKOgoKc291cmNlLnVpZBIsEiprdWJlcm5ldGVzOi8vc2xlZXAtYjQ4YmZkYjY3LTc5c3NzLmRlZmF1bHQKPwoYZGVzdGluYXRpb24uc2VydmljZS5ob3N0EiMSIWh0dHBiaW4uZGVmYXVsdC5zdmMuY2x1c3Rlci5sb2NhbAo9ChdkZXN0aW5hdGlvbi5zZXJ2aWNlLnVpZBIiEiBpc3RpbzovL2RlZmF1bHQvc2VydmljZXMvaHR0cGJpbg==
x-b3-traceid: 8bd08e38b48384de25be6a75798bf068
x-b3-spanid: 25be6a75798bf068
x-b3-sampled: 1
x-envoy-internal: true
x-forwarded-for: 10.244.0.99
content-length: 0


05:50:37.900503 IP 10-244-0-98.httpbin.default.svc.cluster.local.80 > sleep-b48bfdb67-79sss.48708: Flags [P.], seq 1:237, ack 793, win 306, options [nop,nop,TS val 616957012 ecr 616957010], length 236: HTTP: HTTP/1.1 200 OK
E.. ..@.@.m.
..b
..c.P.Dc..'0. ....2.......
$..T$..RHTTP/1.1 200 OK
server: istio-envoy
date: Mon, 30 Dec 2019 05:50:37 GMT
content-type: application/json
content-length: 343
access-control-allow-origin: *
access-control-allow-credentials: true
x-envoy-upstream-service-time: 0


05:50:37.900513 IP sleep-b48bfdb67-79sss.48708 > 10-244-0-98.httpbin.default.svc.cluster.local.80: Flags [.], ack 237, win 302, options [nop,nop,TS val 616957012 ecr 616957012], length 0
E..4,.@.@..f
..c
..b.D.P0. .c..............
$..T$..T
05:50:37.900672 IP 10-244-0-98.httpbin.default.svc.cluster.local.80 > sleep-b48bfdb67-79sss.48708: Flags [P.], seq 237:580, ack 793, win 306, options [nop,nop,TS val 616957012 ecr 616957012], length 343: HTTP
E.....@.@.l.
..b
..c.P.Dc...0. ....2.*.....
$..T$..T{
  "headers": {
    "Accept": "*/*", 
    "Content-Length": "0", 
    "Host": "httpbin-shadow:8000", 
    "User-Agent": "curl/7.35.0", 
    "X-B3-Parentspanid": "25be6a75798bf068", 
    "X-B3-Sampled": "1", 
    "X-B3-Spanid": "a80ddc34652a28dd", 
    "X-B3-Traceid": "8bd08e38b48384de25be6a75798bf068", 
    "X-Envoy-Internal": "true"
  }
}

05:50:37.900679 IP sleep-b48bfdb67-79sss.48708 > 10-244-0-98.httpbin.default.svc.cluster.local.80: Flags [.], ack 580, win 311, options [nop,nop,TS val 616957012 ecr 616957012], length 0
E..4,.@.@..e
..c
..b.D.P0. .c..j...7.......
$..T$..T
05:50:37.900856 IP 10-244-0-97.httpbin.default.svc.cluster.local.80 > sleep-b48bfdb67-79sss.39106: Flags [P.], seq 1:540, ack 732, win 299, options [nop,nop,TS val 616957012 ecr 616957010], length 539: HTTP: HTTP/1.1 200 OK
E..O..@.@.=.
..a
..c.P...I..R.I....+.......
$..T$..RHTTP/1.1 200 OK
server: istio-envoy
date: Mon, 30 Dec 2019 05:50:37 GMT
content-type: application/json
content-length: 303
access-control-allow-origin: *
access-control-allow-credentials: true
x-envoy-upstream-service-time: 1

{
  "headers": {
    "Accept": "*/*", 
    "Content-Length": "0", 
    "Host": "httpbin:8000", 
    "User-Agent": "curl/7.35.0", 
    "X-B3-Parentspanid": "25be6a75798bf068", 
    "X-B3-Sampled": "1", 
    "X-B3-Spanid": "5afe9681c8064c5d", 
    "X-B3-Traceid": "8bd08e38b48384de25be6a75798bf068"
  }
}

05:50:37.900868 IP sleep-b48bfdb67-79sss.39106 > 10-244-0-97.httpbin.default.svc.cluster.local.80: Flags [.], ack 540, win 297, options [nop,nop,TS val 616957012 ecr 616957012], length 0
E..4."@.@...
..c
..a...PR.I..I.#...).......
$..T$..T

清理现场

1
2
3
4
kubectl delete virtualservice httpbin
kubectl delete destinationrule httpbin
kubectl delete deploy httpbin-v1 httpbin-v2 sleep
kubectl delete svc httpbin