aws alb ingress controller

AWS-alb-ingress-controller

项目:https://github.com/kubernetes-sigs/aws-load-balancer-controller

aws-alb-ingress-controller

  • 步骤1: controller 监听api-services的event事件,当发现 Ingress 资源满足要求,则将开始创建 AWS 资源。
  • 步骤2: 为 Ingress 资源创建 ALB
  • 步骤3: 为 Ingress 资源中指定的每个后端创建目标组
  • 步骤4: 为 Ingress 资源注释中指定的每个端口创建侦听器
  • 步骤5: 为 Ingress 资源中指定的每个路径创建规则

    注意

    • ingress-controller 需要有权限访问aws创建资源,具体权限可参考:iam-policy.json。本例子中直接对eks worknode role进行授权。理论上也可以进行OIDC接入aws iam针对于pod级别进行权限管理(未测试)。
    • 确保ingress annotation的资源存在,否则创建会失败,具体可看pod日志。
    • service 的type为 nodeport

Install controller

rbac+sa+ns.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
apiVersion: v1
kind: Namespace
metadata:
name: alb-ingress-controller
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
labels:
app: alb-ingress-controller
name: alb-ingress-controller
rules:
- apiGroups:
- ""
- extensions
resources:
- configmaps
- endpoints
- events
- ingresses
- ingresses/status
- services
verbs:
- create
- get
- list
- update
- watch
- patch
- apiGroups:
- ""
- extensions
resources:
- nodes
- pods
- secrets
- services
- namespaces
verbs:
- get
- list
- watch
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
labels:
app: alb-ingress-controller
name: alb-ingress-controller
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: alb-ingress-controller
subjects:
- kind: ServiceAccount
name: alb-ingress
namespace: alb-ingress-controller
---
apiVersion: v1
kind: ServiceAccount
metadata:
labels:
app: alb-ingress-controller
name: alb-ingress
namespace: alb-ingress-controller
controller.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: alb-ingress-controller
name: alb-ingress-controller
# Namespace the ALB Ingress Controller should run in. Does not impact which
# namespaces it's able to resolve ingress resource for. For limiting ingress
# namespace scope, see --watch-namespace.
namespace: alb-ingress-controller
spec:
replicas: 1
selector:
matchLabels:
app: alb-ingress-controller
strategy:
rollingUpdate:
maxSurge: 1
maxUnavailable: 1
type: RollingUpdate
template:
metadata:
creationTimestamp: null
labels:
app: alb-ingress-controller
spec:
containers:
- args:
# Limit the namespace where this ALB Ingress Controller deployment will
# resolve ingress resources. If left commented, all namespaces are used.
# - --watch-namespace=uat

# Setting the ingress-class flag below ensures that only ingress resources with the
# annotation kubernetes.io/ingress.class: "alb" are respected by the controller. You may
# choose any class you'd like for this controller to respect.
- --ingress-class=alb

# Name of your cluster. Used when naming resources created
# by the ALB Ingress Controller, providing distinction between
# clusters.
- --cluster-name=aux-eks

# AWS VPC ID this ingress controller will use to create AWS resources.
# If unspecified, it will be discovered from ec2metadata.
# - --aws-vpc-id=vpc-xxxxxx

# AWS region this ingress controller will operate in.
# If unspecified, it will be discovered from ec2metadata.
# List of regions: http://docs.aws.amazon.com/general/latest/gr/rande.html#vpc_region
# - --aws-region=us-west-1

# Enables logging on all outbound requests sent to the AWS API.
# If logging is desired, set to true.
# - ---aws-api-debug
# Maximum number of times to retry the aws calls.
# defaults to 10.
# - --aws-max-retries=10
env:
# AWS key id for authenticating with the AWS API.
# This is only here for examples. It's recommended you instead use
# a project like kube2iam for granting access.
#- name: AWS_ACCESS_KEY_ID
# value: KEYVALUE

# AWS key secret for authenticating with the AWS API.
# This is only here for examples. It's recommended you instead use
# a project like kube2iam for granting access.
#- name: AWS_SECRET_ACCESS_KEY
# value: SECRETVALUE
# Repository location of the ALB Ingress Controller.
image: 894847497797.dkr.ecr.us-west-2.amazonaws.com/aws-alb-ingress-controller:v1.0.0
imagePullPolicy: Always
name: server
resources: {}
terminationMessagePath: /dev/termination-log
dnsPolicy: ClusterFirst
restartPolicy: Always
securityContext: {}
terminationGracePeriodSeconds: 30
serviceAccountName: alb-ingress
serviceAccount: alb-ingress
iam-policy.json
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"acm:DescribeCertificate",
"acm:ListCertificates",
"acm:GetCertificate"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"ec2:AuthorizeSecurityGroupIngress",
"ec2:CreateSecurityGroup",
"ec2:CreateTags",
"ec2:DeleteTags",
"ec2:DeleteSecurityGroup",
"ec2:DescribeInstances",
"ec2:DescribeInstanceStatus",
"ec2:DescribeSecurityGroups",
"ec2:DescribeSubnets",
"ec2:DescribeTags",
"ec2:DescribeVpcs",
"ec2:ModifyInstanceAttribute",
"ec2:ModifyNetworkInterfaceAttribute",
"ec2:RevokeSecurityGroupIngress"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"elasticloadbalancing:AddTags",
"elasticloadbalancing:CreateListener",
"elasticloadbalancing:CreateLoadBalancer",
"elasticloadbalancing:CreateRule",
"elasticloadbalancing:CreateTargetGroup",
"elasticloadbalancing:DeleteListener",
"elasticloadbalancing:DeleteLoadBalancer",
"elasticloadbalancing:DeleteRule",
"elasticloadbalancing:DeleteTargetGroup",
"elasticloadbalancing:DeregisterTargets",
"elasticloadbalancing:DescribeListeners",
"elasticloadbalancing:DescribeLoadBalancers",
"elasticloadbalancing:DescribeLoadBalancerAttributes",
"elasticloadbalancing:DescribeRules",
"elasticloadbalancing:DescribeSSLPolicies",
"elasticloadbalancing:DescribeTags",
"elasticloadbalancing:DescribeTargetGroups",
"elasticloadbalancing:DescribeTargetGroupAttributes",
"elasticloadbalancing:DescribeTargetHealth",
"elasticloadbalancing:ModifyListener",
"elasticloadbalancing:ModifyLoadBalancerAttributes",
"elasticloadbalancing:ModifyRule",
"elasticloadbalancing:ModifyTargetGroup",
"elasticloadbalancing:ModifyTargetGroupAttributes",
"elasticloadbalancing:RegisterTargets",
"elasticloadbalancing:RemoveTags",
"elasticloadbalancing:SetIpAddressType",
"elasticloadbalancing:SetSecurityGroups",
"elasticloadbalancing:SetSubnets",
"elasticloadbalancing:SetWebACL"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"iam:GetServerCertificate",
"iam:ListServerCertificates"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"waf-regional:GetWebACLForResource",
"waf-regional:GetWebACL",
"waf-regional:AssociateWebACL",
"waf-regional:DisassociateWebACL"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"tag:GetResources",
"tag:TagResources"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"waf:GetWebACL"
],
"Resource": "*"
}
]
}
ingress.yaml
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
alb.ingress.kubernetes.io/actions.ssl-redirect: '{"Type": "redirect", "RedirectConfig":
{ "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}'
alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:ap-northeast-1:xxxxxxxxx:certificate/281bee29-717c-4b50-bf8e-41a39748c548
alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80, "HTTPS": 443}]'
alb.ingress.kubernetes.io/scheme: internet-facing
alb.ingress.kubernetes.io/subnets: subnet-32ce5544, subnet-41cf6c19
alb.ingress.kubernetes.io/security-groups: sg-018347dc7a3ade5a2
kubernetes.io/ingress.class: alb
name: xiemx-web-ingress
spec:
rules:
- host: www.xiemx.com
http:
paths:
- backend:
serviceName: echo-v1
servicePort: 80
- host: '*.xiemx.com'
http:
paths:
- backend:
serviceName: ssl-redirect
servicePort: use-annotation
path: /*
- backend:
serviceName: echo-v1
servicePort: 80
path: /*

测试:

  • 创建alb-ingress-controller
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    ➜  aws-alb-ingress-controller git:(master) ✗ k get all
    NAME READY STATUS RESTARTS AGE
    pod/alb-ingress-controller-9596b67b9-d7p69 1/1 Running 0 80d

    NAME READY UP-TO-DATE AVAILABLE AGE
    deployment.apps/alb-ingress-controller 1/1 1 1 361d

    NAME DESIRED CURRENT READY AGE
    replicaset.apps/alb-ingress-controller-9596b67b9 1 1 1 361d



    ➜ aws-alb-ingress-controller git:(master) ✗ k logs -f pod/alb-ingress-controller-9596b67b9-d7p69
    -------------------------------------------------------------------------------
    AWS ALB Ingress controller
    Release: v1.0.0
    Build: git-c25bc6c5
    Repository: https://github.com/kubernetes-sigs/aws-alb-ingress-controller
    -------------------------------------------------------------------------------

    W0917 09:50:21.934737 1 client_config.go:552] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
    I0917 09:50:21.990251 1 :0] kubebuilder/controller "level"=0 "msg"="Starting EventSource" "Controller"="alb-ingress-controller" "Source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{},"status":{"loadBalancer":{}}}}
    I0917 09:50:21.990434 1 :0] kubebuilder/controller "level"=0 "msg"="Starting EventSource" "Controller"="alb-ingress-controller" "Source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{},"status":{"loadBalancer":{}}}}
    I0917 09:50:21.990555 1 :0] kubebuilder/controller "level"=0 "msg"="Starting EventSource" "Controller"="alb-ingress-controller" "Source"={"Type":{"metadata":{"creationTimestamp":null}}}
    I0917 09:50:21.990841 1 :0] kubebuilder/controller "level"=0 "msg"="Starting EventSource" "Controller"="alb-ingress-controller" "Source"={"Type":{"metadata":{"creationTimestamp":null},"spec":{},"status":{"daemonEndpoints":{"kubeletEndpoint":{"Port":0}},"nodeInfo":{"machineID":"","systemUUID":"","bootID":"","kernelVersion":"","osImage":"","containerRuntimeVersion":"","kubeletVersion":"","kubeProxyVersion":"","operatingSystem":"","architecture":""}}}}
    I0917 09:50:21.993054 1 leaderelection.go:185] attempting to acquire leader lease alb-ingress-controller/ingress-controller-leader-alb...
    I0917 09:50:38.515944 1 leaderelection.go:194] successfully acquired lease alb-ingress-controller/ingress-controller-leader-alb
    I0917 09:50:38.716164 1 :0] kubebuilder/controller "level"=0 "msg"="Starting Controller" "Controller"="alb-ingress-controller"
    I0917 09:50:38.816322 1 :0] kubebuilder/controller "level"=0 "msg"="Starting workers" "Controller"="alb-ingress-controller" "WorkerCount"=1
  • 创建ingress测试
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    ➜  aws-alb-ingress-controller git:(master) ✗ k get ingress
    NAME HOSTS ADDRESS PORTS AGE
    xiemx-web-ingress www.xiemx.com,*.xiemx.com 71a14391-albingresscontrol-2ac6-648325502.ap-northeast-1.elb.amazonaws.com 80 14m
    ➜ aws-alb-ingress-controller git:(master) ✗ curl 71a14391-albingresscontrol-2ac6-648325502.ap-northeast-1.elb.amazonaws.com -H host:www.xiemx.com -I
    HTTP/1.1 200 OK
    Date: Mon, 07 Dec 2020 06:39:29 GMT
    Content-Type: text/plain
    Connection: keep-alive
    Server: echoserver

    ➜ aws-alb-ingress-controller git:(master) ✗ curl 71a14391-albingresscontrol-2ac6-648325502.ap-northeast-1.elb.amazonaws.com -H host:www.xiemx.com -i
    HTTP/1.1 200 OK
    Date: Mon, 07 Dec 2020 06:39:37 GMT
    Content-Type: text/plain
    Transfer-Encoding: chunked
    Connection: keep-alive
    Server: echoserver



    Hostname: echo-85fb7989cc-d556c

    Pod Information:
    node name: ip-10-200-2-113.ap-northeast-1.compute.internal
    pod name: echo-85fb7989cc-d556c
    pod namespace: alb-ingress-controller
    pod IP: 10.200.2.197

    Server values:
    server_version=nginx: 1.12.2 - lua: 10010

    Request Information:
    client_address=10.200.2.21
    method=GET
    real path=/
    query=
    request_version=1.1
    request_scheme=http
    request_uri=http://www.xiemx.com:8080/

    Request Headers:
    accept=*/*
    host=www.xiemx.com
    user-agent=curl/7.54.0
    x-amzn-trace-id=Root=1-5fcdce29-473e8ac375ce203f4961bec3
    x-forwarded-for=101.231.43.114
    x-forwarded-port=80
    x-forwarded-proto=http

    Request Body:
    -no body in request-

    ➜ aws-alb-ingress-controller git:(master) ✗ curl 71a14391-albingresscontrol-2ac6-648325502.ap-northeast-1.elb.amazonaws.com -H host:blog.xiemx.com -I
    HTTP/1.1 301 Moved Permanently
    Server: awselb/2.0
    Date: Mon, 07 Dec 2020 06:39:44 GMT
    Content-Type: text/html
    Content-Length: 134
    Connection: keep-alive
    Location: https://blog.xiemx.com:443/
  • aws alb 规则

    aws-alb-ingress-ex1
    aws-alb-ingress-ex2
    aws-alb-ingress-ex3
    aws-alb-ingress-ex4