Styjun

In my setup we send all the calls going out of cluster to an Internal Load Balancer in GCP. We do this by creating a egress service and manually adding endpoints to this service. The endpoint to this service is the IP of the Internal load balancer.

[sourabh.w@K9-MAC-035 r19-3]$ k get svc,ep -n egproxy-lle
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
service/egproxy ClusterIP 10.206.180.135 80/TCP,443/TCP 4d

NAME ENDPOINTS AGE
endpoints/egproxy 10.207.132.8:30443,10.207.132.8:30080 4d
[sourabh.w@K9-MAC-035 r19-3]$

For all micro-services in my setup, they have to run an "openssl s_Client" command at startup. This command is failing for me.

openssl s_client -servername ae17-api.kohlsecommerce.com -connect ae17-api.kohlsecommerce.com:443 -debug -state

While doing this I make sure ae17-api.kohlsecommerce.com is mapped to egproxy service's IP(10.206.180.135) in /etc/hosts.

Here is the o/p when I run openssl inside pod:

root@product-26-655f4f55b6-g2bpq:/# openssl s_client -servername ae17-api.kohlsecommerce.com -connect ae17-api.kohlsecommerce.com:443 -state -debug
CONNECTED(00000003)
SSL_connect:before SSL initialization
write to 0x556dc50b2860 [0x556dc50c3a20] (212 bytes => 212 (0xD4))
0000 - 16 03 01 00 cf 01 00 00-cb 03 03 43 59 24 26 31 ...........CY$&1
0010 - 4f 13 80 47 f2 09 25 f7-ec 74 40 57 7c d0 bc c6 O..G..%..t@W|...
0020 - 18 9b a7 a3 3c 38 80 d6-f4 99 62 00 00 38 c0 2c ....<8....b..8.,
0030 - c0 30 00 9f cc a9 cc a8-cc aa c0 2b c0 2f 00 9e .0.........+./..
0040 - c0 24 c0 28 00 00 c0 23-c0 88 00 67 c0 0a c0 14 .$.(.k.#.'.g....
0050 - 00 39 c0 09 c0 00 00 33-00 9d 00 9c 00 3d 00 3c .9.....3.....=.<
0060 - 00 35 00 2f 00 ff 01 00-66 6a 00 00 00 20 00 1e .5./.....j... ..
0070 - 00 00 1b 61 65 31 37 2d-61 70 69 2e 6b 6f 68 6c ...ae17-api.
0080 - 73 65 63 6f 6d 6d 65 72-63 65 2e 63 6f 6d 00 0b ecommerce.com..
0090 - 00 04 03 00 01 02 00 8a-00 0a 00 08 00 1d 00 17 ................
00a0 - 00 19 00 18 00 00 00 66-00 16 00 00 00 17 00 00 .....#..........
00b0 - 00 0d 00 20 00 00 06 01-06 02 06 03 05 01 05 02 ... ............
00c0 - 05 03 04 01 04 02 04 03-03 01 03 02 03 03 02 01 ................
00d0 - 02 02 02 03 ....
SSL_connect:SSLv3/TLS write client hello
read from 0x556dc50b2860 [0x556dc50ba803] (5 bytes => -1 (0xFFFFFFFFFFFFFFFF))
SSL_connect:error in SSLv3/TLS write client hello

write:errno=104

no peer certificate available

No client certificate CA names sent

SSL handshake has read 0 bytes and written 212 bytes

Verification: OK

New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : 0000
Session-ID:
Session-ID-ctx:
Master-Key:
PSK identity: None
PSK identity hint: None
SRP username: None
Start Time: 1553126020
Timeout : 7200 (sec)
Verify return code: 0 (ok)

Extended master secret: no

I tried creating various set of serviceentries and virtualservices but nothing worked:

1. With Load Balancer IP:

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
 generation: 1
 name: egproxy-ext
 namespace: r19-3-mui-qa
spec:
 addresses:
 - 10.207.132.8/32
 endpoints:
 - address: 10.207.132.8
 hosts:
 - istio-ilb.lle-mcommerce.com
 location: MESH_INTERNAL
 ports:
 - name: http
 number: 30080
 protocol: HTTP
 - name: https
 number: 30443
 protocol: HTTPS
 resolution: STATIC

2. with egproxy serivce FQDN:

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
 name: egproxy-headless-service-fqdn-ext
 namespace: r19-3-mui-qa
spec:
 addresses:
 - 10.206.117.116/32
 endpoints:
 - address: 10.207.132.8
 hosts:
 - egproxy.egproxy-lle.svc.cluster.local
 location: MESH_INTERNAL
 ports:
 - name: http
 number: 30080
 protocol: HTTP
 - name: https
 number: 30443
 protocol: HTTPS
 resolution: STATIC

3. Destination rule for Load balancer:

 apiVersion: networking.istio.io/v1alpha3
 kind: DestinationRule
 metadata:
 name: egress-gateway
 namespace: default
 spec:
 host: istio-ilb.lle-mcommerce.com
 trafficPolicy:
 loadBalancer:
 simple: ROUND_ROBIN
 portLevelSettings:
 - port:
 number: 30443
 tls:
 mode: SIMPLE

4. Destination rule for egproxy service

Want the communication to work from microservices to ILB via headless egproxy service.

Workarounds like "egress-gateway" is also a viable option but for that also facing problems in putting together correct config to make it work.

In Istio, to access a service, you need to configure either Kubernetes Service or Istio ServiceEntry. You may need to disable mutual TLS. See this preliminary example https://deploy-preview-3899--preliminary-istio.netlify.com/docs/examples/advanced-gateways/egress-kubernetes-services/.