Styjun

Using the helm chart:
https://github.com/helm/charts/tree/master/stable/jenkins

Previously I had everything working with this chart on my k8s cluster. I recently re-installed the chart using the same persistent storage volume. I only changed the resource requests for the Agents in the values.yaml file. Now when I try to run a deployment through the pipeline, the master pod fails to connect to the worker pods that it is creating. I have verified there is space available on the Nodes, and they seem to get added without a problem. Just the startup for the Worker pods looks like it is having some issues. Most notably with the missing docker group. Not sure how I am suppose to configure that to create the group, or why this is an issue in the first place?

What am I doing wrong to get this deployment pipeline working again?

overriding values.yaml for the chart:

rbac:
 install: true

Master:
 HostName: jenkins.grapenuts.flytedesk.com
 ServiceType: ClusterIP
 NodePort:
 Ingress:
 Annotations:
 kubernetes.io/ingress.class: traefik
 JavaOpts: "-Dorg.jenkinsci.plugins.durabletask.BourneShellScript.HEARTBEAT_CHECK_INTERVAL=1000"
 resources:
 requests:
 cpu: "500m"
 memory: "512Mi"
 limits:
 cpu: "2000m"
 memory: "2048Mi"
 InstallPlugins:
 - amazon-ecr:1.6
 - blueocean:1.7.1
 - kubernetes:1.12.1
 - kubernetes-cli:1.2.0
 - workflow-aggregator:2.5
 - workflow-job:2.23
 - credentials-binding:1.16
 - git:3.9.1
 - slack:2.3

Agent:
 resources:
 requests:
 cpu: "200m"
 memory: "256Mi"
 limits:
 cpu: "1000m"
 memory: "4096Mi"

Persistence:
 Enabled: true
 Size: 8Gi
 ExistingClaim: jenkins-master-pvc

Logs from the Docker Worker Node created by the Jenkins Master node

WARN[2019-03-27T01:20:30.099307297Z] could not change group /var/run/docker.sock to docker: group docker not found 
 WARN[2019-03-27T01:20:30.099429310Z] [!] DON'T BIND ON ANY IP ADDRESS WITHOUT setting --tlsverify IF YOU DON'T KNOW WHAT YOU'RE DOING [!] 
 INFO[2019-03-27T01:20:30.111486018Z] libcontainerd: started new containerd process pid=17
 INFO[2019-03-27T01:20:30.111603708Z] parsed scheme: "unix" module=grpc
 INFO[2019-03-27T01:20:30.111622892Z] scheme "unix" not registered, fallback to default scheme module=grpc
 INFO[2019-03-27T01:20:30.117760193Z] ccResolverWrapper: sending new addresses to cc: [unix:///var/run/docker/containerd/containerd.sock 0 <nil>] module=grpc
 INFO[2019-03-27T01:20:30.117825438Z] ClientConn switching balancer to "pick_first" module=grpc
 INFO[2019-03-27T01:20:30.117931427Z] pickfirstBalancer: HandleSubConnStateChange: 0xc420740170, CONNECTING module=grpc
 INFO[2019-03-27T01:20:30.700526951Z] starting containerd revision=e6b3f5632f50dbc4e9cb6288d911bf4f5e95b18e version=v1.2.4
 INFO[2019-03-27T01:20:30.701746137Z] loading plugin "io.containerd.content.v1.content"... type=io.containerd.content.v1
 INFO[2019-03-27T01:20:30.701897179Z] loading plugin "io.containerd.snapshotter.v1.btrfs"... type=io.containerd.snapshotter.v1
 WARN[2019-03-27T01:20:30.702171413Z] failed to load plugin io.containerd.snapshotter.v1.btrfs error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.btrfs must be a btrfs filesystem to be used with the btrfs snapshotter"
 INFO[2019-03-27T01:20:30.702215021Z] loading plugin "io.containerd.snapshotter.v1.aufs"... type=io.containerd.snapshotter.v1
 WARN[2019-03-27T01:20:30.712326057Z] failed to load plugin io.containerd.snapshotter.v1.aufs error="modprobe aufs failed: "ip: can't find device 'aufs'nmodprobe: can't change directory to '/lib/modules': No such file or directoryn": exit status 1"
 INFO[2019-03-27T01:20:30.712352225Z] loading plugin "io.containerd.snapshotter.v1.native"... type=io.containerd.snapshotter.v1
 INFO[2019-03-27T01:20:30.712464182Z] loading plugin "io.containerd.snapshotter.v1.overlayfs"... type=io.containerd.snapshotter.v1
 INFO[2019-03-27T01:20:30.712635973Z] loading plugin "io.containerd.snapshotter.v1.zfs"... type=io.containerd.snapshotter.v1
 WARN[2019-03-27T01:20:30.716216413Z] failed to load plugin io.containerd.snapshotter.v1.zfs error="path /var/lib/docker/containerd/daemon/io.containerd.snapshotter.v1.zfs must be a zfs filesystem to be used with the zfs snapshotter"
 INFO[2019-03-27T01:20:30.716242462Z] loading plugin "io.containerd.metadata.v1.bolt"... type=io.containerd.metadata.v1
INFO[2019-03-27T01:20:30.766699317Z] scheme "unix" not registered, fallback to default scheme module=grpc
 INFO[2019-03-27T01:20:30.768163376Z] ccResolverWrapper: sending new addresses to cc: [unix:///var/run/docker/containerd/containerd.sock 0 <nil>] module=grpc
 INFO[2019-03-27T01:20:30.768190514Z] ClientConn switching balancer to "pick_first" module=grpc
 INFO[2019-03-27T01:20:30.768365344Z] ccResolverWrapper: sending new addresses to cc: [unix:///var/run/docker/containerd/containerd.sock 0 <nil>] module=grpc
 INFO[2019-03-27T01:20:30.768639743Z] ClientConn switching balancer to "pick_first" module=grpc
 INFO[2019-03-27T01:20:30.768644457Z] pickfirstBalancer: HandleSubConnStateChange: 0xc420740a90, CONNECTING module=grpc
 INFO[2019-03-27T01:20:30.768760912Z] pickfirstBalancer: HandleSubConnStateChange: 0xc4207f3450, CONNECTING module=grpc
 INFO[2019-03-27T01:20:30.769170856Z] pickfirstBalancer: HandleSubConnStateChange: 0xc420740a90, READY module=grpc
 INFO[2019-03-27T01:20:30.769869637Z] pickfirstBalancer: HandleSubConnStateChange: 0xc4207f3450, READY module=grpc
 INFO[2019-03-27T01:20:30.822166956Z] Graph migration to content-addressability took 0.00 seconds 
 INFO[2019-03-27T01:20:30.825122094Z] Loading containers: start. 
 WARN[2019-03-27T01:20:30.854711776Z] Running modprobe bridge br_netfilter failed with message: ip: can't find device 'bridge'
 bridge 172032 1 br_netfilter
 stp 16384 1 bridge
 llc 16384 2 bridge,stp
 ipv6 544768 497 bridge,[permanent]
 ip: can't find device 'br_netfilter'
 br_netfilter 24576 0 
 bridge 172032 1 br_netfilter
 modprobe: can't change directory to '/lib/modules': No such file or directory
 , error: exit status 1 
 WARN[2019-03-27T01:20:30.887386825Z] Running modprobe nf_nat failed with message: `ip: can't find device 'nf_nat'
 nf_nat_masquerade_ipv4 16384 1 ipt_MASQUERADE
 nf_nat_ipv4 16384 1 iptable_nat
 nf_nat 36864 3 xt_nat,nf_nat_masquerade_ipv4,nf_nat_ipv4
 nf_conntrack 151552 8 xt_nat,ipt_MASQUERADE,nf_nat_masquerade_ipv4,nf_conntrack_netlink,nf_conntrack_ipv4,nf_nat_ipv4,xt_conntrack,nf_nat
 libcrc32c 16384 3 nf_nat,nf_conntrack,xfs
 modprobe: can't change directory to '/lib/modules': No such file or directory`, error: exit status 1 
 WARN[2019-03-27T01:20:30.895238719Z] Running modprobe xt_conntrack failed with message: `ip: can't find device 'xt_conntrack'
 xt_conntrack 16384 2 
 nf_conntrack 151552 8 xt_nat,ipt_MASQUERADE,nf_nat_masquerade_ipv4,nf_conntrack_netlink,nf_conntrack_ipv4,nf_nat_ipv4,xt_conntrack,nf_nat
 x_tables 45056 12 xt_statistic,xt_nat,xt_recent,ipt_REJECT,xt_tcpudp,xt_comment,xt_mark,ipt_MASQUERADE,xt_addrtype,iptable_filter,xt_conntrack,ip_tables
 modprobe: can't change directory to '/lib/modules': No such file or directory`, error: exit status 1 
 INFO[2019-03-27T01:20:31.093705399Z] Default bridge (docker0) is assigned with an IP address 172.17.0.0/16. Daemon option --bip can be used to set a preferred IP address 
 INFO[2019-03-27T01:20:31.332494859Z] Loading containers: done. 
 INFO[2019-03-27T01:20:31.470554395Z] Docker daemon commit=774a1f4 graphdriver(s)=overlay2 version=18.09.3
 INFO[2019-03-27T01:20:31.470683993Z] Daemon has completed initialization 
 INFO[2019-03-27T01:20:31.485026627Z] API listen on [::]:2375 
 INFO[2019-03-27T01:20:31.485180400Z] API listen on /var/run/docker.sock 

I think that the problem would be that user jenkins cannot run docker commands.

1. for the log " could not change group /var/run/docker.sock to docker: group docker not found ", according to https://access.redhat.com/solutions/3745881.
   It's informative messages and not an error.




2. Create the docker group. You may need to add jenkins user to docker group.
   $ sudo groupadd docker
   Add your user to the docker group.
   $ sudo usermod -aG docker $USER
   https://docs.docker.com/install/linux/linux-postinstall/#manage-docker-as-a-non-root-user

In https://github.com/helm/charts/tree/master/stable/jenkins, the default settings of this helm chart let Jenkins run as root user with uid 0.
In other cases, Jenkins is run with user jenkins, uid = 1000

You may provide more errors that you meet ?