Styjun

I'm trying to create a chroot to run a program that needs internet access to build myself a sandboxed, immutable developer environment. So far my jail is working well: I can run bash inside it and run simple programs from there! DNS resolution doesn't work however:

bash-3.2$ curl google.ca
curl: (6) Could not resolve host: google.ca

I am almost positive this is because the inner process is unable to connect to the mDNSResponder daemon running outside the jail. Outside the jail there is an mDNSResponder socket for the whole system to use:

host ➜ ls -lA /var/run/mDNSResponder
srw-rw-rw- 1 root daemon 0 22 Oct 10:41 /var/run/mDNSResponder

But, inside the jail there is not. So, I tried to use socat to create a unix socket "proxy" from inside the jail to outside it: I run socat (command below) to create a socket inside my jail, and then inside the jail run curl again, but curl still gives the same error message. I see this in syslog after turning on the verbose logging of mDNSResponder with SIGUSR1:

2015-10-26 5:32:30.835 PM mDNSResponder[95]: 12: connect_callback: Adding FD for uid 0
2015-10-26 5:32:30.835 PM mDNSResponder[95]: 12: DNSServiceCreateConnection START PID[23271](socat)
2015-10-26 5:32:30.836 PM mDNSResponder[95]: 12: read_msg: ERROR failed to get errsd via SCM_RIGHTS
2015-10-26 5:32:30.836 PM mDNSResponder[95]: 12: DNSServiceCreateConnection STOP PID[23271](socat)
2015-10-26 5:32:30.836 PM mDNSResponder[95]: 12: Removing FD
2015-10-26 5:32:31.339 PM curl[23269]: dnssd_clientstub read_all(5) failed 0/4 0 
2015-10-26 5:32:31.339 PM curl[23269]: dnssd_clientstub write_all(4) failed -1/28 32 Broken pipe
2015-10-26 5:32:31.341 PM mDNSResponder[95]: 12: connect_callback: Adding FD for uid 0
2015-10-26 5:32:31.341 PM mDNSResponder[95]: 12: DNSServiceCreateConnection START PID[23272](socat)
2015-10-26 5:32:31.342 PM mDNSResponder[95]: 12: read_msg: ERROR failed to get errsd via SCM_RIGHTS
2015-10-26 5:32:31.342 PM mDNSResponder[95]: 12: DNSServiceCreateConnection STOP PID[23272](socat)
2015-10-26 5:32:31.342 PM mDNSResponder[95]: 12: Removing FD
2015-10-26 5:32:31.844 PM curl[23269]: dnssd_clientstub read_all(5) failed 0/4 0 
2015-10-26 5:32:31.846 PM mDNSResponder[95]: 12: connect_callback: Adding FD for uid 0
2015-10-26 5:32:31.846 PM mDNSResponder[95]: 12: DNSServiceCreateConnection START PID[23274](socat)
2015-10-26 5:32:31.847 PM mDNSResponder[95]: 12: read_msg: ERROR failed to get errsd via SCM_RIGHTS
2015-10-26 5:32:31.847 PM mDNSResponder[95]: 12: DNSServiceCreateConnection STOP PID[23274](socat)
2015-10-26 5:32:31.847 PM mDNSResponder[95]: 12: Removing FD
2015-10-26 5:32:32.349 PM curl[23269]: dnssd_clientstub read_all(5) failed 0/4 0 
2015-10-26 5:32:32.350 PM mDNSResponder[95]: 12: connect_callback: Adding FD for uid 0
2015-10-26 5:32:32.351 PM mDNSResponder[95]: 12: DNSServiceCreateConnection START PID[23275](socat)
2015-10-26 5:32:33.361 PM mDNSResponder[95]: 12: DNSServiceCreateConnection STOP PID[23275](socat)
2015-10-26 5:32:33.361 PM mDNSResponder[95]: 12: Removing FD

which looks to me like curl via dnssd_clientstub is trying three times to resolve the name. Here's the socat log while the jailed process was trying to connect:

host ~/C/jail (master*) ➜
sudo socat -v -d -d UNIX-LISTEN:/Users/hornairs/Code/jail/jail-test/private/var/run/mDNSResponder,mode=666,fork,user=root,group=daemon UNIX-CLIENT:/private/var/run/mDNSResponder
Password:
2015/10/26 18:16:03 socat[24334] N listening on LEN=67 AF=1 "/Users/hornairs/Code/jail/jail-test/private/var/run/mDNSResponder"
2015/10/26 18:16:07 socat[24334] N accepting connection from LEN=16 AF=1 "" on LEN=67 AF=1 "/Users/hornairs/Code/jail/jail-test/private/var/run/mDNSResponder"
2015/10/26 18:16:07 socat[24334] N forked off child process 24341
2015/10/26 18:16:07 socat[24334] N listening on LEN=67 AF=1 "/Users/hornairs/Code/jail/jail-test/private/var/run/mDNSResponder"
2015/10/26 18:16:07 socat[24341] N opening connection to LEN=32 AF=1 "/private/var/run/mDNSResponder"
2015/10/26 18:16:07 socat[24341] N successfully connected from local address LEN=16 AF=1 ""
2015/10/26 18:16:07 socat[24341] N starting data transfer loop with FDs [6,6] and [5,5]
> 2015/10/26 18:16:07.081847 length=28 from=0 to=27
............................< 2015/10/26 18:16:07.082019 length=4 from=0 to=3
....> 2015/10/26 18:16:07.082167 length=50 from=28 to=77
...............b...............P.....google.ca....> 2015/10/26 18:16:07.082287 length=1 from=78 to=78
.2015/10/26 18:16:07 socat[24341] N socket 2 (fd 5) is at EOF
2015/10/26 18:16:07 socat[24341] N exiting with status 0
2015/10/26 18:16:07 socat[24334] N childdied(): handling signal 20

For comparison, here's what a successful lookup looks like when I run it from the host:

2015-10-26 5:31:56.524 PM mDNSResponder[95]: 12: connect_callback: Adding FD for uid 501
2015-10-26 5:31:56.524 PM mDNSResponder[95]: 12: DNSServiceCreateConnection START PID[23190](curl)
2015-10-26 5:31:56.524 PM mDNSResponder[95]: 12: Result code socket 27 created 00000000 00000001
2015-10-26 5:31:56.524 PM mDNSResponder[95]: 12: DNSServiceQueryRecord(15000, 0, google.ca., Addr) START PID[23190]()
2015-10-26 5:31:56.525 PM mDNSResponder[95]: 12: Result code socket 27 closed 00000000 00000001 (0)
2015-10-26 5:31:56.525 PM mDNSResponder[95]: 12: DNSServiceQueryRecord(google.ca., Addr) ADD 4 google.ca. Addr 74.216.233.251
2015-10-26 5:31:56.525 PM mDNSResponder[95]: 12: DNSServiceQueryRecord(google.ca., Addr) ADD 4 google.ca. Addr 74.216.233.249
2015-10-26 5:31:56.525 PM mDNSResponder[95]: 12: DNSServiceQueryRecord(google.ca., Addr) ADD 4 google.ca. Addr 74.216.233.227
*snip*
2015-10-26 5:31:56.525 PM mDNSResponder[95]: 12: Result code socket 27 created 00000000 00000002
2015-10-26 5:31:56.525 PM mDNSResponder[95]: 12: DNSServiceQueryRecord(15000, 0, google.ca., AAAA) START PID[23190]()
2015-10-26 5:31:56.526 PM mDNSResponder[95]: 12: Result code socket 27 closed 00000000 00000002 (0)
2015-10-26 5:31:56.526 PM mDNSResponder[95]: 12: DNSServiceQueryRecord(google.ca., AAAA) ADD 16 google.ca. AAAA 2607:F8B0:400B:080A:0000:0000:0000:100F
2015-10-26 5:31:56.526 PM mDNSResponder[95]: 12: Cancel 00000000 00000001
2015-10-26 5:31:56.526 PM mDNSResponder[95]: 12: DNSServiceQueryRecord(google.ca., Addr) STOP PID[23190]()
2015-10-26 5:31:56.526 PM mDNSResponder[95]: 12: Cancel 00000000 00000002
2015-10-26 5:31:56.526 PM mDNSResponder[95]: 12: DNSServiceQueryRecord(google.ca., AAAA) STOP PID[23190]()
2015-10-26 5:31:56.587 PM mDNSResponder[95]: 12: DNSServiceCreateConnection STOP PID[23190](curl)
2015-10-26 5:31:56.587 PM mDNSResponder[95]: 12: Removing FD

The major differences I notice between the failure and the success are that the uid is 0 for the request inside the jail and 501 for the request outside it. Curious, but that doesn't seem to be where the request actually fails.

The error message that comes from mDNSResponder seems to be concerning getting the errsd from the incoming request across the socket. https://github.com/jevinskie/mDNSResponder/blob/2942dde61f920fbbf96ff9a3840567ebbe7cb1b6/mDNSShared/uds_daemon.c#L3660

At this point it seems to me like mDNSResponder expects its clients to pass it a pair of fd's across the socket to respond to the client with, which I am not sure is even possible to do from inside the chroot. I am a very poor C programmer so I could be wrong about that, but if that is the truth, is it even possible to do this, and is there a better path to getting DNS to work inside the chroot?

Other tidbits:

• You can configure mDNSResponder to listen on more than one socket in it's launchd plist file, but that file is now protected by System Integrity Protection, which I don't want to disable to get this to work. It's janky, and easy to fall out of sync with the jail filesystems I am often changing on the hosts, which breaks DNS for every process if it can't create one of the sockets because the file doesn't exist. Running a proxy seems much more resilient




• ping dies instantly in the jail, which is why I am using curl. It gets a Killed: 9 message on the console instantly.



• 
  

  I am getting some other files into the chroot using a bind mount, but I can't get that to work for the mDNSResponder socket. I use http://bindfs.org (since OS X doesn't support Linux' mount --bind) to mount in /var/run to the chroot, but this comes up in the logs when trying to connect:

  
  
  

  2015-10-26 6:39:40.833 PM curl[25002]: dnssd_clientstub ConnectToServer: connect()-> No of tries: 1
  2015-10-26 6:39:41.837 PM curl[25002]: dnssd_clientstub ConnectToServer: connect()-> No of tries: 2
  2015-10-26 6:39:42.843 PM curl[25002]: dnssd_clientstub ConnectToServer: connect()-> No of tries: 3
  2015-10-26 6:39:43.848 PM curl[25002]: dnssd_clientstub ConnectToServer: connect() failed path:/var/run/mDNSResponder Socket:4 Err:-1 Errno:61 Connection refused
  

  
  

The problem here is that requests and responses on the /var/run/mDNSResponder socket use the "ancillary data" feature of the recvmsg and sendmsg system calls, and socat doesn't proxy ancillary data. I was able to get a custom proxy that does relay the ancillary data to work and provide DNS to processes inside a chroot.

You should make sure you have a copy of /etc/resolv.conf in your chroot. Without it, it wouldn't know which DNS servers to contact, so the only fallback option is to use mDNS. But even if mDNS would work, it would probably only be able to resolve local domain names with it, and not google.ca.