Styjun

I'm working on a C# server/client project at the moment and have come across a strange problem that took me a while to solve but am not entirely happy with the solution or more precisely why I need it. Essentially I found sending tcp socket messages over 10kbytes to a localhost target required a time delay of 1ms.

As a background, I have a server on a machine that many clients connect into and pass information back and forth which all seems to work fine. I also have clients that are local to the server that also connect in. The issue I faced was that when message sizes went over around ~10kbytes in size the messages would not go through. I only just noticed this because most messages sent were around 1-2kbytes in size, but the client connected on the same machine as the server (localhost) is more of a management client and thus sends/receives more data.

The real problem was the the server (sender) was returning true / success from the C# send commands that data was being sent, however, the receiver indicated only the first chunk (if I buffered) would come through or nothing at all. So I ended up putting wireshark on and saw that even though the send calls completed successfully, no data would actually be sent over the wire (remembering this is on the local loopback interface).

I tried playing with how I am sending data and trying all the different calls (NetworkStream.Write / NetworkStream.WriteAsync / Socket.Send / Socket.SendAsync / Socket.BeginSend) along with buffering the data or sending it all in one hit. Nothing seemed to make a difference until I put a time delay between the send calls in my loop then everything work perfectly (I tested up to a 1.5GB stream of data with no issue).

I also found a delay of anything under 1ms / 10,000 ticks would again cause issues, I would get through so many of the send calls to work then they would just stop again. Setting the TcpClient.NoDelay to true also did not seem to have much impact.

Below is a cut from my sending code as an example with the different send commands I have tried that all have the exact same behavior.

// _client is an abstracted/based off a TcpClient object (the target)
byte[] dataBytes = Serializer.SerializeMessage(message); 
int bytesSent = 0;

while (bytesSent < dataBytes.Length) 
 int bytesToSend = ((dataBytes.Length - bytesSent) < 8192) ? (dataBytes.Length - bytesSent) : 8192;
 //_client.Socket.Send(dataBytes, bytesSent, bytesToSend, SocketFlags.Partial);
 //_client.NetworkStream.Write(dataBytes, bytesSent, bytesToSend);
 //_client.Socket.BeginSend(dataBytes, bytesSent,bytesToSend,SocketFlags.None, ar => int bytes = ((Socket)ar.AsyncState).EndSend(ar);, _client.Socket);
 _ = _client.NetworkStream.WriteAsync(dataBytes, bytesSent, bytesToSend);

 bytesSent += bytesToSend;
 Thread.Sleep(TimeSpan.FromTicks(10000));

So while this works fine now, my question is why is it even needed, from previous experience I always find if you need to "delay" there is something else that is wrong. Also, without the delay, nothing errors out, all the calls report they successfully sent the bytes but wireshark shows no data being transmitted. My only thought would be something in the library detecting it is a local target and using some internal named piped abstracted from me and a buffer fills up somewhere?

I have tried searching for any issue like this for over a week and could not find anything obvious so any insight would be greatly appreciated.

from previous experience I always find if you need to "delay" there is something else that is wrong.

Righto. Here I suspect you're assuming that a Read on the socket will return a complete message.

But TCP/IP is a streaming protocol, and doesn't include any notion of Messages. To successfully use TCP/IP directly, you must define a "message framing protocol", to communicate to the remote host how many bytes to expect. I don't see you writing anything before the message, so I suspect that's what's missing.

The simple solution that many people use is to always send a 4-byte integer at the beginning of a message indicating the number of bytes that will be sent. Then the reader can perform a 4-byte read, followed by a read loop until the expected number of bytes are read.

So the issue turned out to be a problem with something on the computer itself (or rather on a number of our dev boxes) that affected any service being hosted/connected on a localhost binding. This included a number of network services including even IISExpress and IIS proper if the hosting was local. This resulting in any data over ~10kbytes in size never being sent/received.

After days of trying to work out what was affecting localhost connections we ended up rebuilding our dev boxes and the problem is now gone (albeit we still don't know why/what caused the issue).