Styjun

I am writing a huge amount of data to an Azure table from a console application in C#. This does the following

1. Open a connection to an already existing table.


2. Reads a file using StreamReader


3. Collect 100 queries at a time and does a batch write to the table.

A single process has a write speed of about 500-700/s without parallelization and to write 1 billion entries would take about 30 days to complete. I did the following to optimize:

1. Created 20 processes executing the above exe and this ran in parallel without any issues, reducing the write time to 1.5 day. [Ideal scenario which I cannot do unfortunately due to restrictions in our code base]

6 cores/12 logical processors:

+----------+---------------------------------+--------+------------------------------------
| #process | Time per process per 10k writes | W/s | Total time in hours (1.3b queries) |
+----------+---------------------------------+--------+------------------------------------+
| 2 | 14.2s/10k | 1408/s | 256h |
| 4 | 14.5s/10k | 2758/s | 130h |
| 6 | 14.6s/10k | 4109/s | 87h |
| 8 | 15s/10k | 5333/s | 65h |
| 12 | 16.1s/10k | 7453/s | 48h |
| 16 | 17.9s/10K | 8888/s | 42h |
| 18 | 19s/10k | 9473/s | 38h |
| 20 | 21.37s/10k | 9358/s | 39h |
+----------+---------------------------------+--------+------------------------------------

1. core/1 logical processor took almost the same time. As observed, the write time increases linearly with number of processes and independent of number of cores and logical processors weirdly. Azure tables have a max IOPS of about 20K per second.




2. Create a set of 20 tasks in the console application. This was not optimal and performance deteriorated as the number of cores reduced or number of threads increased. The best performance was observed for 2 Tasks. I tried changing the min limit in threadPool but that did not change anything.ThreadPool.SetMinThreads(20, 20);

Code:

foreach (var index in processIndex)

 Task t = Task.Run(() => 
 
 //gets the appropriate file to read and write to table
 string currentFile = string.Format(outFileFormat, index);
 Service service = new Service(currentFile);
 service.JustReadFile();
 );
 tasks.Add(t);

tasks.WaitAll();

Performance:

+--------+--------+------------------------------------+
| #tasks | W/s | Total time in hours (1.3b queries) |
+--------+--------+------------------------------------+
| 2 | 1408/s | 256h |
| 16 | ~800/s | ~488h |
| 18 | ~800/s | ~488h |
| 20 | ~800/s | ~488h |
+--------+--------+------------------------------------+

In the above code all I am doing is reading the file for the respective task. Each task has its pre-assigned file to read. No writing to azure table is happening here and that itself is having a detrimental performance on increasing number of tasks. I thought perhaps the tasks are competing for resources or there is too much overhead in context switching. Since each task has its own Service object, I believe that may not be the case. I also do think reading files and creating objects is a I/O intensive task but if 20 processes can handle it, then so could 20 tasks?

What is happening here and how can I solve this using threads/tasks?

thanks!