Majestic-12 blog

These days programmers have to deal with setups that contain multiple cores so coding in a way that takes advantage of extra parallel processors is becoming matter of life and death. At Majestic-12 we have a small framework that allows to parallelise long running tasks but from time to time we run into strange things one of which happened again today. Our application was processing data on 8 cores, about 8 TB of data in fact so high IO can be expected to make processors wait for data to be crunched however it turned out that application was running slower than it should be - disk IO could not be to blame. Have a look at CPU usage history below:

Roughly CPU usage was about 60-63%, this is well below what it should have been. To cut long story short it turned out that adding the following configuration option to .NET application configuration file helped:

According to Microsoft gcServer can help in multi-CPU setups, and it does - just have a look at CPU usage of the same application when it was enabled:

This pretty much got CPU usage closer to 93-95% per CPU, which is about where it should have been in the first place when allowing for a fair amount of disk reads.

But what exactly happens here? It appears that default mode of garbage collection would stop ALL threads while collecting garbage, this was effectively pausing processing on multiple cores. You can see another option above - gcConcurrent - this in theory should make garbage collection even faster, however I found its usage is buggy in .NET 2.0, so I’d recomment to be very careful when enabling this option - I keep it turned off.

Finally since I started talking about memory stuff in .NET and garbage collection the biggest lesson of all to learn was to reuse memory - this is the best way to avoid overheads in parallel processing as well as save on actual memory usage - this is the key to high performance processing in .NET (and really any other language that uses garbage collection).

This entry was posted on Saturday, June 7th, 2008 at 2:01 pm and is filed under C# / .NET. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.