Majestic-12 blog

I am a pretty big user of hard disks, and have had all sorts of troubles with them. Normally I buy value for money kind of disks losing which is not big of a deal, but when you buy top of the line hard disk - Hitachi 1 TB worth £250, and then lose it, then it makes you angry! At least 3 out of 20 turned into bricks that BIOS thought to be 32 MB (or 33 MB) big! Apparently this is a bug in Gigabyte motherboard, but no fix for my BIOS (even mobo is recent), and Hitachi’s feature tool that supposedly can fix it was crashing. All was saved however by a great little utility for Restoring Factory Hard Drive Capacity, it is free and it worked! Even all data on disk was not damaged, so today ended up with a smile on my face

This is going to be the last Hitachi I bought I think, I said it couple of years ago when discovering click-o’-death in some of my disks, and only had to go for them this year because there was no choice at the time…

Usually I try to make incremental changes to a debugged big piece of software in order to avoid introducing new bugs that can bite rather painfully later when you least expect it. One reasonably good way of testing that new changes do not break old stuff is to use same inputs and save knowingly good (I call it “gold”) output from software and then compare it with the new output. It should be identical if the changes that you made were designed to improve things like performance or scalability of the same code but not change actual output. This output can be easily compared using fc /b command to know that outputs are exact or just visually check size (more dangerous). Say for example new code might do complex calculations on multiple CPU cores and then merge results, but such results should be exactly the same as if it was running serially on just one core. Sounds simple, but not always!

Read the rest of this entry »

After just having spent some hours trying to compile a C/C++ piece into a 64-bit DLL I came across with a number of error messages searching for which does not yield a lot of great results, so I thought to post them here for the benefit of the others who come across:

1. “unresolved external symbol __security_check_cookie”: Configuration Properties->C/C++->Code Generation switch off Buffer Security Check (/GC- command line) and this error goes away
2. “unresolved external symbol _DllMainCRTStartup” (not to be confused with __DllMainCRTStartup@): Configuration Properties->Linker->Input - make sure “Ignore All Default Libraries” is set to “no”.
3. Could not find kernel32.lib user32.lib etc - if you search for this files in VS2005 directory you will find them in various places, but for 64-bit mode they should be in AMD64 - if they are not there, then go to setup of VS and make sure you installed 64-bit compilers and tools: you can try install recent Platform SDK from Microsoft, however even though it contains these files in correct directory they don’t appear to help compile even if you provide path to them in “Additional Library Directories” in Linker options.
4. “module machine type ‘X86′ conflicts with target machine ‘X64′” - this message appears if you have not got correct 64-bit kernel32.lib or other similar DLLs, so that VS takes 32-bit versions and then can’t link them since they are not 64-bit. This error comes up after you think you “fixed” error #3 by giving path to place where you think correct kernel32.lib exists - solution is the same as in error #3.

Hope this will save a few hours of fruitless efforts trying to understand what the heck Visual Studio is on about when trying to compile a wee 64-bit .DLL

In the last 10 days I tested and found good way to charm grey squirrels that otherwise can be called pretty shy, or to put it less diplomatically rather cowardly. However I managed to find a way that allowed them to overcome their shyness and finally eat tasty roasted peanuts from my hand! You just can’t beat having around 10 squirrels around you greedily looking at you waiting for their turn to grab a nut

When dealing with wild life one has to be careful since they are not called wild for nothing! Squirrels are very furry and cute, but they have very sharp teeth! Today a baby squirrel that was over-enthusiastic over getting a peanut run at high speed and grabbed by index finger rather than peanut, despite wearing a glove it was a successful bite

Well, it was time for a tetanus shot in arm as a precautinary measure as I have not been vaccinated in the last 10 years. Not to put you off feeding squirrels by hand, but one has to be careful - next time it is going to be a bigger glove and that baby squirrel will have to act like adults who come very politely and get nut very carefully, the younster has not learnt any manners yet, he will have to or no peanuts for him!

My long held belief that cheap memory costs more in a long run was confirmed in the last few days when 3 out of 4 chips in two OCZ 4 GB (2×2GB) PC2-5400 Vista Upgrade edition kits failed miserably in a new Intel Q6600 based system - one memory chip would have errors in memtest and with 2 others the system won’t post at all. New delivery of chips from Crucial sorted it all perfectly - 50% more expensive, but can you really afford bad memory that will lead to very weird crashes that would imply other hardware components are at fault?

While on the subject of memory - testing it with memtest also provides nice benchmark showing bandwidth available to processor in case of L1 and L2 caches and actual RAM: the stats for Q6600 (2.4 Ghz) system running with dual channel memory 4×2 GB at 667 Mhz (CL5) shows that actual speed of ram is just below 4 GB/sec - to put this into perspective it suggests that reading all RAM in that server would take whole 2 seconds! This is of course much faster than reading same 4 GB from hard disk, but still - RAM is anything but very fast insofar as processor speeds are concerned: L1 cache for example is rated as almost 400 GB/sec by the very same memtest, this is almost 100 times faster than accessing data from RAM! L2 cache is slower, but still almost 170 GB/sec.

What this means is that those who wish to obtain very high performance from their code should think carefully about algorithms used so that they are cache friendly - otherwise software might run slow because it is bottlenecked by comparatively slow RAM accesses.

You may come across with a very annoying situation whereby your otherwise nice system won’t show more than 3.5 GB of RAM even though you have got 4 GB installed, or even more - granted you need a good 64-bit system to take advantage of that memory in the first place, but if BIOS won’t allow OS to see that memory then you will lose half a gig or more.

The trick is to use “H/w hole mapping” option and set it to Enabled in Award BIOS: this is present in Frequency/Voltage control submenu…only after you press secret key combination to show this option in the first place - Ctrl-Shift-F1, this the case on all of my NForce4 motherboards with AMD X2 CPUs. Information about solution to this problem is rather scarce, so I thought to post it here, enjoy your full 4 GB of RAM!

Squirrels are very cute animals and I always liked them, perhaps probably because my native city was in the middle of woods and there were lots of red squirrels there, very friendly too - they would eat from your hand, an amazing experience for a child, even though parents were not impressed with squirrels getting into the house and stealing all nuts

In the UK there are lots of squirrels too, even though locals do not seem to particularly rate them highly - perhaps because grey squirrels are actually alien species that invaded British Isles and driven off cute local red squirrels. But while I too prefer red squirrels, the greys are also very cute - unfortunately I never managed to get them to eat from my hands, until yesterday that is

Read the rest of this entry »

You might be unlucky to get hit with a very weird .NET application crash with about the following information:

Description:
Stopped working

Problem signature:
Problem Event Name:    APPCRASH
Application Name:    MJ12merger.exe
Application Version:    1.0.2687.33859
Application Timestamp:    4644ba87
Fault Module Name:    mscorwks.dll
Fault Module Version:    2.0.50727.128
Fault Module Timestamp:    45011d30
Exception Code:    c0000005
Exception Offset:    00000000001d7c4c
OS Version:    6.0.5728.2.0.0.274.3
Locale ID:    2057

App would die instantly, no exceptions to show where exactly it happened, very annoying if it happens at the end of very long processing. There seems to be very little information on the net as to why exactly this happens probably because most people won’t push framework as hard as we have to.

The issue here is due to buffer overruns - if you use pointers in .NET code then you have to be very careful not to overwrite your memory buffers because otherwise you will destroy data used by .NET itself and this can result in rather weird failures that are not obviously due to your code: if you use unsafe code you need to assume the worst and be on guard at all times.

Naturally one can choose to use languages that do not support pointers at all, like say Java, but this is the choice of convenience that goes against goals of high performance: in my experience usage of pointers in a handful of tight places can double performance of the same C# code written without using pointers.

A few of our fairly complex applications appear to use a LOT more memory when run on 64-bit Longhorn system - at least 50% more, or even 75-100% when compared to same data being processed on 32-bit system. This is rather annoying as it seems to defeat the purpose of expanding memory from 2 GB to 4 GB, and what’s worse this high wastage makes it hard to use consumer grade Quad cores as too little memory left per core. There seems to be very little information on the topic - looks like adoption rates for 64-bit .NET systems are fairly low and those who happen to come across with these issues keep it mum, hence this blog post which hopefully will attract attention of those who experience this problem.

It is obvious that pointer size is double in 64-bit mode, however this particular application tends to deal with primitive value types (ie: int[], long[], byte[]) that have no pointers to them, though a lot of created and disposed small byte[] arrays might be the reason.

If you start an application in debugging mode under Visual Studio 2005 but then realise that you have forgotten to change some bits of code and quickly kill the app, then VS will greet you with message box about not being able to set breakpoint… for each of the breakpoints in your code and there is no way to skip that damn message so you have to click or press Enter for all those dialogs, grrrrrrrrrrrrr!