Excel isn't the tool for handling two dimensional arrays of 10000 by
10000.

Use MatLab, Mathematica, S, or APL (or J for Excel).

Excel is a presentation package with a backend Visual Scripting
language: VBA.

As to the fact that it worked in 2002 but not in 2007. Yeah, there are
many subtleties associated with building a C++ DLL and having that DLL
use memory like you want it to.

It could be that the 2002 Excel engine was 16 bit and 2007 is 32 bit.

But who cares? Excel isn't the tool for that job.