Solved!

This took a lot of trial and error.

I made a small test CHM and it worked fine with Application.Help. I looked
up all the settings in the test and main CHM and all were identical. The main
CHM continued to give the 'Out of memory' error in Excel.

Finally found the difference. It was that the main CHM filename contained
spaces. Removed the spaces and everything is working ok now. Go figure.