I have a Excel 2007 macro which scrapes a multi-page site. The section of the
site I am querying has 302 pages with 10 items of interest on each page. My
macro alters the ?start=xxxxx" parameter on the end of the URL to advance
through the pages.

By page 50 I was starting to get the occasional error 1004. If I waited a
few seconds and retried the qt.refresh, the refresh would work. When this
became painful, I added code around the qt.refresh to try up to 10 times,
with a two-second delay.

My query has now retrieved 250 pages, but has failed again. In this latest
failure, it appears that Excel has not created the querytable at all during
the last pass through the loop.

If I create the web query manually for page 1, I can see the web site in the
wizard. When I click Import, it fails with Error 1004.

If I waited 10 minutes it would refresh.

Any idea what is going on?