View Single Post
  #11   Report Post  
Posted to microsoft.public.excel.programming
Rick Rothstein \(MVP - VB\)[_1577_] Rick Rothstein \(MVP - VB\)[_1577_] is offline
external usenet poster
 
Posts: 1
Default Smart way to delete the first row of a huge file

Okay, I also get the loop quitting well below the maximum string size. Part
of that may be the tight loop (probably interfering with the memory "garbage
collection" routine) and continual concatenations. Each concatenation will
require a larger and larger chunk of contiguous memory which I guess becomes
harder and harder to find within the available (unused, non-garbage
collected) RAM. Anyway, another approach is to process the file in large
"chunks" which I am guessing will still be faster than what you are now
doing. Below my signature is a subroutine that handles the process this way.
You can experiment with this a little bit. I set the chunk size at 20MB (via
the ChunkSize constant assignment)... you can try upping that to, say, 40MB
and see if your system will handle it (I don't have a 200MB file to test the
code out on). Anyway, give it a try and let me know how it works out.

Rick

Sub RemoveFirstLine(PathAndFileName As String)
Dim InFileNum As Long
Dim OutFileNum As Long
Dim ReadLocation As Long
Dim LoopChunk As Long
Dim BackSlash As Long
Dim PartialFile As String
Const ChunkSize As Long = 20000000
BackSlash = InStrRev(PathAndFileName, "\")
' The next 4 lines must be executed in the order shown
InFileNum = FreeFile
Open PathAndFileName For Binary As #InFileNum
OutFileNum = FreeFile
Open Left$(PathAndFileName, BackSlash) & "TEMP_" & _
Mid$(PathAndFileName, BackSlash + 1) For Append As #OutFileNum
LoopChunk = 5000
PartialFile = Space$(LoopChunk)
Get #InFileNum, , PartialFile
Print #OutFileNum, Mid$(PartialFile, InStr(PartialFile, vbCrLf) + 2);
Do While Not (EOF(InFileNum))
PartialFile = Space$(ChunkSize)
Get #InFileNum, , PartialFile
If FileLen(PathAndFileName) - Seek(InFileNum) <= 0 Then
PartialFile = Left(PartialFile, InStr(PartialFile, Chr$(0)) - 1)
End If
Print #OutFileNum, PartialFile;
Loop
Close #InFileNum
Close #OutFileNum
Kill PathAndFileName
Name Left$(PathAndFileName, BackSlash) & "TEMP_" & _
Mid$(PathAndFileName, BackSlash + 1) As PathAndFileName
End Sub






"Charles" wrote in message
...
I did a test using the following loop

Sub test2()
Dim txt As String, i As Long
For i = 1 To 190
Application.StatusBar = i
txt = txt & Space(1000000)
Next i
MsgBox "Success"
End Sub

it breaks at the step 105, which seems to mean that VBA is running out
of memory with a string larger than 104m characters. Excel total
memory usage was around 350MB, so far below the 2GB XL limit, and my
installed memory (about 2GB as well)
So it looks like there is another memory limit for strings.

Charles