spilly39 expressed precisely :
I'd like to see the code you are using to read the lines
- shouldn't be more than a few lines of CODE (as opposed to DATA that is!)
I've handled large log files, tho' not in the GB range
It's vital you do not read the whole file, but just a block at a time and
within that only one logical line at a time. My issue was that some data
files were exUnix (Lf) and other data ex Windoze (CrLf line terminated) and
VB Line Input statement needs CRLF data.
My solution was to write a class (UnixDosReader) with properties of filename,
size etc, and methods of fOpen, GetNextLine and fClose
I'm looking at my code now (not having used it since I retired)
I see I handled the file unblocking in VBA myself if it was a Unix file,
reading one block of data (of whatever size I chose) at a time.
What I did was to scan the 1st lump of the data at file open time to
determine if it was LF or CRLF data
If CRLF in block 1, then my GetNextLine method reads the file one line at at
time using std VB IO code
Line Input #pFileNo, dataLine
If it's Unix it gets a bit more complicated, with a Private Sub inside the
class:
ReadaBlock(s as String, Size as Integer) 'size is size of block buffer
which internally did
Get #pFile,,s '(read into s String argument)
You have to gwt it right at the end of the file, which doesn't fill your
standard blocksize.
Altogether there's 273 lines of code (of which the 1st 94 are solid comments)
I could send the code - it's a bit long to post here
BUT I have one little nagging worry...
FileSize is typed as Long, which on my 32 bit machine only gets to 2GB. I
assume you're on a 64 bit box? Just checking..
You'll have to look carefully at any such Typing matters if you want to use
my code
spilly
Thanks for stepping in!
I ran into the same issues with reading larger files in all at once.
Like you I've never done GB-size files but have had to read files as
large as 10MB in blocks as you describe. For me, however, this was a
onetime workaround and so I never bothered to refine it beyond that.
Your solution sounds exactly what this OP needs. I'd also be very
interested if you're willing to share.
--
Garry
Free usenet access at
http://www.eternal-september.org
ClassicVB Users Regroup! comp.lang.basic.visual.misc