Question : Visual Basic Search and Replace through large text file

Hello. A few months ago an expert helped me develop visual basic code to perform a search and replace in a data file. At the time the file was about 100mb. The file has grown to over 200mb and the program does not work anymore. Is there a way to modify the code so that it won't bomb on files over 200mb? Any help would be greatly appreciated.

Thank you
Diammond

Code Snippet:

         
           
             1:
2:
3:
4:
5:
6:
7:
8:
9:
10:
11:
12:
13:
14:
15:
16:
17:
18:
19:
20:
21:
22:
23:
24:
25:
26:
27:
28:
29:
30:
31:
32:
33:
34:
35:
36:
37:
38:
39:
40:
41:
42:

           
           
             Imports System.Text.RegularExpressions
Module Module1
    Public strNewPhoneNumber As String = """2126420001"""
    Public intRepCount As Integer = 0
    Sub Main()
        If IO.File.Exists("c:\reports\replaced.log") Then
            IO.File.Delete("c:\reports\replaced.log")
        End If
        Dim srDataFile As New System.IO.StreamReader("c:\reports\data.txt")
        Dim strDataFile As String = srDataFile.ReadToEnd
        srDataFile.Close()
 
        Dim srPhoneNumbers As New System.IO.StreamReader("c:\realPhoneNumbers.txt")
        Dim strPhoneNumbers As String = srPhoneNumbers.ReadToEnd
        srDataFile.Close()
 
        Dim reNumbers As Regex = New Regex("(?<=#PhoneNumber\r\n"")\d+(?="")")
        Dim mcNumbers As MatchCollection = reNumbers.Matches(strPhoneNumbers)
        Dim strPhoneNumbersToExclude As String = ""
        For Each mNumber As Match In mcNumbers
            strPhoneNumbersToExclude = strPhoneNumbersToExclude & mNumber.Groups(0).Value & "|"
        Next
        strPhoneNumbersToExclude = Left(strPhoneNumbersToExclude, Len(strPhoneNumbersToExclude) - 1)
 
        Dim matchpattern As String = "(?<=#PhoneNumber\r\n)""(?:(?!" & strPhoneNumbersToExclude & ")\d+)"""
        Dim myEval As MatchEvaluator = New MatchEvaluator(AddressOf ReplaceMatch)
 
        Dim swNewDataFile As New System.IO.StreamWriter("c:\reports\data.txt")
        swNewDataFile.Write(Regex.Replace(strDataFile, matchpattern, myEval))
        swNewDataFile.Close()
        Console.WriteLine(intRepCount & " replacements, finished at " & Now())
    End Sub
    Public Function ReplaceMatch(ByVal m As Match) As String
        Dim swLogFile As New System.IO.StreamWriter("c:\reports\replaced.log", True)
        Dim strLogLine As String = "Replaced " & m.Groups(0).Value & " with " & strNewPhoneNumber & " " & Now()
        Console.WriteLine(strLogLine)
        swLogFile.WriteLine(strLogLine)
        swLogFile.Close()
        intRepCount = intRepCount + 1
        Return strNewPhoneNumber
    End Function
End Module
           
         

Open in New Window

Select All

Answer : Visual Basic Search and Replace through large text file

We could try "flushing" the streams before continuing...and place a hard-coded pause before attempting the delete/rename to allow the operating system time to catch up with writing/releasing the file handles:

                       
         
           
             1:
2:
3:
4:
5:
6:
7:
8:
9:
10:
11:
12:
13:
14:
15:
16:
17:
18:
19:
20:
21:
22:
23:
24:
25:
26:
27:
28:
29:
30:
31:
32:

           
           
                     Using sr As New System.IO.StreamReader(DataInFile)
            Using sw As New System.IO.StreamWriter(DataOutFile, False)
                Using log As New System.IO.StreamWriter(LogFile, False)
                    Dim line As String = sr.ReadLine
                    While Not IsNothing(line)
                        If Not IsNothing(PreviousLine) Then
                            If PreviousLine.StartsWith("#PhoneNumber") Then
                                If line.StartsWith(Chr(34)) AndAlso line.Length > 2 Then
                                    If Not Phones.ContainsKey(line) Then
                                        log.WriteLine("Replaced " & line & " with " & AllowedNumber & " on " & DateTime.Now.ToString)
                                        line = AllowedNumber
                                    End If
                                End If
                            End If
                        End If
 
                        sw.WriteLine(line)
                        PreviousLine = line
                        line = sr.ReadLine
                    End While
 
                    sw.Flush() ' <----------------- flush the streams
                    log.Flush()
                End Using
            End Using
        End Using
 
        ' <----------------- hard-coded pause ----------------->
        System.Threading.Thread.Sleep(TimeSpan.FromSeconds(10).TotalMilliseconds) ' let the OS do it's thing before we continue
 
        My.Computer.FileSystem.DeleteFile(DataInFile)
        My.Computer.FileSystem.RenameFile(DataOutFile, DataInFile)
           
         

Open in New Window

Select All