Searching and Manipulation with Grep

Grep is a very convenient way to manipulate and search through lots of data and logs. Also, it is in Module 3.3 of the OSCP Syllabus too. I often use grep to parse log files and to quickly arrange and access information in an easily readable manner. It is a time saver and a great tool for CTF’s, Pentesting, log analysis, and information gathering.

What is grep?

Grep stands for Global Regular Expression Print. What that means is that grep will Globally search for a Regular Expression and then Print the matching lines in the console. I find it a little easier to understand when broken down that way.

Grep is used in Linux and Unix machines to search for strings of characters in files and when a match is found, it will print the targeted results. The grep command is a life saver when doing any type of data parsing and is a must to learn.

How grep Works

Grep’s basic usage starts with the grep command, a Flag if one is issued, followed by the string or text you are searching for, then the name of the file that the grep will search through. Using grep at its core function is quite simple but can become complicated as we add flags later.

What a Simple grep Command Looks Like

┌──(kali㉿kali)-[~]
└─$ grep test ~/Desktop/grep_test.txt 
This is a test document. 
This is the end of the grep test document. 

The above command shows grep, followed by test, followed by ~/Desktop/grep_test.txt. This tells grep to search for the word test in the file grep_test.txt that is located on the Desktop. The output also displays any line that contained the word test in the file.

The grep command can be used to manipulate data and show many different pattern variations to suite your needs. Just be sure to combine as many flags as necessary to find and display the results you require. Below are some common grep flag commands and brief examples for them.

Noob Note: grep is case-sensitive by default. Use the -i flag to remove case-sensitivity.

Grep Examples

Search for a String in a Specified File

This is the basic grep command to print all lines in a file that contain a specific string, in our case hello in the file sample1.

┌──(kali㉿kali)-[~/Desktop]
└─$ grep hello sample1
(botany) phelloderm is part of the periderm made up of cells produced inwardly by the cork cambium.
Ending my hello and adding goodbye! 

As shown above, using grep this way will display every line where hello is a match. Instead of showing exact matches, the terminal prints any line containing the string and substring hello. Therefore, the line with the word phelloderm is shown. Two lines were displayed since the file had two instances of hello, even though one is a substring in phelloderm.

Tip: If your search pattern includes any blank spaces or symbols, use quotation marks to encapsulate the string and make it searchable.

Search for String in Multiple Files

Using grep to search multiple files is straight forward, just add each file you want to search through at the end of the command.

┌──(kali㉿kali)-[~/Desktop]
└─$ grep hello sample1 sample2 sample3      
sample1:(botany) phelloderm is part of the periderm made up of cells produced inwardly by the cork cambium.
sample1:Ending my hello and adding goodbye!
sample2:(botany) phelloderm, ya im using this again
sample3:hello?

The example above, shows grep searching for the string hello in three separate files named sample1, sample2, sample3. It also prints the name of each line that contained the matching string hello.

Noob Note: You can add as many files needed in your search.

Searching the Current Directory

When searching for a string in every file that is located in the current directory, replace the filename with an asterisk *. This will search through every file in the current directory for the specified string.

┌──(kali㉿kali)-[~/Desktop]
└─$ grep hello *                               
grep: Caldera: Is a directory
grep: elf: Is a directory
grep: Hashcat: Is a directory
grep: Ophcrack: Is a directory
sample1:(botany) phelloderm is part of the periderm made up of cells produced inwardly by the cork cambium.
sample1:Ending my hello and adding goodbye!
sample2:(botany) phelloderm, ya im using this again
sample3:hello?
sample4:and yet another file with hello inside!
grep: wireshark dump: Is a directory
grep: Zenmap: Is a directory

The example above, shows a grep search in the current directory ~/Desktop for any file with the matching string hello. It then returns the name of the file and every line with the matching string hello. It will also display the name of any sub directory encountered but will not search in them. This is because the asterisk * tells grep to only search the current directory and not the sub directories.

Note: A new file called sample4 was found containing the string hello, this was not found using the previous flags.

Find an Exact Match and Exclude Substrings

Grep can also be used to find and display any line that contains the exact string. This means it can exclude partial matches when they are a substring of a larger word (show whole string and exclude all substrings). To use this, we will include the -w flag.

┌──(kali㉿kali)-[~/Desktop]
└─$ grep -w hello sample1 sample2 sample3      
sample1:Ending my hello and adding goodbye!
sample3:hello? 

Above, grep shows a search through sample1 sample2 sample3 for lines with the string hello and will exclude any substring with hello. This result differs from the previous results that included the word phelloderm. Since hello was a substring within that word, it was not included in the current results.

Ignore Case in grep Search

Grep commands are case sensitive by default, remembering the flag -i is very useful for many grep searches. This flag will cause all lowercase, uppercase, and mixed case results to be displayed instead of only the default lowercase results.

┌──(kali㉿kali)-[~/Desktop]
└─$ grep -i hello sample1 sample2 sample3 
sample1:Hello!
sample1:(botany) phelloderm is part of the periderm made up of cells produced inwardly by the cork cambium.
sample1:Ending my hello and adding goodbye!
sample2:Hello everybody!
sample2:(botany) phelloderm, ya im using this again
sample3:Hello and goodnight!
sample3:hello?

The example above, shows more results containing the string hello when compared to the previous results and flags used. We can now see uppercase, lowercase, and mixed case strings that were previously not shown in the results.

Recursively Search Directories

Use the -r flag to recursively search the directories for every files that matches the string. This command prints every line that contain the matching string for all files in the current directory and subdirectories.

┌──(kali㉿kali)-[~]
└─$ grep -r 'another file with hello'                                                                                             
.zsh_history:grep 'another file with hello' -R
.zsh_history:grep 'another file with hello' -r
.zsh_history:find . -type f -exec grep -l 'yet another file with hello' {} \;
.zsh_history:grep -r 'another file with hello' .
Desktop/sample4:and yet another file with hello inside!

The example above, shows a string search for another file with hello with the -r flag used to search the current and subdirectories. The search results this time included commands from .zsh_history as well, because zsh_history is in a subdirectory of the current directory grep was issued on.

Tip: Quotation marks were used to encapsulate the search string since it contained black spaces.

Inverted grep Search

Grep can be used for an inverted search and will work in opposite of what we have done so far. Instead of printing the line that contains the string you are searching for, it will exclude that line and print every other line in the file that does not contain the matching string. The -v flag is used for this inverted search.

┌──(kali㉿kali)-[~/Desktop]
└─$ grep -vi hello sample1

Just filling in this space.
hope you are all doing well. 

The example above, omits all lines in the file sample1 that contains the string hello and instead displays any line in the file that does not contain the string hello. I added the -i flag to make the inverted search case insensitive.

Find Files that Contain Matching String

When you need to quickly find a file that contain a certain string. To print the file name that contains the string and omit the lines the string is located on, use the -l flag.

┌──(kali㉿kali)-[~/Desktop]
└─$ grep -lri hello
Hashcat/10-million-password-list-top-1000000.txt
Hashcat/10-million-password-list-top-10000.txt
Hashcat/probable-top-1575.txt
Hashcat/rockyou.txt
Hashcat/pokemon-wordlist
Ophcrack/XP Special/table0.bin
sample3
sample4
sample2
sample1

The example above, shows the name of any file that contains the matching string hello. This is used for file searching only and does not print the lines with the string in it. In this example the -r and -i flags are used to make the search recursive and case insensitive too.

Display Line Numbers with grep Results

When searching through log files or files with many results, it can be important to see the line number the string is located on. To do this, use the -n flag to show the line numbers.

┌──(kali㉿kali)-[~]
└─$ grep -niw hello ~/Desktop/sample1
1:Hello!
6:Ending my hello and adding goodbye!

The example above, shows a search for hello in ~/Desktop/ in the file sample1. Adding the -i and -w flags will make the search case insensitive and also search for the exact string and omit substrings in the search.

Search for Multiple Strings

Searching for multiple strings at once can also be done in grep. This is done by encapsulating the different search strings and using a separator or | in the grep search. This is activated with the flag -E.

┌──(kali㉿kali)-[~]
└─$ grep -Eiw '(hello|goodnight)' ~/Desktop/*     /home/kali/Desktop/sample1:Hello!
/home/kali/Desktop/sample1:Ending my hello and adding goodbye!
/home/kali/Desktop/sample2:Hello everybody!
/home/kali/Desktop/sample3:Hello and goodnight!
/home/kali/Desktop/sample3:hello?
/home/kali/Desktop/sample4:and yet another file with hello inside!

The example above, shows multiple flags and grep commands used together. The flag -E allows for multiple string searches to be used and the syntax '(hello|goodnight)' can be broken down as: Use quotations to encapsulate the entire string and then allows grep to read anything inside the quotations as part of the search parameter; (hello|goodnight) tells grep that two different strings hello and goodnight are being used for the search and are separated with the | command.

Why it is Useful

The grep command can be used as a basic search tool or can be used as a powerful data manipulator to help simplify complex log results. Grep is commonly used to parse data from large log files to easily search for what is required.

Example: If you had thousands of log entries from a compromised phone and only needed to know the android log data of times the camera was on, you could use grep -i -E 'torchmode' to search and omit the other log information.

┌──(kali㉿kali)-[~]
└─$ cat /home/kali/Desktop/system.log | grep -i -E 'torchmode' 
07-18 13:08:24.626  7251  7310 I CameraManager: setTorchMode : cameraId = 0, enabled = true
07-18 13:08:24.626  1286  6815 D CameraService: setTorchMode E - enabled: 1
07-18 13:08:24.626  1286  6815 I CameraService: setTorchMode[2272] enabled(1)
07-18 13:08:24.626  1286  6815 I CameraFlashlight: setTorchMode[78]: set torch mode of camera 0 to 1
07-18 13:08:24.626  1286  6815 V CameraFlashlight: setTorchMode: set camera 0 torch mode to 1
07-18 13:08:24.632  1286  6815 D CameraService: setTorchMode X
07-18 13:08:24.632  7251  7310 I CameraManager: setTorchMode : cameraId = 0, enabled = true, strength = 3
07-18 13:08:24.632  1286  7195 D CameraService: setTorchModeStrength E - enabled: 1, strength: 3
07-18 13:08:24.632  1286  7195 I CameraService: setTorchModeStrength[2410] enabled(1)
07-18 13:08:24.632  1286  7195 I CameraFlashlight: setTorchMode[141]: set torch mode of camera 0 to 1 with strength 3
07-18 13:08:24.632  1286  7195 V CameraFlashlight: setTorchMode: set camera 0 torch mode to 1 with given strength 3
07-18 13:08:24.635  1286  7195 D CameraService: setTorchModeStrength X
07-18 13:08:24.635  7251  7608 I CameraManagerGlobal: onTorchModeChanged
07-18 13:09:29.919  7251  7310 I CameraManager: setTorchMode : cameraId = 0, enabled = false
07-18 13:09:29.921  1286  9279 D CameraService: setTorchMode E - enabled: 0
07-18 13:09:29.921  1286  9279 I CameraService: setTorchMode[2272] enabled(0)
07-18 13:09:29.922  1286  9279 I CameraFlashlight: setTorchMode[78]: set torch mode of camera 0 to 0
07-18 13:09:29.922  1286  9279 V CameraFlashlight: setTorchMode: set camera 0 torch mode to 0
07-18 13:09:29.925  1286  9279 D CameraService: setTorchMode X
07-18 13:09:29.926  7251  7608 I CameraManagerGlobal: onTorchModeChanged

That is just one example of the variety of ways that grep can be used. It has such an abundance of variations and usefulness that I can only scratch the surface on a single post! Practice makes perfect and grep definitely becomes easier the more you use it.

TL;DR

Grep is simple yet can become a powerful search tool when used in conjunction with the right flags. It is especially useful when paired with piping and combining it with other tools.

grep string_here file_here search for string in file.

grep -i ignore case sensitivity, search for string.

grep -v exclude string and print the other lines (inverted search).

grep string_here * use * instead of filename to search the entire directory.

grep -r recursive search for string (current directory, subdirectory, and files inside them).

grep -w extract string search and excludes substrings.

grep -v -E '(string1|string2)' Include multiple strings in search.

grep -E '(string1|string2)' exclude multiple strings in search.