By Alvin Alexander. Last updated: February 1, 2024
Unix/Linux tip: How to change/edit the content on thousands of files using awk
and a Unix shell script.
As a note to self, today I had to delete the same lines (or line patterns) from over 10,000 files on this Unix/Linux server. Basically I was looking in each file for a specific pattern, and if that pattern was found, a ceertain number of lines should be deleted, starting at that point.
WARNING: Mass-changing thousands of files like this is dangerous. Before doing anything like this, always make a backup of your files.
I decided the best solution was to create an Awk script like this:
#!/bin/awk -f
# set a flag when the trigger line is found
/MY_PATTERN/ {
trigger_found = 1
line_count = 0
next
}
# increment line_count and do not print if trigger_found and line_count <= 10
trigger_found && line_count <= 10 {
line_count++
next
}
# reset trigger_found and line_count if trigger_found and line_count > 10
trigger_found && line_count > 10 {
trigger_found = 0
line_count = 0
}
# print all other lines
{
print
}
Then I put a list of all the files I wanted to modify in a file named file_list
, and ran this Unix/Linux shell script:
#!/bin/sh
for file in `cat file_list`
do
echo "$file ..."
awk -f delete_unwanted_lines.awk $file > TMP
mv TMP $file
done