How to loop through lines of text in a text file from a shell script dynamically

There are times when we need to dynamically run shell commands based on the items listed in a text file.

The shell script that I ran in my bash shell does the following:
for each line separated by newline and not preceded by a '#' character, print the line to console.

In order to build the shell script, I would need the following code pieces:

  • A way to remove/delete empty lines or leading and trailing whitespace in a text file
  • A way to loop through the text file, line by line
  • A way to check if the first character of the string

This post documents what I had ran in my bash shell (in Ubuntu 14.0.4) to loop through items separated by newlines in a text file.

1. Using the sed command to remove/delete empty lines or leading and trailing whitespace in a text file

The text files that my shell script process could include empty lines or leading and trailing whitespace that probably makes the files readable to the author. However, lines that are empty or are strings that contain only leading and trailing whitespace, could affect the commands that my shell script would run against.

To solve this problem, we can use the sed command to remove/delete all unnecessary whitespace from the text file:

sed '/^[ \t\r\n]*$/d' textFile.txt

In this code, I had passed two input parameters to the sed command.

The first input consists of a regular expression and the corresponding action to perform if any line matches the regular expression. /^[ \t\r\n]*$/ is the regular expression that matches every string that contains either \t, \r or \n consecutively. The d tells sed to delete every string that matches the regular expression.

The second input is the path to my text file.

2. Looping through the contents in the file, line by line.

We can loop through the contents of file with the following shell script code:

while read line; do
    # do something with the 'line' variable
done

3. Checking if the first character of a line is a character that is not a '#' character

To check if the first character of a line is not a '#', I can use the following shell script code:

if [[ ${line:0:1} != "#" ]]; then
    # The first character of $line will not contain the '#' character
fi

Putting all parts together: the shell script code to loop through lines of text in a text file

Putting the main code elements together, my shell script looks like the following:

sed '/^[ \t\r\n]*$/d' textFile.txt | while read line; do
    if [[ ${line:0:1} != "#" ]]; then
	  echo "$line"
    fi
done

I directed the output of the sed command to a while loop that reads content line by line. Inside the while loop, I check for the first character of a line that is not a '#' character. If the first letter of that line is not a '#' character, I use the echo command to print out the contents of the line.

About Clivant

Clivant a.k.a Chai Heng enjoys composing software and building systems to serve people. He owns techcoil.com and hopes that whatever he had written and built so far had benefited people. All views expressed belongs to him and are not representative of the company that he works/worked for.