Using pipe (|) and grep in Linux and Unix (macOS)

Saturday, July 13, 2024 at 10:02 AM | 8 min read

Last modified on Thursday, August 6, 2026 at 10:30 PM

#macOS, #linux, #grep, #fifo, #named pipe, #pipe, #redirect operator, #unix

Photo by CHUTTERSNAP on unsplash.com

• The pipe (|) command
• The grep command
• The unnamed pipe (|) command in Linux
• Basic use of the grep command
• Combining pipe (|) with grep
• The advantages of using the unnamed pipe and grep together
• So what is a named pipe anyway?
• Creating a named pipe
• Unnamed pipe vs named pipe
• Conclusion
• Related Resources
• Related Posts

I use the grep and unnamed pipe commands together to solve various issues in my daily workflow, so I thought I would write a post about the two commands so that others could benefit from my knowledge and so I could also benefit from having access to a solid reference.

The pipe command, |, aka unnamed pipe, transmits the output of one command to another command. It connects the stdout of the left command to the stdin of the right command.

The grep command, short for "global regular expression print", is used for searching and manipulating text patterns in files.

A basic use of the unnamed pipe (|) is:

ls -la /etc | less

If I ran this command in Terminal, I would get the complete listing of the contents in the /etc folder in Linux, including dot (.) hidden files thanks to the -a flag which includes hidden files, and the long list of the contents inside /etc represented by the -l flag. -la is just short for -l and -a together.

Long list (-l) returns file permissions, the number of links, owner name, owner group, file size, time of last modification, and the file or directory name. To learn more, run man ls in Terminal.

After ls -la /etc, comes pipe (|) followed by less. The pipe command transmits ls -la /etc to the less command. The less command is a Terminal pager program used to view the contents of a text file one Terminal screen (window) at a time. This is great for scrolling through very long files. You just use the up or down arrow keys to scroll through the file contents.

So what the above command means is that the less command is being applied to the ls -la /etc command, which makes it possible to scroll through the contents of /etc with the up or down arrow keys. If I simply used ls -la /etc, I would only see the end screen of the contents of /etc. I would not be able to scroll up to see the rest of it.

A basic use of the grep command would be something like the following:

grep "Doe" names2.txt

where the content of names2.txt is the following:

1 Doe
2 Doe
3 Sue

grep "Doe" names2.txt returns the following in Terminal:

1 Doe
2 Doe

If I ran:

grep -c "Doe" names2.txt

it would return:

2

The -c flag prints the count of matching lines for each grep search. Here, there are two instances, each on a different line, of "Doe".

If I were to run:

grep -v "Doe" names2.txt

it would return:

3 Sue

The -v flag, or --invert-match, selects "non-matching lines". In this instance, it is "Sue" on line three.

Let's say I ran:

grep -i "doe" names2.txt

It would provide:

1 Doe
2 Doe

The -i flag, or --ignore-case, ignores whether the text pattern being searched is upper or lowercase, so that's why, when searching "doe", I am able to get back "Doe".

If I ran:

grep -L "John" *

it would yield:

duplicates.txt
names2.txt

The L flag, short for --files-without-match, returns the files in the current directory that do not contain the text pattern "John".

But if I ran:

grep -l "John" *

It would respond with:

names1.txt

The -l flag, short for --files-with-matches, returns the files that do contain the text pattern being searched in the current working directory. In this case, "John".

"John" followed by * or the wildcard character, is telling grep to search through all the files in the current working directory.

There is much more to grep, and you can learn about it by running man grep in Terminal to access the grep manual.

pipe (|) and grep are frequently used together. Let's say I ran the following in Terminal using both | and grep:

cat names2.txt | grep "Sue"

This returns:

3 Sue

The grep command filters through cat names2.txt to match the text search pattern "Sue". Sweet, huh? I'm going to take this a step further and actually use | and grep to search through a large file. In fact, I'll use /etc again. So:

ls -la /etc | grep -c "root"

would return something like:

252

This means that "root" appears in 252 lines in /etc. /etc sure does have a large number of lines in its output to Terminal!

Let's make | and grep a bit more interesting. Let's say I want to redirect the output of ls -la to /home/maria/Desktop/list-home-directory.txt. I add an absolute path to a new file I want to create called list-home-directory.txt, because I am running it from /home/maria/Desktop. First I type:

ls -la /home/maria > /home/maria/Desktop/list-home-directory.txt

Then I cd into the Desktop directory again, and run cat list-home-directory.txt to make sure that my command works. And it does! I redirect the complete contents of /home/maria to the newly created file on the Desktop called list-home-directory.txt!

Next, I run another command using | AND grep:

cat list-home-directory.txt | grep drwx

This returns the directories where I have read, write, and execute permissions. The command yields:

drwxr-x--- 16 maria maria 4096 Jul 13 11:14 .
drwxr-xr-x  3 root  root  4096 Jul 12 20:47 ..
drwx------ 10 maria maria 4096 Jul 12 20:56 .cache
drwxr-xr-x 15 maria maria 4096 Jul 12 20:56 .config
drwxr-xr-x  2 maria maria 4096 Jul 13 12:23 Desktop
drwxr-xr-x  2 maria maria 4096 Jul 12 20:54 Documents
drwxr-xr-x  2 maria maria 4096 Jul 12 20:56 .icons
drwxrwxrwx  3 maria maria 4096 Jul 12 21:06 .linuxmint
drwxrwxr-x  4 maria maria 4096 Jul 12 20:56 .local
drwxr-xr-x  2 maria maria 4096 Jul 12 20:54 Music
drwxr-xr-x  2 maria maria 4096 Jul 12 20:54 Pictures
drwxr-xr-x  2 maria maria 4096 Jul 12 20:54 Public
drwxr-xr-x  2 maria maria 4096 Jul 12 20:54 Templates
drwxrwxr-x  2 maria maria 4096 Jul 12 20:56 .themes
drwxr-xr-x  2 maria maria 4096 Jul 12 20:54 Videos

I filter through the contents of list-home-directory.txt to find file permissions for directories where user permissions are rwx, and the search text pattern is dwrx, d standing for directory. There are a lot fewer lines than are contained in list-home-directory.txt. This is a quick way of going through a long list of lines to find out which directories in /home/maria have rwx permissions.

Using the unnamed pipe and grep together lets me filter through large files instantly. The combination of both allows me to integrate multiple commands such as cat, ps, or ls with grep. grep's filtering feature permits me to filter out content and show only the lines I need.

An unnamed pipe (|) is not persistent by default. The | command cannot be accessed by another Terminal window session. Only within the current window. The | is created temporarily to accommodate the execution of the command to the left of it and redirect stdout. | is deleted after successful execution of the command.

On the other hand, a named pipe persists as a file until it's deleted. It is a special file that follows the FIFO (first in, first out) method. In fact, it is a FIFO method.

To create a named pipe, I run the following command in the directory I want to save it to:

# Running the command from ~/Desktop/text-files directory/path
mkfifo first-named-pipe

You can call your named pipe whatever you want.

And when I run the ls -la command in the text-files directory to double check my command:

ls -la

it returns:

total 8
drwxrwxr-x  2 maria maria 4096 Aug  6 07:32 .
drwxr-xr-x 11 maria maria 4096 Jul 25 09:15 ..
-rw-rw-r--  1 maria maria    0 Aug  6 07:30 duplicates.txt
prw-rw-r--  1 maria maria    0 Aug  6 07:32 first-named-pipe
-rw-rw-r--  1 maria maria    0 Jul 17 11:53 image.txt
-rw-rw-r--  1 maria maria    0 Aug  6 07:30 list-home-directory.txt
-rw-rw-r--  1 maria maria    0 Aug  6 07:30 names1.txt
-rw-rw-r--  1 maria maria    0 Aug  6 07:30 names2.txt

Next, I run the command below to make sure that executing first-named-pipe does not keep hanging in Terminal:

exec 3<>first-named-pipe

Then I redirect the command I want to capture with first-named-pipe by running:

# Running the command from the ~/Desktop/text-files directory/path
ls -la > first-named-pipe

This redirects ls -la for the contents of text-files as stdout to the first-named-pipe named pipe file.

Finally, I execute the named pipe itself with the cat command:

# Running the command from the ~/Desktop/text-files directory/path
cat first-named-pipe

Which returns:

total 8
drwxrwxr-x  2 maria maria 4096 Aug  6 07:47 .
drwxr-xr-x 11 maria maria 4096 Jul 25 09:15 ..
-rw-rw-r--  1 maria maria    0 Aug  6 07:30 duplicates.txt
prw-rw-r--  1 maria maria    0 Aug  6 07:47 first-named-pipe
-rw-rw-r--  1 maria maria    0 Jul 17 11:53 image.txt
-rw-rw-r--  1 maria maria    0 Aug  6 07:30 list-home-directory.txt
-rw-rw-r--  1 maria maria    0 Aug  6 07:30 names1.txt
-rw-rw-r--  1 maria maria    0 Aug  6 07:30 names2.txt

I did have problems getting my named pipe to work. The fact that I had to set exec 3<> on first-named-pipe so that execution of the named pipe would not hang, and that after the first execution, I was not able to execute again, told me that named pipes just are not worth it. And they cannot handle much data either. I would just stick with shell scripts! If you want your named pipe to work, make sure to take the following steps in the exact order displayed below:

mkfifo first-named-pipe
ls -la
exec 3<>first-named-pipe
ls -la > first-named-pipe
cat first-named-pipe

If you do not run the above commands in that exact sequence, cat first-named-pipe will not work!

I subsequently found the following on Ask Ubuntu:

cat $file >/tmp/stream_pipe didn't work, but I was able to make it work (to some degree) by assigning the pipe to a file descriptor like so:... I said it works to some degree, because running it stand-alone clogs the pipe and the script appears to hang. This is due to the fact that pipes in Ubuntu have a 1M buffer limit as seen via the command systcl fs.pipe-max-size. Unfortunately, settings this value via sysctl doesn't seem to change it. However, so long as the pipe data is being consumed by another process, there shouldn't be an issue. Any writes to a full pipe will block (it looks like a hang, but it's not) until data is consumed at which point the write will continue.

I am using Linux Mint, which is a derivative of Ubuntu, and there is a 1 MiB buffer limit, so I ran the following to see how big my buffer limit was when executing my named pipe:

sysctl fs.pipe-max-size

returns:

fs.pipe-max-size = 1048576

which denotes 1048576 bytes. 1048576 bytes is 1MiB. That is not much in the grand scheme of things, but I was having issues with very small amounts of data as well. Each time I wanted to execute my named pipe, I had to do something like the following:

exec 3<>first-named-pipe
ls -l > first-named-pipe
cat first-named-pipe

I execute the named pipe, but it remains hanging after its output is completed. So I have to press the Control key + C key to exit the pipe.

What is interesting, however, is that after having done the above (several times), I had no problem redirecting stdout to first-named-pipe and then cat > first-named-pipe. I just had to make sure to redirect a command to first-named-pipe before re-executing it. I could not simply re-run cat first-named-pipe right after running cat first-named-pipe. It would result in an empty hang containing no stdout. But if I were to try and redirect a large amount of data, let's say from /etc, that would be way beyond the 1 MiB limit, and first-named-pipe would never execute!

I am still not quite sure what the true benefits of named pipes are, but as I learn more, I will share my findings!

Note: The one advantage of a named pipe is if I run the following in one Terminal window inside the text-files folder:

ls -la > first-named-pipe

And then open up another Terminal window inside text-files and run:

cat first-named-pipe

cat first-named-pipe would execute in Terminal window two.

In this post, I talk about the unnamed and named pipe, the grep command, and how they can be combined together with other commands. The unnamed pipe is implicitly created by typing |, and the named pipe is created with the mkfifo command. However, for the named pipe to work, I have to follow an exact sequence of steps. I also describe how the grep command is often combined with the unnamed pipe. This is advantageous because it allows me to filter through large amounts of text very quickly. I can send the output of one command right into grep without saving temporary files. This saves me both time and disk space.

• An introduction to pipes and named pipes in Linux: by Archit Modi, opensource.com
• How to Use Linux Pipe for Redirection: by Matthew Stevens, Liquid Web
• Grep Command in Linux/UNIX: by Pankaj, Digital Ocean
• Anonymous and Named Pipes in Linux: baeldung.com

• The pipe (|) and tee commands in Linux and Unix (macOS): mariadcampbell.com