Mastering Batch Processing with Linux xargs Command

📆 · ⏳ 4 min read · ·

Introduction

In today’s fast-paced world, time is of the essence, and getting things done quickly and efficiently is key. Linux, being a popular operating system, provides a wide range of powerful command-line tools to facilitate speedy execution of tasks.

One such tool is the xargs command, which can be used to perform batch processing on multiple files or inputs.

What is xargs Command

The xargs command is a Linux utility that is used to build and execute command lines from standard input. It is commonly used in combination with other commands such as find, grep, and ls, to process a large number of files or inputs.

The command reads input items separated by whitespace, and executes a specified command on each item.

The basic syntax of the xargs command is as follows:

Terminal window
xargs [options] [command [initial-arguments]]

Some of the commonly used options include:

  • -a file: read items from a file instead of standard input.
  • -I replace-str: replace occurrences of replace-str in the initial-arguments with the input item.
  • -n num: use at most num arguments per command line.
  • -P max-procs: run up to max-procs processes at once.

Here are some practical examples of how the xargs command can be used for batch processing:

Find and Delete Files

Suppose you have a directory containing thousands of files, and you want to delete all files with a particular extension, say .log. You can use the following command to accomplish this:

Terminal window
find . -name "*.log" | xargs rm

The find command searches for all files with the .log extension, and passes them to the xargs command.

The xargs command then executes the rm command on each file, effectively deleting them.

Convert Multiple Files

Suppose you have a directory containing several image files in the .png format, and you want to convert them all to .jpg format. You can use the following command to achieve this:

Terminal window
ls *.png | xargs -I {} convert {} {}.jpg

The ls command lists all files with the .png extension, which are then passed to the xargs command.

The xargs command uses the -I option to replace occurrences of {} in the convert command with the input items. The convert command then converts each file to the .jpg format.

Parallel Execution

Suppose you have a directory containing several large text files, and you want to compress them all using gzip. You can use the following command to achieve parallel execution using the xargs command:

Terminal window
ls *.txt | xargs -P 4 -n 1 gzip

The ls command lists all files with the .txt extension, which are then passed to the xargs command.

The xargs command uses the -P option to specify the maximum number of processes to run at once (in this case, 4), and the -n option to specify the number of arguments per command line (in this case, 1).

The gzip command then compresses each file in parallel, effectively speeding up the process.

Execute Multiple Commands

Suppose you have a list of files and you want to perform two commands on each file: convert it to a PDF and then compress it. You can use the following command to accomplish this:

Terminal window
cat files.txt | xargs -I {} sh -c 'convert {} -compress zip {}.pdf'

The cat command reads the list of files from the files.txt file, which are then passed to the xargs command.

The xargs command uses the -I option to replace occurrences of in the sh command with the input items.

The sh command executes the convert command on each file, converting it to a PDF and then compressing it.

Conclusion

The xargs command is a versatile and powerful tool that can be used for batch processing tasks on Linux. It allows you to efficiently process a large number of files or inputs in one go, saving you time and effort.

By combining it with other commands, you can perform complex operations on multiple files in a single command, simplifying your workflow and increasing productivity.

Whether you’re managing large data sets or automating repetitive tasks, the xargs command is a valuable addition to your Linux toolkit.

You may also like

  • Mount a drive permanently with fstab in Linux

    Let's see how to mount a drive permanently in Linux using the fstab file which will mount the drive automatically on boot.

  • Setup Jellyfin with Hardware Acceleration on Orange Pi 5 (Rockchip RK3558)

    Recently I moved my Jellyfin to an Orange Pi 5 Plus server. The Orange Pi 5 has a Rockchip RK3558 SoC with integrated ARM Mali-G610. This guide will show you how to set up Jellyfin with hardware acceleration on the Orange Pi 5.

  • HTTPS with self-signed certificates for your Homelab services

    In this article we will deep dive into understanding how we can setup HTTPS with self-signed certificates for our Homelab services.This is often required when you are running your own services and you want to access them over HTTPS.