Welcome to Terminal Tips, a series where we’ll be sharing some shortcuts and tricks we’ve picked up along the way as we use the command line. Today’s post covers a command that is more frequently associated with a text editor than the command line: “Find and Replace”. If you need to do a find and replace on a file, especially a large file, it becomes cumbersome to do so in a text editor. Github’s new editor, Atom, won’t let you open files larger than 2MB as of the time of this post.
So let’s say you have a CSV file with a couple hundred thousand rows and you want to change any occurrences of the string ‘old’ to the string ‘new’. You could do this using a word processor or an editor, but it would likely crash before you even got the file open, or it would take forever to actually perform the find and replace. What you need is something that gets the job done without all the overhead of an editor. This is where sed comes in. A quick look at the manual for sed tells us:
The sed utility reads the specified files, or the standard input if no files are specified, modifying the input as specified by a list of commands.
(If you want to read more, open up the Terminal and run the command ‘man sed’ without the quotes)
So now that we know what we’re working with, let’s start to construct our command.
From the man page, the -i flag edits files in place, allowing you to specify a backup. If you’re not concerned about running out of disk space this isn’t absolutely necessary, so we’ll just leave that blank for now. Our command now looks like:
sed -i ''
Now we actually have to specify the text we want to find and replace. If we want to replace the word old with the word new, we need to construct a suitable expression to tell sed what it is looking for:
's/old/new/g'
From the manual, the -e flag will “Append the editing commands specified by the command argument to the list of commands.” Sounds perfect. Our command now looks like this:
sed -i '' -e 's/old/new/g'
Now all we have to is specify the path to the file you want to run the command on. If we had a CSV file called list.csv in our current directory, we could just append that to the command. Now we have:
sed -i '' -e 's/old/new/g' list.csv
When we run this command and go look in the file, we can see that all instances of ‘old’ have been replaced with ‘new’ and you’ve learned a new shell command!
Extra Credit
What if you wanted to find and replace a URL? You can’t use a slash as the seperator character, because there will most likely be slashes in your URL. sed uses whatever follows the ‘s’ in the regular expression as the seperator. So, for example, we could use a comma as a seperator.
sed -i '' -e 's,http://www.oldurl.com,http://www.newurl.com,g'
We can really use any symbols we want.
sed -i '' -e 's#http://www.oldurl.com#http://www.newurl.com#g'
Keep an eye on the blog in the coming weeks for more Terminal Tricks!