Unix

Basic Unix

Note: I will refer to Linux and Unix interchangeably. For the purposes of this (basic) tutorial, they are interchangeable. The differences come when you start using more advanced programs, such as 'ldd' and 'otool'.

First steps: Where is Unix

There are many ways to access Unix. Some systems run Unix as the default operating system. Other systems, like Linux, run a graphical operating system (for example, Red Hat Linux or Ubuntu Linux have windows, toolbars, wallpaper, etc., much like the Windows operating system).

To access the Unix parts of the operating system, the user interacts with the computer through a command line. The interpreter between the user and the low-level programs and scripts is called the "shell". Two common shells are Bash and Tcsh. There are many others, however, like the Korn shell (ksh), the Bash shell (sh), the C shell (csh), the Z shell (zsh), and even a Python shell (pysh)!

In many systems, including Mac OS X, the underlying Unix layer is accessed using the Terminal application. Opening the Terminal will give you a new, empty window with a command prompt.

When you use unix or linux through a command line, you give certain commands to do things like open folders, copy things, and edit files. You issue these commands, and the system goes and finds the script or program responsible for that command and executes it. These commands have customizable options, called flags, that modfiy what the script or program does. We should start with an example:

Basic Linux Commands

The first command we'll cover is the most important. It is the 'manual' command, which gives you information about other commands:

$ man cd

This will give you information about the 'cd' (change directory) command. We'll be exploring some of the information that the Unix manual gives you below.

The 'list' command:

$ ls

(Note: I will represent the prompt with a (#)$, where (#) is the reference number/command number.) This command will list all of the files that are in my current folder. When I open a Terminal, by default the terminal is in my "home folder" (we'll see what exactly that means in a little bit). I have a folder for my Documents, my Pictures, maybe folders for Downloads and Libraries. I may have files here, too.

But there are also "hidden" files in Unix (usually these are configuration files for programs). If I make a new folder or a new file with a . in front of its name, like .report or .passwords, these files and folders will not show up. In order to show those files, we have to add a flag to 'ls':

$ ls -a

This lists ALL files. The following lists files in a particular directory:

$ ls -a /usr/bin

This will list ALL files in the folder /usr/bin (this folder holds many system programs). This example should demonstrate one problem with 'ls': if there is a large number of files or folders, there isn't enough screen space to display them all. There's a simple way around this, however:

$ ls | less

This demonstrates the concept of a '|'. This script will take the output of the command 'ls' and use it as an input command to 'less'. In this case, 'less' is a program called a pager, which allows you to scroll up and down from the command line through large amounts of text. It accepts different key commands than the command line does, however. For example, in order to scroll through text, I can use my arrow keys. To quit, I use the 'q' key.

$ ls -R

This command will list (recursively) the contents of the current folder (unless a location is given, as in (4) above). The output of this command is usually large, so you should use it with less, as in (5) above.

$ ls -lh

This command shows how multiple flags can be combined. ls -l lists files in a list, with each line representing the file and information about the file. ls -h converts the default size listing (which is in bytes, w/o any units given) to being ``human-readable, e.g. with units like GB (gigabytes), MB (megabytes), KB (kilobytes), etc. Simply putting them together does both. You can combine all these things for a big, complicated, but concise command:

$ ls -lha /dev | less

Can you figure out what this will do? All the cryptic-looking files are part of how Unix runs. Some files represent devices, some represent pieces of hardware in your computer, and some represent interfaces programs can use to communicate with software or hardware. You can use Unix a lot without ever having to know what these files do, but it's good to be aware they're there.

Here are a few more useful commands:

The 'print working directory' (pwd) command

$ pwd

The pwd command tells you where you're at in your file structure. If you listed the directory /usr/local, does that mean you're in /usr/local? No! You told the command to list a specific folder, not move to a specific folder! If you wanted to move to that folder, you have to use a different command.

So, as mentioned earlier, a new terminal window will open to your home directory. What does that mean? Use 'pwd' to show you where your home directory really is relative to / (called the root directory). So, when I say the folder /usr, I mean the folder usr, which is located in the root folder /.

The 'change directory' (cd) command

$ cd /usr/local

Issuing this command will move your current location to the folder local, which is in the folder usr, which is in the root folder /. Now use pwd to confirm where you are.

Now try issuing it without an argument: just type cd at the command prompt, and use pwd to see where it takes you.

$ cd

This command will take you to your home directory. Instead of typing out /home/username, you can use the symbol '~', which represents your home directory. So command (11) above is the same as:

$ cd ~/

Because both mean 'change to home directory'.

$ cd ..

This is a handy command that will move you up one folder (for example, from /some/obscure/directory/ to /some/obscure/).

The 'list' (ls) command

This will list the contents of a directory. You can give it no arguments to list the contents of your current directory:

(14) $ ls
Desktop 	Library 	Pictures
Documents 	Movies 	Public
Downloads 	Music 	Sites

The 'who am i' command

$ whoami

In case you are having an identity crisis in the middle of your dozens of open terminals. However, it only gives you your username; it's not a built-in psychiatrist. For that, you will need Emacs.

The 'copy' (cp) command

$ cp /home/username/secrets.txt /ftp/public

This command makes all of "username"'s secrets public (ftp is a way for you to make documents available for people to download). There are lots of handy flags, like:

$ cp -i

You can use the -i flag with a source and target, and it will prompt you before overwriting any files or directories. Another useful one is:

$ cp -l /some/obscure/directory/customprogram/ /home/username/programs/customprogram/

This command will create links to a folder in the specified destination. But the command, as typed, won't work, because we are trying to copy the contents of a folder recursively. Normal copy is only used to copy single files, so we need to use a flag to tell it to copy recursively:

$ cp -r -l /some/obscure/directory/customprogram/ /home/username/programs/customprogram/

Editing Files

(Note: if you want to learn more, you're gonna want to go to the Vim or Emacs page.)

One of the really important tasks in Unix is editing files using a text editor. There are many options, but one text editor that is built into virtually every Unix system is vim. It's a really useful program to know because it is so universal.

Let's say I have a file I want to edit, called some_boring_file. To edit it in vim, I will use the cd command to change to the directory where some_boring_file is located. Once I'm there, I can use the command

$ vim some_boring_file

to open the file in vim. If there is no file name some_boring_file, then vim will open an empty document with the name some_boring_file.

One of the first things you'll notice is that when you type, no letters show up. This is because vim has two "modes" - command mode, and text editing mode. You start out in command mode. When you press 'i' (without the quotes), you enter text edit mode and you can start typing just like normal. To leave text edit mode, you can press Esc (escape key), and you'll be returned to command mode. And finally, the command used to quit is ':q'... the colon is used to enter a command, and q quits. If you have changed the file, vim will tell you that you either have to save the file, or you have to quit without saving. To save a file, use ':w' to write the changes to the file. To quit without saving, put an exclamation point after the q, like this: ':q!' (but without the quotes).

Configuration files & System variables

In Unix systems, almost every program can be customized by editing an associated configuration file or files. For example, the file ~/.vimrc can contain configuration lines for the text editor vim. The shell you use to issue commands on the command line can also be customized. If you're using the bash shell, you can edit the file ~/.bashrc, while if you're using the csh shell or the tcsh shell, you can edit the file ~/.cshrc or ~/.tchsrc. These files are passed, line-by-line, to the shell, which runs the commands contained in them. These files are perfect for adding or modifying system variables, among other things.

One of the most important system variables that is automatically set is $PATH. You can see what $PATH is equal to from a command prompt:

$ echo $PATH

You should see something that looks more or less like this: /usr/bin:/bin:/usr/sbin:/sbin:/usr/local/bin:/usr/texbin:/usr/X11/bin:/opt/local/bin:/usr/X11R6/bin. So you can see $PATH is just a long list of locations. These are all locations where Unix looks for commands when you type them on the command line. If you type asdfjklqwertyuiop to run the command named asdfjklqwertyuiop, asdfjklqwertyuiop must correspond to a script or a program that will perform the task you want, and so Unix looks inside all of the folders listed on $PATH for asdfjklqwertyuiop. If it can't find it, it will give you an error:

$ asdfjklqwertyuiop

-bash: asdfjklqwertyuiop: command not found

There are three important folders located in your $PATH variable, and they are: /usr/bin, /usr/sbin, and /usr/local/bin. First, usr stands for Unix Shared Resources, so this folder contains things that are shared by all users. Second, bin stands for binary. When you download a program, it is often written in C or C++ or some high-level programming language. In order to run that program, you must compile the code, which creates a binary file. This binary file is made specifically for your machine and your operating system - you couldn't take a binary compiled for an Intel Mac OS X and run it on a Dell workstation runing Red Hat Linux.

/usr/bin contains system resources; this is where most of the commands like ls, cd, cp, etc. reside.

/usr/sbin contains system resources reserved only for the super-user (the computer administrator). These are system programs that nobody but the administrator should be running.

/usr/local/bin contains binaries for locally installed software. So if you download source code from the internet, unpack it, and compile it to run on your machine, this is where most programs will be installed by default.

One other handy system variable is $SHELL. I mentioned earlier that there are different shells, such as bash or tcsh. The system variable $SHELL is always set to what shell you're using. So if you type:

$ echo $SHELL

you can see the name of the shell you're using. If you want a list of all the system variables that are set on your system, you can use

$ env

This will print all variables currently set.

I can use the file .bashrc or .tcshrc to modify $PATH (I don't want to completely change it, however, as the already-existing locations in $PATH are very important to keeping things working. To modify ~/.bashrc or ~/.tcshrc, I can use vim:

$ vim ~/.bashrc

This file may exist already (in which case we'll add our lines to the end of it), or it may be empty. Either way, adding the following lines to the file will modify our $PATH variable to include our home directory:

For .bashrc:

export PATH=/home/username:$PATH

wq (in vim... this is the vim command that will write changes to the disk, then quit).

For .tcshrc:

set PATH /home/username:$PATH

Compiling Software

There is another page on the topic of software compilation here: Compiling Software.

Intermediate Unix

Okay, so you've figured out how to find your way out of a paper bag using the command prompt. Here's some more stuff to keep you busy.

*nix Family Tree

As you can see, the foundations of each all have elements of Unix, but Linux (originating from the GNU project) is the furthest from this. Mac runs on a variation of the BSD kernel, called Darwin, so the differences between Mac and Linux are the same as the differences between BSD and Linux.

If you're curious, you can use a virtualization program like VirtualBox to install and run any of these operating systems.

To understand how Linux, Unix, and Mac are related, refer to the graphic here: http://upload.wikimedia.org/wikipedia/commons/5/51/Unix_history.svg.

Filesystem

There is an established standard dictating what goes where in the filesystem. This is shared by BSD and Linux, as well as most other Unix systems (hey, it is a standard, after all). You can read more about it at Wikipedia. And if you don't like it, you can use Gobo Linux, it uses a more intuitive filesystem layout.

Remote Computing (SSH)

I often have to use a remote computer, for work or to access files on my computer at home. To do that, I can use SSH. The machine which I connect to must be running an SSH server. To connect with a remote computer, I simply execute the ssh command, and tell it where to connect to:

$ ssh user@hostname.com

I can also use SSH to transfer files using the scp command. This command takes two arguments: (1) what you're copying, and (2) where you're copying it to. Either the from or the to command can be a remote host - it doesn't matter. So you could say:

$ scp ~/secret_file user@secretfileserver.com:~/

This will copy a secret file from the local home directory to a user's home directory on the remote computer secretfileserver.com. The LOCAL username is completely independent of the REMOTE username, so the two do not need to match. Likewise, this copy command could have been executed from the secretfileserver computer remotely:

[secretfileserver] $ scp yourname@homecomputer.com:~/secret_file ~/.

The only restriction here is that the user's home (or, local) computer must have a public address. In many cases, computers will have public addresses. However, if your computer doesn't, you can use an IP address in the place of a hostname, like so:

[secretfileserver] $ scp yourname@101.102.103.104:~/secret_file ~/.

Once you've mastered ssh and scp, you can visit the SSH page to learn how to use SSH to create password-secured encrypted tunnels for data - any kind of data - passing between two computers. This can be used for things like remote desktop, file sharing, sharing a web browser window, retrieving email messages, etc...

Dot Files

Main article: Dot files

Recommended Resources

Books

I highly recommend O'Reilly books. Two very nice Unix books from O'Reilly include:

Powers, Shelly; Peek, Jerry; O'Reilly, Tim (2002). Unix Power Tools (3rd ed.). O'Reilly Media. ISBN 978-0596003302. 

Robbins, Arnold (2005). Unix in a Nutshell (4th ed.). O'Reilly Media. ISBN 978-0596100292. 

Sites

Grymoire home for unix wizards (covers find/bash/sed/awk/regexps/etc) http://www.grymoire.com/Unix/