Navigation and Filesystem
list ﬁles: ls
The most basic and heavily used command. You will use this constantly to get your bearings and conﬁrm what things do.
# 교재에 ll 이라고 쓰여져 있는데, 기본값은 아니어서 별명으로 선언해줘야 함. alias ll=’ls -alrth’ 또는 alias ll=’ls -al’로 선언하면 됨. ls는 폴더와 파일의 이름만 보여주고, ll(=ls -al)은 파일의 퍼미션(Permissions ;권한)과 작성자, 용량, 생성일자와 시각을 보여줌.
modify/create empty ﬁles: touch
Used to change timestamps and quickly create zero byte ﬁles as placeholders. Useful for various kinds of tests.
# 타임스탬프를 변경하거나, 0바이트의 파일을 만드는데 이용.
display text: echo
Useful for debugging and creating small ﬁles for tests.
# 교제에서 echo -e 라는 명령어를 치는데, echo foo 와 echo -e “foo”와의 정확한 차이를 모르겠다. 입력해보면 echo자체는 입력한 그대로 출력, echo -e는 n, t 등 문장부호로 쓰는 녀석들을 처리해서 출력해주는 듯.
copy ﬁles: cp
Powerful command with many options, works on everything from single ﬁles to entire archival backups of huge directories.
move/rename ﬁles: mv
Move or rename ﬁles. Also extremely powerful.
remove ﬁles: rm
Powerful command that can be used to delete individual ﬁles or recursively delete entire trees. Use with caution.
symbolic link: ln
Very useful command that allows a ﬁle to be symbolically linked, and therefore in two places at once. Extremely useful for large ﬁles, or for putting in one level of indirection.
# yes | nl | head -1000 > 1~1000까지 data1.txt로 기록(1~1000까지)
# yes | nl | head -2000 | tail -50 > 1~2000중에 끝에서 1~50개의 숫자를 data2.txt로 기록 (1951~2000까지)
- nl : 이 명령어의 기능은 기본적으로 cat과 동일하나 출력할때 라인넘버를 같이 출력한다. 사용형식은 cat과 동일
# ln -s 특정 파일에 대한 다른 형태의 pathname만 설정. data1.txt, latest.txt 두개의 파일이 생성됨
# ln -sf 심볼링크 파일이 가리키는 오리지날 타겟을 변경하는 듯. sf에 대한 설명은 찾지 못하고 있다.ㅠㅠ
print working directory: pwd
Simple command to print current working directory. Used for orientation when you get lost.
create directories: mkdir
Create directories. Has a few useful options
change current directory: cd
Change directories. Along with ls, one of the most frequent commands.
remove directories: rmdir
Not used that frequently; usually recursive rm is used instead. Note that rm -rf is the most dangerous command in Unix and must be used with extreme caution, as it means “remove recursively without prompting” and can easily nuke huge amounts of data.
Downloading and Syncing
synchronize local and remote ﬁles: rsync
Transfer ﬁles between two machines. Assuming you have set up your ~/.ssh/config as in previous lectures, run this command on your local machine, making the appropriate substitutions:
retrieve ﬁles via http/https/ftp: wget
Very useful command to rapidly pull down ﬁles or webpages. Note: the distinction between wget and rsync is that the wget is generally used for publicly accessible ﬁles (accessible via a web browser, perhaps behind a login) while rsync is used for private ﬁles (usually accessible only via ssh).
interact with single URLs: curl
A bit diﬀerent from wget, but has some overlap in functionality. curl is for interacting with single URLs. It doesn’t have the spidering/recursive properties that wget has, but it supports 6a much wider array of protocols. It is very commonly used as the building block for API calls.
send test packets: ping
See if a remote host is up. Very useful for basic health checks or to see if your computer is online. Has a surprisingly large number of options.
Basic Text Processing
view ﬁles: less
Paging utility used to view large ﬁles. Can scroll both up and down. Use q to quit.
print/concatenate ﬁles: cat
Industrial strength ﬁle viewer. Use this rather than MS Word for looking at large ﬁles, or ﬁles with weird bytes.
ﬁrst part of ﬁles: head
Look at the ﬁrst few lines of a ﬁle (10 by default). Surprisingly useful for debugging and inspecting ﬁles.
last part of ﬁles: tail
Look at the bottom of ﬁles; default is last 10 lines. Useful for following logs, debugging, and in combination with head to pull out subsets of ﬁles.
extract columns: cut
Pull out columns of a ﬁle. In combination with head/tail, can pull out arbitrary rectangular subsets of a ﬁle. Extremely useful for working with any kind of tabular data (such as data headed for a database).
numbering lines: nl
Number lines. Useful for debugging and creating quick datasets. Has many options for formatting as well.
concatenate columns: paste
Paste together data by columns.
sort by lines: sort
Industrial strength sorting command. Very powerful standalone and in combination with others.
uniquify lines: uniq
Useful command for analyzing any data with repeated elements. Best used in pipelines with sort beforehand.
line, word, character count: wc
Determine ﬁle sizes. Useful for debugging and conﬁrmation, faster than nl if no intermediate
information is needed.
split large ﬁles: split
Split large ﬁle into pieces. Useful as initial step before many parallel computing jobs.
⬆ ========== COMMENT UPDATED ========== ⬆
Single page manual ﬁles. Fairly useful once you know the basics. But Google or StackOverﬂow are often more helpful these days if you are really stuck.
A bit more detail than man, for some applications.
System and Program information
computer information: uname
Quick diagnostics. Useful for install scripts.
current computer name: hostname
Determine name of current machine. Useful for parallel computing, install scripts, conﬁg ﬁles.
current user: whoami
Show current user. Useful when using many diﬀerent machines and for install scripts.
current processes: ps
List current processes. Usually a prelude to killing a process.
most important processes: top
Dashboard that shows which processes are doing what. Use q to quit.
stop a process: kill
Send a signal to a process. Usually this is used to terminate misbehaving processes that won’t stop of their own accord.
become root temporarily: sudo
Act as the root user for just one or a few commands.
become root: su
Actually become the root user. Sometimes you do this when it’s too much of a pain to type sudo a lot, but you usually don’t want to do this.
Storage and Finding
create an archive: tar
Make an archive of ﬁles.
compress ﬁles: gzip
Compress ﬁles. Can also view compressed gzip ﬁles without fully uncompressing them with zcat.
ﬁnd a ﬁle (non-indexed): find
Non-indexed search. Can be useful for iterating over all ﬁles in a subdirectory
ﬁnd a ﬁle (indexed): locate
For locate to work, updatedb must be operational. This runs by default on Ubuntu but you need to manually conﬁgure it for a mac.
system disk space: df
Very useful when working with large data sets
directory utilization: du
Use to determine which subdirectories are taking up a lot of disk space.
Intermediate Text Processing
searching within ﬁles: grep
Powerful command which is worth learning in great detail. Go through grep –help and try out many more of the options
simple substitution: sed
Quick ﬁnd/replace within a ﬁle. You should review this outstanding set of useful sed oneliners. Here is a simple example of using sed to replace all instances of kinase with STANFORD in the ﬁrst 10 lines of a ptt ﬁle, printing the results to STDOUT:
advanced substitution/short scripts: awk
Useful scripting language for working with tab-delimited text. Very fast for such purposes, intermediate size tool.
Given how much time you will spend using bash, it’s important to learn all the keyboard shortcuts.
• Ctrl + A: Go to the beginning of the line you are currently typing on
• Ctrl + E: Go to the end of the line you are currently typing on
• Ctrl + F: Forward one character.
• Ctrl + B: Backward one character.
• Meta + F: Move cursor forward one word on the current line
• Meta + B: Move cursor backward one word on the current line
• Ctrl + P: Previous command entered in history
• Ctrl + N : Next command entered in history
• Ctrl + L: Clears the screen, similar to the clear command
• Ctrl + U : Clears the line before the cursor position. If you are at the end of the line, clears the entire line.
• Ctrl + H : Same as backspace
• Ctrl + R: Lets you search through previously used commands
• Ctrl + C : Kill whatever you are running
• Ctrl + D: Exit the current shell
• Ctrl + Z: Puts whatever you are running into a suspended background process. fg restores it.
• Ctrl + W : Delete the word before the cursor
• Ctrl + K: Kill the line after the cursor
• Ctrl + Y : Yank from the kill ring
• Ctrl + _: Undo the last bash action (e.g. a yank or kill)
• Ctrl + T: Swap the last two characters before the cursor
• Meta + T: Swap the last two words before the cursor
• Tab: Auto-complete ﬁles and folder names
Let’s go through a brief screencast of the most important shortcuts.
Sometimes you want to use the results of a bash command as input to another command, but not via STDIN. In this case you can use backticks:
Running processes in the background: foo &
Sometimes we have a long running process, and we want to execute other commands while it completes. We can put processes into the background with the ampersand symbol as follows.
Execute commands from STDIN: xargs
This is a very powerful command but a bit confusing. It allows you to programmatically build up command lines, and can be useful for spawning parallel processes. It is commonly used in combination with the find or ls commands. Here is an example which lists all ﬁles under /etc ending in .sh, and then invokes head -2 on each of them.
Pipe and redirect: tee
Enables you to save intermediate stages in a pipeline.
The reason it is called tee is that it is like a “T-junction”, where it passes the data through while also serializing it to a ﬁle. It can be useful for backing up a ﬁle while simultaneously modifying it; see some examples.
time any command: time
This is a bash built in useful for benchmarking commands.