[Startup Engineering] The Linux Command Line (#note)

Navigation and Filesystem

list files: ls
The most basic and heavily used command. You will use this constantly to get your bearings and confirm what things do.

# 교재에 ll 이라고 쓰여져 있는데, 기본값은 아니어서 별명으로 선언해줘야 함. alias ll=’ls -alrth’  또는  alias ll=’ls -al’로 선언하면 됨.  ls는 폴더와 파일의 이름만 보여주고, ll(=ls -al)은 파일의  퍼미션(Permissions ;권한)과 작성자, 용량, 생성일자와 시각을 보여줌.

modify/create empty files: touch
Used to change timestamps and quickly create zero byte files as placeholders. Useful for various kinds of tests.

# 타임스탬프를 변경하거나, 0바이트의 파일을 만드는데 이용.

display text: echo
Useful for debugging and creating small files for tests.

# 교제에서 echo -e 라는 명령어를 치는데, echo foo 와 echo -e “foo”와의 정확한 차이를 모르겠다. 입력해보면 echo자체는 입력한 그대로 출력,  echo -e는 n, t 등 문장부호로 쓰는 녀석들을 처리해서 출력해주는 듯.

copy files: cp
Powerful command with many options, works on everything from single files to entire archival backups of huge directories.

move/rename files: mv
Move or rename files. Also extremely powerful.

remove files: rm
Powerful command that can be used to delete individual files or recursively delete entire trees. Use with caution.

symbolic link: ln
Very useful command that allows a file to be symbolically linked, and therefore in two places at once. Extremely useful for large files, or for putting in one level of indirection.

# yes | nl | head -1000 > 1~1000까지 data1.txt로 기록(1~1000까지)

# yes | nl | head -2000 | tail -50 > 1~2000중에 끝에서 1~50개의 숫자를 data2.txt로 기록 (1951~2000까지)

• nl : 이 명령어의 기능은 기본적으로 cat과 동일하나 출력할때 라인넘버를 같이 출력한다.  사용형식은 cat과 동일

# ln -s 특정 파일에 대한 다른 형태의 pathname만 설정. data1.txt, latest.txt 두개의 파일이 생성됨

# ln -sf 심볼링크 파일이 가리키는 오리지날 타겟을 변경하는 듯. sf에 대한 설명은 찾지 못하고 있다.ㅠㅠ

print working directory: pwd
Simple command to print current working directory. Used for orientation when you get lost.

create directories: mkdir
Create directories. Has a few useful options

change current directory: cd
Change directories. Along with ls, one of the most frequent commands.

remove directories: rmdir
Not used that frequently; usually recursive rm is used instead. Note that rm -rf is the most dangerous command in Unix and must be used with extreme caution, as it means “remove recursively without prompting” and can easily nuke huge amounts of data.

Downloading and Syncing

synchronize local and remote files: rsync
Transfer files between two machines. Assuming you have set up your ~/.ssh/config as in previous lectures, run this command on your local machine, making the appropriate substitutions:

retrieve files via http/https/ftp: wget
Very useful command to rapidly pull down files or webpages. Note: the distinction between wget and rsync is that the wget is generally used for publicly accessible files (accessible via a web browser, perhaps behind a login) while rsync is used for private files (usually accessible only via ssh).

interact with single URLs: curl
A bit different from wget, but has some overlap in functionality. curl is for interacting with single URLs. It doesn’t have the spidering/recursive properties that wget has, but it supports 6a much wider array of protocols. It is very commonly used as the building block for API calls.

send test packets: ping
See if a remote host is up. Very useful for basic health checks or to see if your computer is online. Has a surprisingly large number of options.

Basic Text Processing

view files: less
Paging utility used to view large files. Can scroll both up and down. Use q to quit.

print/concatenate files: cat
Industrial strength file viewer. Use this rather than MS Word for looking at large files, or files with weird bytes.

first part of files: head
Look at the first few lines of a file (10 by default). Surprisingly useful for debugging and inspecting files.

last part of files: tail
Look at the bottom of files; default is last 10 lines. Useful for following logs, debugging, and in combination with head to pull out subsets of files.

extract columns: cut
Pull out columns of a file. In combination with head/tail, can pull out arbitrary rectangular subsets of a file. Extremely useful for working with any kind of tabular data (such as data headed for a database).

numbering lines: nl
Number lines. Useful for debugging and creating quick datasets. Has many options for formatting as well.

concatenate columns: paste
Paste together data by columns.

sort by lines: sort
Industrial strength sorting command. Very powerful standalone and in combination with others.

uniquify lines: uniq
Useful command for analyzing any data with repeated elements. Best used in pipelines with sort beforehand.

line, word, character count: wc
Determine file sizes. Useful for debugging and confirmation, faster than nl if no intermediate
information is needed.

split large files: split
Split large file into pieces. Useful as initial step before many parallel computing jobs.

⬆ ========== COMMENT UPDATED ========== ⬆

Help

manuals: man
Single page manual files. Fairly useful once you know the basics. But Google or StackOverflow are often more helpful these days if you are really stuck.

info: info
A bit more detail than man, for some applications.

System and Program information

computer information: uname
Quick diagnostics. Useful for install scripts.

current computer name: hostname
Determine name of current machine. Useful for parallel computing, install scripts, config files.

current user: whoami
Show current user. Useful when using many different machines and for install scripts.

current processes: ps
List current processes. Usually a prelude to killing a process.

most important processes: top
Dashboard that shows which processes are doing what. Use q to quit.

stop a process: kill
Send a signal to a process. Usually this is used to terminate misbehaving processes that won’t stop of their own accord.

Superuser

become root temporarily: sudo
Act as the root user for just one or a few commands.

become root: su
Actually become the root user. Sometimes you do this when it’s too much of a pain to type sudo a lot, but you usually don’t want to do this.

Storage and Finding

create an archive: tar
Make an archive of files.

compress files: gzip
Compress files. Can also view compressed gzip files without fully uncompressing them with zcat.

find a file (non-indexed): find
Non-indexed search. Can be useful for iterating over all files in a subdirectory

find a file (indexed): locate
For locate to work, updatedb must be operational. This runs by default on Ubuntu but you need to manually configure it for a mac.

system disk space: df
Very useful when working with large data sets

directory utilization: du
Use to determine which subdirectories are taking up a lot of disk space.

Intermediate Text Processing

searching within files: grep
Powerful command which is worth learning in great detail. Go through grep –help and try out many more of the options

simple substitution: sed
Quick find/replace within a file. You should review this outstanding set of useful sed oneliners. Here is a simple example of using sed to replace all instances of kinase with STANFORD in the first 10 lines of a ptt file, printing the results to STDOUT:

advanced substitution/short scripts: awk
Useful scripting language for working with tab-delimited text. Very fast for such purposes, intermediate size tool.

Intermediate bash

Keyboard shortcuts
Given how much time you will spend using bash, it’s important to learn all the keyboard shortcuts.
• Ctrl + A: Go to the beginning of the line you are currently typing on
• Ctrl + E: Go to the end of the line you are currently typing on
• Ctrl + F: Forward one character.
• Ctrl + B: Backward one character.
• Meta + F: Move cursor forward one word on the current line
• Meta + B: Move cursor backward one word on the current line
• Ctrl + P: Previous command entered in history
• Ctrl + N : Next command entered in history
• Ctrl + L: Clears the screen, similar to the clear command
• Ctrl + U : Clears the line before the cursor position. If you are at the end of the line, clears the entire line.
• Ctrl + H : Same as backspace
• Ctrl + R: Lets you search through previously used commands
• Ctrl + C : Kill whatever you are running
• Ctrl + D: Exit the current shell
• Ctrl + Z: Puts whatever you are running into a suspended background process. fg restores it.
• Ctrl + W : Delete the word before the cursor
• Ctrl + K: Kill the line after the cursor
• Ctrl + Y : Yank from the kill ring
• Ctrl + _: Undo the last bash action (e.g. a yank or kill)
• Ctrl + T: Swap the last two characters before the cursor
• Meta + T: Swap the last two words before the cursor
• Tab: Auto-complete files and folder names
Let’s go through a brief screencast of the most important shortcuts.

Backticks
Sometimes you want to use the results of a bash command as input to another command, but not via STDIN. In this case you can use backticks:

Running processes in the background: foo &
Sometimes we have a long running process, and we want to execute other commands while it completes. We can put processes into the background with the ampersand symbol as follows.

Execute commands from STDIN: xargs
This is a very powerful command but a bit confusing. It allows you to programmatically build up command lines, and can be useful for spawning parallel processes. It is commonly used in combination with the find or ls commands. Here is an example which lists all files under /etc ending in .sh, and then invokes head -2 on each of them.

Pipe and redirect: tee
Enables you to save intermediate stages in a pipeline.

The reason it is called tee is that it is like a “T-junction”, where it passes the data through while also serializing it to a file. It can be useful for backing up a file while simultaneously modifying it; see some examples.

time any command: time
This is a bash built in useful for benchmarking commands.